OpenTelemetry: What It Is and How It Fits Together
OpenTelemetry is a CNCF project that standardises how applications produce and export telemetry. Before it existed, every observability vendor required its own SDK, its own agent, and its own wire format. Switching backends meant reinstrumenting the application. Teams running more than one tool maintained multiple pipelines for the same data.
OTel collapses that to one instrumentation layer. You add the SDK once. It emits OTLP — the OpenTelemetry Protocol — a vendor-neutral wire format that every major observability backend now accepts. The choice of backend becomes an operational decision rather than a code change.
The Three Signals
OTel’s three established telemetry signals are traces, metrics and logs, and each answers a different class of question. The project also lists baggage as a signal, but it isn’t telemetry you store: it’s key-value context that travels with a request between services. Profiles, a fourth kind of telemetry, are still in development.
Traces answer “what happened to this request?” A trace is a directed acyclic graph of spans — one span per unit of work — recording when each operation started, how long it took, whether it succeeded, and what attributes it carried. Distributed tracing links spans across service boundaries using propagated context headers, so a single trace can span an API gateway, three microservices, a message queue consumer, and a database call. Traces are the signal for understanding request paths and diagnosing latency.
payment.process span is the one you added by hand — and the one holding the time. The dashed span is where it broke.Metrics answer “how is the system behaving right now?” Metrics are numeric measurements aggregated over time: counters that only increase, gauges that represent a current value, and histograms that record distributions. Where a trace tells you that one request took 800ms, a metric tells you that p95 latency has been above 600ms for the last five minutes. Metrics are the signal for alerting and capacity planning.
Logs answer “what happened and in what context?” Logs are time-stamped records of discrete events. When correlated with traces via a shared trace_id, they provide the narrative detail — error messages, business context, state transitions — that explains what a trace reveals about timing. Logs are the signal for root cause analysis.
The three signals are complementary. An alert fires on a metric; the metric points to a time range; correlated traces show which requests degraded; correlated logs explain why.
How the Pieces Fit Together
The SDK lives in your application. Each language has its own implementation — Go, Java, Python, .NET, Node.js, and others — but the APIs are consistent across all of them. The SDK provides TracerProvider, MeterProvider, and LoggerProvider; your instrumentation code uses those to create spans, record measurements, and emit log records. Automatic instrumentation libraries for common frameworks (HTTP servers, database drivers, message brokers) hook into these providers without requiring changes to application code.
The Collector is a standalone proxy that sits between your services and your backends. It receives telemetry over OTLP (or a range of other protocols — Jaeger, Prometheus, Zipkin, and more — though Jaeger itself now speaks OTLP natively, so the dedicated jaegerreceiver is increasingly optional rather than the primary ingestion path), applies a processing pipeline, and exports to one or more destinations. The Collector handles concerns that don’t belong in the application SDK: batching, sampling decisions at the trace level, attribute redaction for PII, fanout to multiple backends, and protocol translation. Running without a Collector is possible for simple setups; at any meaningful scale, the Collector is the right architectural boundary.
Backends are the storage and query layers — Grafana Tempo for traces, Prometheus or Mimir for metrics, Loki or OpenSearch for logs. OTel is backend-agnostic; the Collector’s exporter configuration is the only place vendor specifics appear.
Resource Attributes: The Identity Layer
Every piece of telemetry produced by your application carries a resource — a set of attributes describing the entity that produced it. At minimum: service name, service version, and deployment environment. These attributes appear on every span, metric, and log record the service emits, and they are what makes cross-signal correlation possible in the backend.
Set them in the service itself, where the SDK reads them from standard environment variables in every language:
OTEL_SERVICE_NAME=checkout-api
OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=production
Two traps. There’s no OTEL_SERVICE_VERSION variable, so the version goes in OTEL_RESOURCE_ATTRIBUTES with everything else. And the environment attribute is deployment.environment.name: plain deployment.environment is deprecated in the semantic conventions, though plenty of examples still use it. Don’t set service.name in a shared Collector, either. A resource processor that upserts it rewrites the identity of every service sending through that Collector.
Resource attributes are defined by OTel semantic conventions — a shared vocabulary that makes the same field names mean the same thing across every service and every language SDK.
Where to Go Next
By signal:
- Structured Logging: Making Your Logs Machine-Readable — what schema-consistent logs unlock for automated analysis
By concept:
- Context Propagation — how trace context crosses service boundaries
By language (how-tos):
One observability idea, every Tuesday
A short take, one thing to try that week, and a link to the full article. No vendor pitches.
Subscribe on Substack