OpenTelemetry: API, SDK, Collector, OTLP
What it is
OpenTelemetry is a vendor-neutral standard for producing telemetry, made of four separable pieces that people routinely conflate:
API the interface your code and your LIBRARIES call.
No-op by default. Depending on it commits you to
nothing.
SDK the implementation the APPLICATION wires up:
sampling, batching, resource detection, exporters.
The application chooses it; libraries never do.
OTLP the wire protocol. Protobuf over gRPC (:4317) or
HTTP (:4318). One format for traces, metrics, logs
and profiles.
COLLECTOR a standalone process: receivers -> processors ->
exporters. Where you do sampling, redaction,
enrichment and fan-out, without redeploying
applications.
SEMANTIC agreed attribute names (http.request.method,
CONVENTIONS server.address, db.system.name). What makes a
dashboard portable across services and languages.
The API/SDK split is the whole design. A library can instrument itself against the API and add no runtime behaviour and no backend dependency; the application decides whether anything is recorded and where it goes. That is why instrumentation can live upstream in libraries at all, which is the thing no vendor SDK ever achieved.
What this is confused with: a monitoring backend. OpenTelemetry produces and routes telemetry. It does not store or query it, and choosing it does not choose Prometheus, Jaeger, Grafana, Datadog or Honeycomb. The value is that switching those becomes a Collector config change rather than a re-instrumentation project.
Also confused: OpenTelemetry and distributed tracing. Tracing is one signal. Metrics and logs are first-class, share the same resource model and context, and profiles are the newest addition.
The problem it solves
Before it, instrumentation was a per-vendor commitment baked into application code.
Switching from vendor A to vendor B meant:
- replacing an agent or SDK in every service, in every
language
- re-instrumenting anything custom
- rewriting dashboards and alerts, because attribute names
differ (http.method vs http.request.method vs
request.method vs verb)
- running both for a migration period, at double cost
- and libraries you depend on emitted vendor A's format,
which you could not change
Observed consequence: teams did not switch, which is the
outcome the pricing depended on.
And the second problem, which is worse and less discussed: nothing correlated.
Metrics from Prometheus, traces from a vendor agent, logs from
a shipper. Three systems, three identity models, no shared
trace id.
Debugging: see a latency spike in Grafana, guess a time range,
search logs by service name, find nothing, guess again.
The workflow that actually works, spike -> exemplar -> trace ->
the specific log lines for that trace, requires that all three
signals carry the same identifiers, which requires a common
context propagation mechanism, which is what OpenTelemetry
standardises.
Mechanics
The API/SDK separation, concretely
# LIBRARY code. Depends only on the API package.
# If the application never configures an SDK, every call here
# is a no-op with near-zero cost.
from opentelemetry import trace
tracer = trace.get_tracer("mylib.http", "2.1.0")
def fetch(url: str):
with tracer.start_as_current_span("mylib.fetch") as span:
span.set_attribute("url.full", url)
span.set_attribute("http.request.method", "GET")
resp = _do(url)
span.set_attribute("http.response.status_code", resp.status)
if resp.status >= 500:
span.set_status(trace.StatusCode.ERROR)
return resp
# APPLICATION code. Wires the SDK once, at startup.
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
resource = Resource.create({
# service.name is REQUIRED. Without it everything arrives as
# "unknown_service", which is the single most common
# misconfiguration.
"service.name": "checkout",
"service.version": "2026.03.1",
"deployment.environment.name": "prod",
})
provider = TracerProvider(
resource=resource,
# Head sampling here. Tail sampling belongs in the Collector,
# because only the Collector sees the whole trace.
sampler=ParentBased(root=TraceIdRatioBased(0.1)),
)
# Batch, never simple, in production: SimpleSpanProcessor exports
# synchronously on span end and adds the export latency to the
# request.
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)
Three details that cause most production incidents with OpenTelemetry:
1. service.name missing -> everything is unknown_service
2. SimpleSpanProcessor in prod -> export latency is added to
every request, synchronously
3. no memory limit on the
batch queue -> a slow exporter grows the
queue until the process OOMs.
Bound the queue and accept
dropped spans; telemetry must
never take down the service.
Point 3 is the principle worth stating generally: telemetry is best-effort and must fail open. A tracing pipeline that can crash the application it observes has inverted its purpose.
Context propagation
W3C Trace Context, two headers:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
^ ^ ^ ^
| trace-id (16 bytes) parent span-id flags
version (8 bytes) (01 = sampled)
tracestate: vendor-specific key/value, for systems that need
to carry their own state alongside
Baggage is separate: application key/values propagated with the
request (tenant id, experiment cohort, criticality). Useful and
DANGEROUS: baggage travels to every downstream service and, if
you copy it onto spans, straight into your cardinality bill.
The sampling flag propagating is what makes traces complete. ParentBased sampling means a
service respects the upstream decision, so a sampled trace stays sampled through twelve services
rather than each one flipping its own coin and producing a trace with holes.
The Collector, and why it exists
+-- exporter: Prometheus
app --OTLP--> Collector (agent) --> Collector (gateway) --+
+-- exporter: Jaeger/Tempo
+-- exporter: a vendor
AGENT mode: one per host or a sidecar. Cheap local work:
receive, add host/k8s resource attributes, batch, forward.
Its job is to get data off the application process fast.
GATEWAY mode: a scaled deployment. Expensive, stateful, or
whole-trace work: tail sampling, redaction, aggregation,
fan-out to several backends.
Tail sampling MUST be in a gateway, and the gateway must route
all spans of a trace to the same instance (a load-balancing
exporter keyed on trace id), because you cannot decide whether
a trace was interesting until you have all of it.
What the Collector buys you, stated as capability rather than architecture:
- change your backend without touching applications
- redact a newly discovered PII attribute in one place, in
minutes, rather than in 40 services over a quarter
- drop a high-cardinality attribute that just tripled your bill,
without a redeploy
- send the same data to two backends during a migration
- absorb a backend outage with a queue, instead of dropping
telemetry at the application
The redaction case is the one that justifies it on its own. When someone discovers that an attribute contains an email address, the fix is a Collector processor change and a config rollout, not 40 service deploys.
The annotated pipeline configuration, with processor ordering and the reasoning behind it, is in the observability pipeline; the sampling choices are in the same page.
Semantic conventions
The point: an attribute name that means the same thing
everywhere, so a dashboard and an alert work across services
written in different languages by different teams.
Stable conventions cover HTTP, database, messaging, RPC, GenAI,
FaaS, and resource attributes (service, host, k8s, cloud).
Examples of the current stable names:
http.request.method GET
http.response.status_code 200
server.address api.example.com
url.path /v1/orders
db.system.name postgresql
db.query.text SELECT ...
messaging.system kafka
error.type timeout
The migration cost is real and worth knowing about, because it is the most common complaint:
The HTTP conventions stabilised with renames:
http.method -> http.request.method
http.status_code -> http.response.status_code
net.peer.name -> server.address
http.url -> url.full
Every dashboard, alert and saved query referencing an old name
breaks.
The migration mechanism the project provides:
- a transition period where instrumentation can emit BOTH old
and new names (opt-in via an environment variable), so
dashboards can be migrated without a flag day
- schema_url on the resource, plus a schema transformation
file, so a backend can translate versions
- the Collector's transform processor as the pragmatic tool:
rename attributes in the pipeline while dashboards catch up
Doing the rename in the Collector rather than in applications is the practical answer, and it is a second instance of the same argument for having a Collector at all.
Auto-instrumentation
JVM -javaagent:opentelemetry-javaagent.jar
bytecode instrumentation of ~150 libraries. Zero code
changes. The single highest-return adoption step for
a JVM shop.
Node/Python auto-instrumentation packages, loaded before the app
.NET a similar agent
Go no runtime agent (no bytecode manipulation), so it
needs compile-time wrapping or eBPF
eBPF Beyla, Odigos and similar: instrument at the kernel
boundary, so you get HTTP and gRPC spans with no
application change in any language. Coarser: it sees
the network calls, not your internal spans.
The realistic sequence:
1. auto-instrumentation everywhere, for the map
2. manual spans at the 10 places that matter, for the depth
3. custom attributes on those spans, for the questions you
actually ask
Step 3 is where the value is, and step 1 is what makes step 3
worth doing, because a custom span in an otherwise untraced
system tells you nothing about what happened around it.
The overhead question, answered honestly
Tracing with head sampling at 10%: typically low single-digit
percent CPU, dominated by span creation and attribute
allocation rather than export.
The things that actually cost:
- unsampled tracing at high request rates
- many attributes per span, especially string formatting
done eagerly
- SimpleSpanProcessor (synchronous export)
- auto-instrumentation of very hot, very fine-grained
libraries: a span per Redis call at 50k calls/sec is a lot
of spans
Measure it in your own service rather than trusting a number.
The correct response to unacceptable overhead is usually
narrower instrumentation, not less observability.
A worked example: three vendors, four languages, one migration
A company with 60 services across Java, Go, Python and Node. Metrics in Prometheus, traces in a commercial vendor's agent, logs in a hosted log service. Annual observability spend around $1.4M, of which the tracing vendor was $600k, and a contract renewal in five months.
The starting problem was not cost, it was that nothing correlated:
Debugging a latency regression:
1. Grafana shows p99 up on checkout
2. open the tracing vendor, filter by service and time,
scroll for a slow trace
3. find one, get a trace id
4. search the log service for the trace id -> nothing,
because the logging library never emitted it
5. search by service name and a 30-second window, read
several thousand lines
Measured: median time from "alert fires" to "root cause
identified" was 47 minutes, and 3 of the 5 SREs surveyed said
they usually skipped the trace step because it rarely paid off.
The migration, and the order it was done in:
PHASE 1 (6 weeks): Collector first, applications unchanged.
Deployed a Collector agent DaemonSet and a gateway
deployment. Configured the existing vendor agents to export
OTLP to the Collector instead of directly to the vendor, and
the Collector to forward to the vendor.
Nothing changed functionally. What it bought: a single point
where telemetry could be inspected, redacted, sampled and
re-routed, before any application was touched.
It immediately paid for itself: an attribute containing full
request URLs (including query strings with email addresses in
password-reset links) was found in the gateway and redacted
with a processor change, deployed in 40 minutes. The
alternative had been a 40-service change.
PHASE 2 (1 quarter): auto-instrumentation, language by
language.
Java first, because the javaagent required zero code changes:
28 services in 3 weeks, mostly rollout time rather than work.
Then Node and Python. Go last and slowest, because it needs
compile-time wrapping.
Both the old vendor agent and OpenTelemetry ran in parallel
for 4 weeks per language, with the Collector fanning out to
both backends, so dashboards could be compared side by side
rather than trusted.
PHASE 3 (6 weeks): log correlation, which was the actual goal.
Every logging config gained trace_id and span_id from the
active context. This is a small change per service and it is
the one that changed the debugging workflow.
PHASE 4: sampling and cost.
Head sampling at 100% at the edge, tail sampling in the
gateway keeping: all traces with an error, all traces over
1.5s, and 2% of the rest.
Phase 1 before Phase 2 is the sequencing decision worth copying, because the Collector delivers value on day one with no application changes and no risk, and it makes every later phase reversible.
The tail-sampling arithmetic:
Traffic: ~14,000 requests/sec at peak, ~8 spans per trace.
Head sampling 100%, no tail sampling:
112,000 spans/sec -> the vendor's ingest pricing made this
impossible, which is why the previous setup used 5% head
sampling and therefore missed most errors.
Tail sampling policy:
errors ~0.4% of traces -> all kept
slow (>1.5s) ~1.1% of traces -> all kept
baseline 2% of the rest
effective keep rate ~3.4%
Result: 100% of error traces retained (previously ~5% of them,
by chance), at a lower total volume than the old 5% head
sampling, because the old scheme kept 5% of the boring traces
too.
Keeping every error trace while reducing total volume is the tail-sampling result that surprises people, and it is the strongest argument for a gateway: head sampling has to decide before it knows whether the trace is interesting.
The semantic-convention migration, which was the messiest part:
The vendor's agent emitted its own attribute names. The
OpenTelemetry instrumentation emitted the stable conventions.
217 dashboards and 340 alerts referenced the old names.
What was done:
- the Collector's transform processor duplicated the new
attributes under the old names during the transition, so
nothing broke on day one
- dashboards migrated over a quarter, tracked as a list with
owners
- the duplication was removed at the end, which reduced span
size by about 15%
Attempting the rename as a flag day was proposed and rejected.
The team that had done a similar rename previously reported
three weeks of broken alerting.
Results after two quarters:
before after
median alert-to-root-cause 47 min 11 min
SREs who skip the trace step 3 of 5 0 of 5
error traces retained ~5% 100%
observability spend $1.4M/yr $0.72M/yr
backends 3 vendors 2 (one commercial,
one self-hosted),
switchable by
config
languages instrumented 2 of 4 4 of 4
time to redact a newly
discovered PII attribute ~1 quarter ~40 minutes
The 47-to-11-minute change came from Phase 3, the smallest phase, because putting the trace id in the log line is what makes the spike-to-exemplar-to-trace-to-logs workflow work at all. The expensive phases enabled it; the cheap phase delivered it.
Two things that went wrong:
1. An early Collector gateway deployment had no memory_limiter
processor and no persistent queue. A backend outage caused
the gateway to buffer until it OOMed, in a loop, losing all
telemetry during the incident where it was most needed.
Fixed with memory_limiter first in every pipeline, a bounded
sending queue, and a file-backed queue on the gateway.
2. Baggage was used to propagate a tenant id, and someone
copied all baggage onto spans automatically. Tenant id was
high cardinality. The metrics bill rose 30% in a week before
it was caught by the cardinality alerting that already
existed for exactly this reason.
Both failures are the same lesson: the telemetry pipeline is production infrastructure with its own failure modes, and treating it as a side channel is how it takes down the thing it observes or quadruples a bill.
Production evidence
OpenTelemetry is a CNCF incubating project and, by contributor and commit volume, one of the largest projects in the foundation after Kubernetes. Its API/SDK separation is specified explicitly so that library authors can instrument against a stable API that is a no-op without an SDK.
OTLP is the project's own wire protocol, defined in protobuf, with standard ports 4317 (gRPC) and 4318 (HTTP), and it carries traces, metrics, logs and profiles under a shared resource model.
W3C Trace Context is a W3C Recommendation defining the traceparent and tracestate headers, and
its adoption is what allows traces to cross vendor and organisational boundaries.
Semantic conventions stabilisation and the accompanying rename (for example http.method to
http.request.method) are documented in the project's migration guides, along with the dual-emission
opt-in and schema URL mechanism that exist specifically to make the transition survivable.
The Collector's receiver/processor/exporter pipeline model, agent versus gateway deployment patterns, and the requirement that tail sampling see all spans of a trace (hence the load-balancing exporter keyed on trace id) are all documented in the Collector's own deployment guidance.
Vendor support for OTLP ingest across Datadog, New Relic, Honeycomb, Splunk, Grafana, AWS, Google Cloud and Azure is the practical evidence for the portability claim: the same instrumentation can be routed to any of them by configuration.
The debate
Is OpenTelemetry worth the migration? For a multi-language estate with more than one backend or a renewal coming, clearly. For a single-language shop happy with one vendor, the honest answer is that the benefit is optionality rather than capability, and optionality has a real price: the project moves fast, conventions have churned, and per-language maturity varies.
Agent, gateway, or both? Both, for anything non-trivial. Agent-only means no tail sampling and no central redaction; gateway-only means the application's export path depends on a network hop to a shared service. The cost is another deployment to operate, and it is real: an under-provisioned gateway drops telemetry precisely during the incident that generated the most of it.
Head or tail sampling? Head at the edge for volume control, tail in the gateway for the decisions that need the whole trace. The argument for tail is decisive on errors: head sampling cannot keep all error traces because it decides before the error happens. The cost is that the gateway must be stateful, must buffer entire traces, and must route consistently by trace id.
Should you use baggage? Sparingly, and never copy it wholesale onto spans or metrics. It propagates to every downstream service and straight into cardinality, which is a bill that arrives a week later. Tenant id, criticality and experiment cohort are the legitimate cases, and each one should be a deliberate decision.
Are semantic conventions worth conforming to? Yes, and the churn is a genuine cost that people under-report. The compensating practice is to do renames in the Collector rather than in applications, which turns a 40-service change into a config rollout and is the same argument as the redaction case.
Does auto-instrumentation replace manual spans? No. It produces the map; manual spans and custom attributes answer the questions you actually have. The failure in each direction is real: only auto-instrumentation gives you traces that show which service was slow and never why, and only manual spans gives you islands with no surrounding context.
Follow-up Q&A
"What is the point of separating the API from the SDK?"
So that libraries can instrument themselves without committing anyone to anything. The API is a no-op unless an application configures an SDK, so a library depending on it adds no runtime behaviour and no backend dependency, and the application alone decides whether telemetry is recorded and where it goes. That is what makes upstream instrumentation possible at all, which no vendor SDK ever achieved, and it is the structural reason the ecosystem's instrumentation is shared rather than duplicated per vendor.
"Why run a Collector at all if applications can export directly?"
Because it is the place where you can change behaviour without redeploying applications. Discovering that an attribute contains email addresses becomes a processor change deployed in forty minutes rather than a forty-service change over a quarter. Dropping a high-cardinality attribute that just tripled the bill is a config rollout. Migrating backends is fan-out to two exporters for a period. And it can queue through a backend outage rather than dropping telemetry at the application. It also enables tail sampling, which requires seeing whole traces.
"Where do head and tail sampling belong, and why?"
Head sampling in the SDK, because it controls volume at the source and costs nothing. Tail sampling in a gateway Collector, because the decision requires the whole trace and the application only ever sees its own spans. The decisive argument is errors: head sampling has to decide before the error happens, so it keeps error traces only by chance. In one migration a tail policy of all errors, all traces over 1.5 seconds, and 2 percent of the rest retained 100 percent of error traces at a lower total volume than the previous 5 percent head sampling, because that scheme also kept 5 percent of the boring traces.
"What are the most common OpenTelemetry misconfigurations?"
Missing service.name, so everything arrives as unknown_service. SimpleSpanProcessor in
production, which exports synchronously on span end and adds export latency to every request. An
unbounded batch queue, so a slow exporter grows memory until the process dies. And no memory_limiter
as the first processor in the Collector pipeline, which is how a backend outage turns into a gateway
OOM loop that loses telemetry during exactly the incident that produced it. The principle underneath
all four: telemetry is best-effort and must fail open.
"How do you handle the semantic-convention renames?"
In the Collector, with a transform processor that duplicates new attribute names under the old ones during a transition, so nothing breaks on the day instrumentation changes. Then migrate dashboards and alerts over a quarter as a tracked list with owners, and remove the duplication at the end, which also reduces span size. The alternative, a flag day rename across hundreds of dashboards and alerts, is what produces weeks of broken alerting. The project also supports dual emission from instrumentation and schema URLs for backend-side translation.
"What is baggage and what is the risk?"
Application key-values propagated with the request alongside trace context: tenant id, experiment cohort, request criticality. The risk is that it travels to every downstream service, and if anything copies baggage onto spans or metric attributes automatically, high-cardinality values go straight into your storage bill. In one case a tenant id propagated as baggage and copied onto spans raised the metrics bill 30 percent in a week. Use it deliberately, for a small named set, and never copy it wholesale.
Common misconceptions
"OpenTelemetry is a tracing system." It is a specification and toolkit for producing and routing traces, metrics, logs and profiles. It stores and queries nothing.
"Adopting it means choosing a backend." It means the backend becomes a Collector config, which is the point.
"The SDK is what libraries use." Libraries use the API, which is a no-op without an SDK. If a library pulls in an SDK, that is a bug.
"Sampling is a single decision." Head sampling controls volume at the source; tail sampling makes decisions that require the whole trace. They serve different purposes and are configured in different places.
"Auto-instrumentation is enough." It gives you the map. Custom spans and attributes answer the questions you actually ask, and they are only useful because the map exists around them.
"Telemetry overhead is the reason not to instrument." The usual causes of unacceptable overhead are synchronous export, unsampled high-rate tracing, and per-call spans on very hot libraries, all of which are configuration choices rather than properties of the standard.
Interview delivery note
Say this verbatim: "The API/SDK split is the whole design: a library instruments against the API, which is a no-op unless the application wires an SDK, so libraries can ship instrumentation without committing anyone to a backend. And the Collector is where you change behaviour without redeploying anything, which is why finding PII in an attribute becomes a forty-minute config change instead of a forty-service deploy." Two structural properties and the concrete thing each buys.
The senior-versus-staff separator is deploying the Collector before touching any application. A senior engineer instruments services and points them at a backend. A staff engineer puts the Collector in first, in front of the existing vendor agents, so that redaction, sampling, routing and fan-out become available on day one with no application change and no risk, which makes every later phase reversible and pays for itself before the migration proper begins.
The second signal is knowing why tail sampling cannot live in the SDK. Saying "head sampling decides before the error happens, so it keeps error traces only by chance; a tail policy of all errors plus all slow traces plus two percent kept every error trace at lower total volume than the previous five percent head sampling" demonstrates you understand both the mechanism and its cost, which is a stateful gateway that must route all spans of a trace to the same instance.
Further reading
- The OpenTelemetry specification's API and SDK sections, for the no-op-by-default contract.
- W3C Trace Context, for the
traceparentandtracestateheader format. - OpenTelemetry Collector deployment documentation on agent versus gateway modes and the load-balancing exporter required for tail sampling.
- The OpenTelemetry semantic conventions and their migration guides, including dual emission and schema URLs.
- The observability pipeline page in this chapter, for the annotated Collector configuration and processor ordering, and the cardinality page for what baggage can cost you.