OpenTelemetry

What it is

OpenTelemetry (OTel) is a CNCF project that provides a vendor-neutral API, SDK, and Collector for generating, collecting, and exporting observability telemetry — traces, metrics, and logs. Before OTel, each observability vendor (Jaeger, Zipkin, Datadog, New Relic) had proprietary instrumentation libraries; switching vendors required re-instrumenting applications. OTel standardises instrumentation: applications instrument once against the OTel API, and the OTel Collector routes telemetry to any backend through configurable exporters. This decouples instrumentation from observability vendor choice.

The OTel architecture has three layers: (1) OTel API — the application-facing interface for creating spans, recording metrics, and emitting log records. Applications depend on the API. The API is intentionally minimal and stable — it is safe to add as a library dependency. (2) OTel SDK — the implementation that processes telemetry. It adds context propagation, sampling decisions, batching, and export. The SDK is configured by operators, not developers. (3) OTel Collector — an optional but strongly recommended intermediary between applications and backends. It receives telemetry over OTLP (OTel Protocol), processes it (batching, filtering, enrichment, sampling), and exports to one or more backends. The Collector decouples applications from backend configuration and enables centralised telemetry pipeline management.

Auto-instrumentation adds OTel instrumentation to common libraries (HTTP clients/servers, database clients, message consumers) without requiring application code changes. It works via bytecode manipulation (Java agent), monkey-patching (Python), or LD_PRELOAD (Go). Auto-instrumentation covers the 80% case — standard library calls — and should be combined with manual instrumentation for business-level context (adding attributes like user.id, order.amount, payment.method to spans) that automatic instrumentation cannot provide. The combination of auto + manual instrumentation produces traces with both technical (latency, errors, SQL query) and business (user context, operation metadata) dimensions.

Why it exists

OpenTelemetry emerged from the merger of OpenTracing and OpenCensus, two competing vendor-neutral tracing standards. It is now the second-most contributed CNCF project (after Kubernetes). The core problem it solves is vendor lock-in and instrumentation fragmentation: organisations with 100 services using proprietary vendor SDKs face prohibitive switching costs. With OTel, the instrumentation investment (spans, metrics) is portable — changing observability backends requires only Collector configuration changes, not application changes. The OTLP protocol is now natively supported by virtually every observability platform (Datadog, New Relic, Grafana, Honeycomb, Jaeger), making OTel the de facto standard for new observability instrumentation.

When to use

• All new services — OTel should be the default instrumentation choice for any new service requiring observability telemetry.
• Migrating from proprietary instrumentation — OTel provides a clear migration path with Collector compatibility shims for legacy formats (Jaeger, Zipkin, StatsD).
• Multi-vendor observability — routing the same telemetry to multiple backends (e.g., traces to both Jaeger and Datadog) is trivial with the Collector.

When not to use

• Services already well-instrumented with a stable vendor SDK where migration cost outweighs portability benefit — evaluate per service rather than mandating immediate migration of all existing instrumentation.

Typical architecture

OPENTELEMETRY ARCHITECTURE:

Application (OTel SDK)
    ↓ OTLP (gRPC or HTTP)
OTel Collector (DaemonSet or Sidecar)
    ↓ OTLP/Jaeger/Prometheus/etc.
Observability Backends (Jaeger, Prometheus, Loki)

COLLECTOR PIPELINE (otel-collector-config.yaml):
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
  resourcedetection:
    detectors: [env, k8s_node]
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: sample-10pct
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

exporters:
  otlp:
    endpoint: tempo:4317
    tls: {insecure: true}
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

MANUAL INSTRUMENTATION (Go):
  tracer := otel.Tracer("checkout-service")
  ctx, span := tracer.Start(ctx, "process-payment",
    trace.WithAttributes(
      attribute.String("payment.method", "card"),
      attribute.Float64("payment.amount", order.Total),
      attribute.String("user.id", userID),
    ))
  defer span.End()

AUTO-INSTRUMENTATION (Java agent):
  java -javaagent:opentelemetry-javaagent.jar \
       -Dotel.service.name=checkout-api \
       -Dotel.exporter.otlp.endpoint=http://collector:4317 \
       -jar app.jar

SEMANTIC CONVENTIONS (standardised attribute names):
  http.request.method: "GET"
  http.response.status_code: 200
  url.path: "/api/orders"
  db.system: "postgresql"
  db.statement: "SELECT * FROM orders WHERE id=$1"
  messaging.system: "kafka"
  messaging.destination.name: "orders"

Pros and cons

Implementation notes

Deploy the OTel Collector as a Kubernetes DaemonSet (one per node) for traces and metrics collection, with a centralised Collector deployment for the Collector tier that handles tail-based sampling, backend routing, and enrichment. This two-tier architecture keeps application-side overhead low (applications send to a local DaemonSet over loopback) while centralising the expensive processing (tail sampling decisions require seeing all spans for a trace). Configure memory_limiter processor as the first processor in every pipeline — this prevents OOM kills under traffic spikes by dropping telemetry when memory pressure is high, which is preferable to the Collector crashing entirely.

When adding manual instrumentation, focus on business-level attributes rather than duplicating what auto-instrumentation already captures. Auto-instrumentation covers HTTP request/response, database query, and messaging system attributes. Manual instrumentation should add domain attributes: user identity, business transaction IDs, feature flag states, and domain-specific error categorisation. Use the baggage API to propagate business context across service boundaries automatically — once set in the entry point, baggage values propagate through all downstream spans without requiring explicit passing through function signatures.

Common failure modes

• Collector as SPOF: A single Collector deployment without redundancy becomes a single point of failure for all telemetry; deploy at least 2 Collector replicas behind a service for high availability.
• Missing resource attributes: Spans missing service.name, service.version, or deployment.environment are difficult to filter and correlate; configure resource attributes at the SDK level and supplement with Collector resourcedetection processor.
• No tail sampling: Exporting 100% of traces at high traffic volumes creates prohibitive storage and cost; configure tail-based sampling in the Collector to retain high-value traces (errors, slow) while sampling the rest.
• Baggage misuse: Adding sensitive data (PII, authentication tokens) to OTel baggage propagates them to all downstream services and into trace data; baggage should contain only non-sensitive correlation identifiers.

Decision checklist

• Is the OTel Collector deployed with high availability (at least 2 replicas)?
• Is memory_limiter processor configured as the first processor in all pipelines?
• Are tail-based sampling policies configured to retain errors and slow traces?
• Are semantic convention attribute names used for standard operations (HTTP, DB, messaging)?
• Are resource attributes (service.name, service.version, environment) set for all services?
• Is auto-instrumentation combined with manual instrumentation for business context?

Example use cases

• Vendor migration without re-instrumentation: Organisation migrates from Datadog to Grafana stack; all services already instrument with OTel SDK; Collector exporter configuration updated to point to Grafana Tempo instead of Datadog; no application code changes required; migration completed in 1 week vs estimated 3 months for re-instrumentation.
• Centralised tail sampling: Service handling 50,000 requests/second; 100% trace export is cost-prohibitive at $500K/year; Collector tail sampling configured to retain 100% of error traces, 100% of traces >2s, and 5% of normal traces; trace volume reduces 90%; critical debugging data preserved at $50K/year cost.
• Multi-backend fan-out: Security team requires a copy of all spans for security analysis (UEBA); ops team uses Tempo for performance investigation; Collector configured to export to both Tempo and a security SIEM simultaneously; zero application code changes required to add the second destination.

Related patterns

• Distributed Tracing — OTel is the standard instrumentation layer for distributed traces.
• Metrics and Monitoring — OTel metrics API replaces Prometheus client libraries for new services.
• Three Pillars of Observability — OTel covers all three pillars: traces, metrics, and logs.

Further reading

• OpenTelemetry Official Documentation
• OTel Collector Documentation
• OpenTelemetry Semantic Conventions