Observability & Operations

Distributed Tracing

Tracing the path of requests across distributed services: trace and span structure, W3C TraceContext propagation, head vs tail-based sampling strategies, Jaeger/Zipkin/Tempo backends, and trace-based testing.

⏱ 11 min read

What it is

Distributed tracing records the end-to-end journey of a request as it propagates through a distributed system. A trace is a collection of spans that share a common trace_id. A span represents a single unit of work: it has a name, a start time, a duration, a set of key-value attributes (metadata), zero or more events (timestamped within the span), and a status (OK, ERROR). Spans form a tree: each span has a parent span (except the root span that represents the originating request), creating a directed acyclic graph of the complete request path. The trace visualised as a Gantt chart — the "flame graph" or "trace waterfall" — shows exactly where time was spent across all services and operations.

Context propagation is the mechanism that connects spans across service boundaries. When Service A calls Service B, it injects the current trace context (trace_id, span_id, sampling decision) into the outgoing request headers. Service B extracts this context and creates a child span under the same trace. The W3C TraceContext standard (RFC 7230) defines two HTTP headers: traceparent (e.g., 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01, encoding version, trace_id, parent_id, flags) and tracestate (vendor-specific metadata). Using the W3C standard rather than proprietary headers (Zipkin's X-B3-TraceId, AWS X-Ray's X-Amzn-Trace-Id) enables interoperability across different tracing systems and languages.

Sampling strategies determine which traces are actually recorded and stored. At high request volumes, storing every trace is prohibitively expensive. Head-based sampling makes the sampling decision at the root span before any downstream work is done: a percentage (e.g., 10%) of traces are sampled uniformly. Simple and low-overhead, but blindly samples errors and slow requests at the same rate as normal requests. Tail-based sampling buffers spans until the trace is complete, then makes the sampling decision based on the full trace — retaining all error traces and all traces above a latency threshold while sampling down normal traces. Tail-based sampling requires a stateful Collector (OTel Collector tail_sampling processor) but provides much higher-value trace retention.

Why it exists

The 2010 Google Dapper paper described the original distributed tracing system, motivated by a simple problem: in a system where a single user search touches 100+ services, how do you understand why a specific request was slow? Logs from 100 services with no common identifier cannot be correlated. Metrics show aggregate latency but not the per-request breakdown across services. Distributed tracing was invented to answer "why was request X slow?" at a per-request level. The problem became ubiquitous as microservices spread — Netflix, Twitter, and Uber all built their own tracing systems before open standards (OpenZipkin 2012, Jaeger 2016, OpenTelemetry 2019) enabled common implementations.

When to use

  • Any system with two or more services handling a single user request — the cross-service latency breakdown is essential for debugging.
  • Systems where latency SLOs must be met — traces provide the granular latency breakdown needed to find and fix the slow components.
  • When investigating production incidents where logs alone are insufficient to identify the root cause across service boundaries.

When not to use

  • Monolithic applications with a single process — standard profiling (CPU/memory) and structured logs provide sufficient debugging data without the overhead of trace infrastructure.

Typical architecture

REQUEST FLOW WITH TRACING:

Client → API Gateway → Order Service → Payment Service
                             ↓
                      Inventory Service

TRACE STRUCTURE:
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736

Spans:
├── api-gateway: POST /checkout          [0ms - 450ms]
│   ├── order-service: processOrder     [10ms - 430ms]
│   │   ├── inventory-service: check    [20ms - 80ms]
│   │   │   └── db: SELECT inventory    [25ms - 75ms]
│   │   └── payment-service: charge     [90ms - 420ms]
│   │       └── stripe-api: create      [95ms - 415ms]  ← bottleneck
│   └── (serialization)                 [430ms - 450ms]

W3C TRACEPARENT HEADER PROPAGATION:
  API Gateway sends to Order Service:
    traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-span1-01

  Order Service sends to Payment Service:
    traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-span3-01
    (same trace_id, new parent span_id, flag=01=sampled)

OTEL COLLECTOR TAIL SAMPLING CONFIG:
  processors:
    tail_sampling:
      decision_wait: 10s
      policies:
        - name: errors
          type: status_code
          status_code: {status_codes: [ERROR]}
        - name: slow-traces
          type: latency
          latency: {threshold_ms: 500}
        - name: probabilistic-normal
          type: probabilistic
          probabilistic: {sampling_percentage: 5}

BACKENDS:
  Grafana Tempo: cost-effective, object-storage backed
    (S3/GCS), excellent Grafana integration
  Jaeger:  open-source, Cassandra/ES/memory backends
  Zipkin:  open-source, simpler, B3 propagation

Pros and cons

Pros

  • Traces provide a unique debugging capability: the exact latency breakdown per service and operation for a specific request, enabling root-cause identification in minutes rather than hours of log analysis.
  • Span attributes (HTTP method, DB query, user ID, feature flag) make traces rich debugging artifacts — each span is a structured record of what that operation did.
  • Traces enable service dependency mapping automatically — by aggregating span parent-child relationships, the actual runtime call graph of the system is visible without manual documentation.

Cons

  • Distributed tracing requires instrumentation at every service boundary; a single un-instrumented service breaks the trace chain for any request that passes through it.
  • Context propagation across async systems (message queues, batch jobs, event-driven architectures) requires explicit implementation — context must be serialised into message metadata.
  • Storage costs for full-fidelity traces at production scale are significant; tail-based sampling adds Collector complexity and the stateful buffering window introduces latency before traces are available for search.

Implementation notes

Start with OpenTelemetry auto-instrumentation — for Java, Node.js, Python, and Go, the OTel SDK provides auto-instrumentation of HTTP clients/servers, gRPC, SQL, Redis, and common frameworks with zero or minimal code changes. This provides 80% of the trace value immediately. Then add manual spans for business-critical operations (payment processing, order state machine transitions) that add semantic context beyond what HTTP/DB instrumentation provides.

For Kafka/RabbitMQ/SQS context propagation: inject trace context into message headers at producer time using the OTel messaging propagator, and extract it at consumer time to create a child span with a link to the producer span. OTel uses span links (rather than parent-child) for async messaging to represent the causal relationship without implying synchronous nesting. Grafana Tempo with S3 backend is the recommended cost-effective trace backend for most teams — it charges only for storage (S3 rates) rather than ingestion, making it dramatically cheaper than commercial APM at high trace volumes.

Common failure modes

  • Missing context propagation in async messaging: Producer creates trace, consumer starts a new disconnected trace — the async boundary breaks the trace chain, making end-to-end tracing impossible through the queue.
  • Head sampling discarding all error traces: 1% head sampling also samples 1% of errors; errors need 100% sampling rate since they are the primary debugging target.
  • Sensitive data in span attributes: Automatically instrumented SQL spans capture full query text, which may include PII from parameter values; configure OTel DB instrumentation to sanitise or redact query parameters.
  • Clock skew between services: Span timestamps from different services with unsynchronised clocks produce trace timelines where child spans appear to start before parent spans — always ensure NTP is synchronised (NTP drift < 1ms) across all services.

Decision checklist

  • Is auto-instrumentation deployed for all services (HTTP, gRPC, DB)?
  • Is W3C TraceContext propagation used consistently across all services?
  • Is context propagation implemented in message producers and consumers?
  • Is tail-based sampling configured to retain 100% of error and slow traces?
  • Are span attributes enriched with business context (user ID, order ID, feature flag)?
  • Is a trace backend deployed with sufficient retention for P1 incident investigation?

Example use cases

  • Latency regression investigation: p99 latency increased from 200ms to 800ms after a deploy; search traces for high-latency requests; all slow traces show a new N+1 query pattern in the product listing service introduced by the latest deployment; query batched and latency restored.
  • Microservices dependency discovery: New team joining a complex system uses the service map generated from trace data (aggregated span parent-child relationships) to understand actual runtime dependencies between 40 services — more accurate than outdated architecture documentation.
  • Trace-based testing: Integration tests emit traces; test assertions verify that a specific operation (checkout) produces the expected trace structure (payment span with correct attributes), catching regressions in trace instrumentation alongside functional regressions.

Further reading