Communication PatternsObservabilityEssential

Correlation IDs

Generate a unique identifier at the system entry point and propagate it through every synchronous call and asynchronous message, enabling end-to-end request tracing across distributed services and log correlation in production.

⏱ 10 min read

What it is

A correlation ID is a unique identifier — typically a UUID v4 — generated at the entry point of a request (the API gateway, the first microservice that handles the user action, or the event source) and threaded through every downstream call, message, and log line that forms part of the same logical operation. When something goes wrong anywhere in the chain, engineers can filter logs and traces by this single ID to reconstruct the exact sequence of events across all services, queues, and databases involved in the operation.

The correlation ID is not the same as a distributed trace ID, though they serve related purposes. A trace ID (W3C Trace Context traceparent, Zipkin traceId) is typically generated and managed by a distributed tracing library (OpenTelemetry, Jaeger) and is optimised for latency measurement and span tree reconstruction. A correlation ID is a simpler, business-level identifier you control, suitable for correlating log lines, error reports, audit trails, and async message chains where tracing libraries may not be present.

Why it exists

In a microservices architecture, a single user-visible operation might touch 5–15 services. Each service writes logs independently. Without a shared identifier, reconstructing "what happened for order #12345" requires timestamp-based guessing, service-specific IDs that only some services know about, and manual log stitching — a debugging nightmare in production under time pressure. Correlation IDs make this trivially filterable: grep correlationId=abc-123 /var/log/*.log or a Kibana query fields.correlationId: "abc-123" returns every log line from every service involved in that operation.

When to use

Any system with more than one service — even two-service architectures benefit from log correlation.
Asynchronous pipelines — propagate correlation IDs through message envelopes so the async consumer's log lines are linked to the originating HTTP request.
Long-running workflows — a correlation ID started when a user submits an order should appear in fulfilment, shipping, and notification service logs days later.
Customer support investigations — surface the correlation ID in error responses (safely — not in user-visible UI) so support teams can use it to retrieve the full operation context.

When not to use

There are no scenarios where correlation IDs should be entirely omitted in a distributed system. Even single-service systems benefit from request-level IDs for log filtering.
Do not use correlation IDs as security tokens — they are not secrets; anyone who sees them can use them for log lookup, which is intentional. Never use them to authorise access to data.

Typical architecture

The API gateway or load balancer generates a correlation ID for every incoming request and injects it as an HTTP header. Services read the header, bind it to their logging context (MDC in Java, context variables in Python), include it in all outbound HTTP calls, and embed it in all outbound messages. Logs are structured JSON that always include the correlation ID field. Log aggregation (ELK, Loki, CloudWatch Logs Insights) enables cross-service filtering by correlation ID.

Client → API Gateway
  Gateway generates: correlationId = "550e8400-e29b-41d4-a716-446655440000"
  Injects header: X-Correlation-ID: 550e8400-...

API Gateway → Service A
  Header: X-Correlation-ID: 550e8400-...
  Service A logs: {"correlationId":"550e8400-...","msg":"Processing order","orderId":"12345"}

Service A → Message Queue (async)
  Message envelope:
  {
    "correlationId": "550e8400-...",
    "causationId": "request-id-of-the-call-that-triggered-this-message",
    "payload": { "orderId": "12345" }
  }

Service B (queue consumer) logs:
  {"correlationId":"550e8400-...","msg":"Reserving inventory","orderId":"12345"}

Service B → Service C (HTTP)
  Header: X-Correlation-ID: 550e8400-...
  Service C logs: {"correlationId":"550e8400-...","msg":"Charging payment","orderId":"12345"}

Pros and cons

Pros

Dramatically reduces mean time to diagnose (MTTD) in production incidents.
Works across sync and async boundaries without tracing library instrumentation.
Enables customer support to retrieve the exact context for a reported error.
Zero performance impact — a UUID stored in a logging context is negligible overhead.

Cons

Requires consistent adoption — one service that drops the header breaks the chain and makes cross-service correlation impossible from that hop onward.
Header name fragmentation — different teams may use X-Correlation-ID, X-Request-ID, X-Trace-ID, causing propagation failures at service boundaries.
Does not replace distributed tracing — correlation IDs show what happened in logs; distributed traces show latency and dependency trees.

Implementation notes

Generating correlation IDs: Generate a UUID v4 (crypto.randomUUID() in Node.js, uuid.uuid4() in Python, UUID.randomUUID() in Java) at the entry point — the API gateway or the first service that handles the request. Never reuse or derive a correlation ID from user-visible data (like the order ID) because that creates coupling and potential collision.

HTTP header conventions: Use X-Correlation-ID as the primary header name (widely recognised) or adopt W3C Trace Context's traceparent header if your tracing library supports it. Standardise the header name across your entire organisation with a platform-level decision. Accept both the incoming header (if present, use it; the caller may be propagating an existing correlation) and generate a new one if absent.

Async propagation in message envelopes: For async messaging, embed the correlation ID in the message envelope's metadata, not the payload. This allows consumers to extract it without deserialising the full payload. Additionally include a causationId — the ID of the specific command or request that caused this message to be published — to reconstruct the causation chain (who caused what) separately from the overall correlation chain (what belongs to the same operation).

Correlation ID vs Trace ID distinction: A correlation ID spans business operations (one user interaction = one correlationId, even if it takes 3 days to complete). A trace ID spans a single synchronous request tree and is managed by the tracing library. Use both: the correlation ID for cross-day, cross-async-boundary log search; the trace ID for latency analysis and dependency graphs in your APM tool (Jaeger, Zipkin, Datadog APM).

W3C Trace Context: If your services use OpenTelemetry, the traceparent header carries the trace ID and span ID in a standardised format: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. This is interoperable across instrumented services and avoids the header fragmentation problem. However, it requires OpenTelemetry SDK integration in every service — for polyglot environments, a simpler X-Correlation-ID may be easier to adopt universally.

Common failure modes

Service drops the header: A service does not read or forward X-Correlation-ID; all downstream service logs are uncorrelated. Solution: add middleware/interceptor at the framework level so propagation is automatic.
Service generates a new ID instead of forwarding: Each service generates its own correlation ID rather than reading the incoming one; every hop has a different ID. Solution: read the incoming header first; only generate if absent.
Correlation ID not included in structured log: Services log in plaintext without a correlation ID field; filtering is impossible. Solution: enforce structured JSON logging with correlation ID as a mandatory top-level field via logging policy.
Correlation ID exposed in error responses to end users: While not a security risk per se, surfacing internal IDs in user-facing error messages can be confusing. Expose them in internal dashboards and support tools, not in user-facing UI error messages.

Decision checklist

Is a correlation ID generated at every external entry point (API gateway, event source)?
Is the correlation ID header name standardised across all services in the organisation?
Does every service read the incoming correlation ID header and bind it to the logging context before writing any log lines?
Does every service forward the correlation ID on all outbound HTTP calls and embed it in all outbound message envelopes?
Are all logs structured JSON with correlation ID as a top-level field (not nested or omitted)?
Is a causation ID also propagated for async message chains?
Are correlation IDs included in error responses to internal callers (for support tooling) but not leaked in user-facing error messages?

Example use cases

Production incident investigation: A customer reports that their payment failed at 14:32. Support retrieves the correlation ID from the error log, filters all 12 service logs by that ID, and within 2 minutes reconstructs the exact failure — a downstream tax service returned a 503 that was not retried.
Async audit trail: An order confirmation email sent 5 minutes after checkout shares the same correlation ID as the original checkout request, enabling a complete audit trail from user action to notification.
Fan-out correlation: A single checkout request fans out to 4 services; all 4 log lines share the same correlation ID, enabling the complete fan-out/fan-in trace to be reconstructed from logs alone.

Fan-Out / Fan-In — correlation IDs are essential for matching fan-in responses to their originating fan-out.
Message Routing — correlation IDs allow tracing a message's path through multiple routing hops.
Transactional Outbox Pattern — the outbox message envelope must carry the correlation ID for end-to-end traceability.