Observability & Operations Essential

Three Pillars of Observability

Metrics, logs, and traces — the three complementary telemetry types that together enable understanding of distributed system behaviour. How OpenTelemetry unifies them, and the critical difference between observability and monitoring.

⏱ 10 min read

What it is

Observability is the ability to understand the internal state of a system from its external outputs — specifically, to ask arbitrary questions about system behaviour without deploying new instrumentation. The term originates in control theory: a system is observable if its internal state can be determined from its outputs. Applied to software systems, observability is achieved through three complementary telemetry types:

  • Metrics: Numerical measurements aggregated over time (request rate, error rate, CPU usage, latency percentiles). Metrics are cheap to store (numbers), queryable for dashboards and alerting, but lossy — they aggregate away the details needed for root-cause analysis.
  • Logs: Timestamped, discrete records of events. Logs are the highest-fidelity telemetry type — they capture the full context of each event — but expensive at scale (volume, storage, indexing) and hard to correlate across services.
  • Traces: Records of the end-to-end path of a request as it propagates through distributed services. A trace is a directed acyclic graph of spans (units of work with start/end time, attributes, and events). Traces answer "why is this request slow?" by showing exactly where time was spent across service boundaries.

The MELT model (Metrics, Events, Logs, Traces) extends the three pillars with Events — significant discrete occurrences like deployments, feature flag changes, and scaling events — that provide context for correlating observability signals with system changes. OpenTelemetry (OTel) is the CNCF standard for producing, collecting, and exporting all three telemetry types using a single, vendor-neutral SDK, with a Collector component that receives, processes, and exports telemetry to any backend (Prometheus, Jaeger, Loki, Datadog, etc.).

Observability vs monitoring: Monitoring asks "is this known thing broken?" using predefined dashboards and alerts for known failure modes. Observability allows asking "why is this behaving unexpectedly?" for failure modes not anticipated in advance — which is the dominant category in complex distributed systems. Monitoring is necessary but not sufficient; observability is the property that enables effective debugging of novel failures.

Why it exists

Distributed microservice architectures created a fundamental problem: a single user request may traverse dozens of services and produce no useful error message — just a degraded user experience. Traditional monitoring (CPU thresholds, ping checks) cannot locate the root cause of a latency increase caused by a noisy neighbour in service 7 of 12 in the call chain. The three-pillar model emerged from practitioner experience (notably Charity Majors and the Honeycomb team) that no single telemetry type is sufficient for effective debugging: metrics detect a problem, logs provide event-level detail, and traces correlate that detail across service boundaries.

When to use

  • All production systems benefit from at least metrics and structured logging as a minimum viable observability stack.
  • Distributed systems with multiple services — traces are essential for understanding cross-service latency and error propagation.
  • Teams adopting SLO-based reliability practices — metrics are the foundation for SLI measurement and SLO alerting.

When not to use

  • Full three-pillar observability with head-based 100% sampling is cost-prohibitive at very high request volumes; use tail-based sampling to retain traces for slow and error requests while sampling down normal requests.

Typical architecture

APPLICATION INSTRUMENTATION (OpenTelemetry SDK):
  - Metrics: auto-instrumented HTTP, DB, gRPC
    + custom business metrics (order count, etc.)
  - Traces: auto-instrumented context propagation
    + manual spans for business-critical operations
  - Logs: structured JSON with trace_id injected
    for correlation with active trace

OPENTELEMETRY COLLECTOR:
  Receivers:     OTLP (gRPC/HTTP), Prometheus, Fluent Bit
  Processors:    Batch, Attributes (add k8s metadata),
                 Tail sampling (keep errors/slow traces)
  Exporters:     Prometheus (metrics) → Prometheus/Thanos
                 OTLP (traces)  → Grafana Tempo / Jaeger
                 OTLP (logs)   → Grafana Loki

BACKENDS:
  Metrics:  Prometheus + Grafana (dashboards, alerting)
  Traces:   Grafana Tempo (cost-effective trace storage)
  Logs:     Grafana Loki (log aggregation, label-indexed)

CORRELATION:
  - trace_id in log lines enables log-to-trace pivot
  - exemplars on Prometheus metrics link to trace_id
  - Grafana "Explore" view: metrics → traces → logs
    in a single investigation workflow

SIGNAL CORRELATION WORKFLOW:
  1. Alert fires on high p99 latency (metric)
  2. Open Grafana dashboard → see latency spike
  3. Click exemplar on metric → open slow trace
  4. Trace shows: 800ms in UserService DB query span
  5. Click trace span → open correlated logs for span
  6. Log line: "Full table scan on users table"

Pros and cons

Pros

  • The three signals are complementary — each answers questions the others cannot: metrics for "what is happening", logs for "what happened in detail", traces for "where is the problem across service boundaries".
  • OpenTelemetry provides vendor-neutral instrumentation; switching observability backends does not require re-instrumenting application code.
  • Trace IDs injected into log lines enable powerful correlation — jumping from a metric spike to a trace to the specific log line that explains the error is a dramatically faster debugging workflow than unstructured log search.

Cons

  • Implementing all three pillars with correlation requires upfront investment — OTel SDK integration, Collector deployment, multiple backends, Grafana datasource configuration.
  • High-cardinality labels in metrics (e.g., user_id per metric) can cause Prometheus cardinality explosions that degrade query performance and increase storage.
  • Distributed tracing requires context propagation to work correctly across async messaging systems — trace context must be explicitly carried in message headers, which requires instrumentation of producers and consumers.

Implementation notes

Instrument with OpenTelemetry from the start rather than vendor SDKs — auto-instrumentation for common frameworks (Spring Boot, Express, Django, gRPC) requires minimal code changes and provides traces, metrics, and context propagation automatically. For Java, the OTel Java agent provides zero-code-change auto-instrumentation via JVM agent attachment. The key configuration decision is the sampling strategy: use head-based sampling (sample 10% uniformly) for cost control during initial rollout, then switch to tail-based sampling in the Collector (sample 100% of error traces and slow traces, sample down normal traces) for better debugging coverage at lower cost.

For the correlation to work across all three signals, ensure every log line includes the current trace_id and span_id — OTel SDK provides these in the trace context and most logging frameworks have OTel integration that injects them automatically into structured log output. Choose a Grafana stack (Prometheus + Loki + Tempo) for a unified open-source observability platform with native cross-signal correlation, or a commercial platform (Datadog, Honeycomb, New Relic) that provides this out of the box at the cost of vendor lock-in.

Common failure modes

  • Unstructured logs: Free-text logs without trace_id cannot be correlated with traces; search requires full-text search over large log volumes rather than a direct trace_id lookup.
  • High-cardinality metrics labels: Adding user_id, session_id, or request_id as metric labels creates one time series per unique value — millions of time series that degrade Prometheus performance. Cardinality belongs in traces and logs, not metrics.
  • Missing context propagation: Async message consumers that don't extract and restore trace context from message headers create trace gaps — traces end at the producer and restart at the consumer with no link, making it impossible to trace a request end-to-end through a queue.
  • Sampling all traces equally: Sampling 1% of all traces means sampling 1% of error traces — errors are rare to begin with, and sampling them away makes debugging errors nearly impossible.

Decision checklist

  • Are all three telemetry types (metrics, logs, traces) instrumented?
  • Are trace IDs injected into structured log output for correlation?
  • Is OpenTelemetry used as the instrumentation standard?
  • Does sampling strategy preserve 100% of error and slow traces?
  • Is context propagation implemented across async message boundaries?
  • Is there a unified platform for correlating signals during incident investigation?

Example use cases

  • E-commerce checkout slowdown: Metric alert fires on p99 checkout latency; engineer opens Grafana, finds trace exemplar on the spike, opens trace showing 1.2s spent in payment-service → inventory-service gRPC call; correlated log shows "inventory DB connection pool exhausted"; fix: increase pool size.
  • Microservices migration: Team migrating from monolith to microservices instruments all new services with OTel from day one; unified Grafana dashboard tracks all services; cross-service tracing reveals unexpected dependencies that would have been invisible without traces.
  • SRE on-call investigation: On-call engineer receives PagerDuty alert for elevated error rate; opens traces for erroring requests; all errors share a common span attribute db.host=db-replica-2; identifies failed replica as root cause in under 3 minutes.

Further reading