Log Aggregation

What it is

Log aggregation collects log output from many sources (application instances, containers, infrastructure) into a centralised system where it can be searched, queried, and retained. Structured logging is the practice of emitting logs as machine-parseable key-value pairs (JSON) rather than unstructured text. Compare unstructured: ERROR: Connection refused to db-host:5432 after 3 retries — hard to filter and query — versus structured: {"level":"error","message":"Connection refused","host":"db-host","port":5432,"retries":3,"trace_id":"4bf92f35","service":"order-svc"} — directly queryable by field.

The log shipping pipeline moves logs from where they are produced to where they are stored. Common shippers are: Fluent Bit / Fluentd (CNCF, widely deployed in Kubernetes as a DaemonSet, rich plugin ecosystem); Vector (Rust-based, high performance, low memory, supports transforms and enrichment); Promtail (Grafana Labs, purpose-built for Loki, scrapes Kubernetes pod logs by label). The common backend options are: Elasticsearch (ELK stack) — full-text indexed, powerful query language (Lucene/DSL), expensive storage, excellent for ad-hoc investigation; Grafana Loki — label-indexed (not full-text), cheaper storage using object store (S3), queries via LogQL, excellent Grafana integration, best for Kubernetes environments; Splunk — enterprise, powerful search, SIEM integration, compliance reporting, expensive licensing.

Log-trace correlation injects the active trace ID into every log line. When a trace is slow or errors, the trace view links directly to the corresponding log lines — without trace correlation, finding logs for a specific request requires knowing the request time window and filtering by service, which is slow and imprecise. The OTel SDK automatically injects trace_id and span_id into the logging context when a span is active.

Why it exists

In distributed systems with dozens of services and thousands of instances, logs written to local files are inaccessible for debugging — the relevant log lines might be on any of 20 pods, all of which may be replaced before the investigation starts. Centralised log aggregation solves this by collecting all logs into a single queryable store with a retention window that survives instance restarts. Structured logging transforms logs from text requiring regex parsing into databases of structured events that can be queried like tables — filtering by service, log level, user ID, error code, or any combination, across all services simultaneously.

When to use

• All distributed applications — centralised log aggregation is a baseline operational requirement.
• Compliance environments (PCI DSS, SOC 2, HIPAA) — all require centralised, tamper-evident log retention for defined periods.
• Security monitoring — logs are a primary data source for SIEM and security event detection.

When not to use

• Log aggregation is universally applicable. The choice of system (Loki vs Elasticsearch vs Splunk) depends on scale, cost, and query patterns — but some form of centralised logging is always appropriate.

Typical architecture

KUBERNETES LOG AGGREGATION (Grafana Loki):

Pods write logs to stdout/stderr (Kubernetes convention)
  → Kubernetes captures to /var/log/containers/

Promtail DaemonSet (one pod per node):
  - Tails /var/log/containers/
  - Auto-discovers pods via Kubernetes API
  - Attaches labels: namespace, pod, container, app
  - Sends to Loki via HTTP

Loki:
  - Receives log streams with label sets
  - Compresses and stores in object storage (S3/GCS)
  - Indexes labels only (not log line content)
  - Query: LogQL

Grafana Explore:
  - LogQL queries: {namespace="production", app="order-svc"}
      |= "error" | json | level="error"
  - Metric from log: count_over_time({app="api"}[5m])
  - Link from trace span → logs by trace_id

STRUCTURED LOG FORMAT (JSON):
{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "level": "error",
  "service": "order-svc",
  "version": "1.4.2",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "user_id": "usr_12345",     ← for correlation
  "order_id": "ord_98765",    ← for correlation
  "message": "Payment declined",
  "error_code": "INSUFFICIENT_FUNDS",
  "duration_ms": 45
}

RETENTION POLICY:
  Hot tier (indexed, fast queries): 7–30 days
  Warm tier (compressed, slower): 30–90 days
  Cold tier (archive, S3 Glacier): 1–7 years (compliance)

ELASTICSEARCH (ELK) ALTERNATIVE:
  Beats/Fluentd → Logstash (parse/enrich) → Elasticsearch
  Kibana: full-text search, KQL, Lens visualisations
  More powerful for ad-hoc investigation, more expensive

Pros and cons

Implementation notes

Use structured JSON logging from the start — retrofitting structured logging into applications that use unstructured string formatting is expensive. Configure the logging framework (Logback, Winston, structlog) to output JSON to stdout, not to files. In Kubernetes, stdout is automatically captured by the container runtime and made available for log shippers — no file path management needed.

For Loki, design labels carefully: labels are the index in Loki, so choose labels with low cardinality (namespace, app, environment, log level) and do not put high-cardinality values (trace_id, user_id, request_id) in labels — those belong in the log line itself and are searchable via LogQL's line filter and JSON parser. High-cardinality Loki labels cause the same problems as high-cardinality Prometheus metrics: too many streams, index bloat, poor query performance.

Common failure modes

• Unstructured logs with regex parsing: Parsing unstructured logs with regex in the log shipper is fragile — log format changes break the pipeline silently, resulting in unparsed or dropped log fields.
• No log level control: Applications logging at DEBUG in production generate enormous volumes and costs; runtime log level control (without restart) and per-service log level configuration are essential operational tools.
• Logs as the only observability: Debugging high-traffic systems using only logs requires scanning enormous volumes; distributed tracing is a more efficient tool for latency and cross-service error investigation.
• PII in logs: Logging request bodies, user data, or authentication tokens creates compliance and security exposure; sensitive fields must be masked or excluded at instrumentation time.

Decision checklist

• Are all services emitting structured (JSON) logs with consistent field names?
• Is trace_id injected into every log line when a trace is active?
• Are log levels appropriate and runtime-configurable without restart?
• Are PII and sensitive data excluded or masked from logs?
• Is log retention tiered to balance cost and compliance requirements?
• Are Loki labels low-cardinality (no trace_id, user_id in labels)?

Example use cases

• Incident investigation: Error rate alert fires at 02:00; on-call opens Grafana, filters logs to {app="payment-svc", level="error"} for the alert window; structured logs show error_code: "STRIPE_RATE_LIMIT" on 40% of requests; Stripe rate limit hit during traffic spike; mitigation: add exponential backoff.
• Compliance audit: GDPR right-to-erasure request for user ID 12345; log query {service="order-svc"} | json | user_id="12345" retrieves all log events for the user across 90-day retention; audit evidence provided to compliance team.
• Cost reduction: Team discovers DEBUG logs from auth service generating 50GB/day; implement log level sampling (log 0.1% of DEBUG, 100% of WARN+); logging costs reduce 80% with no loss of operational signal.

Related patterns

• Three Pillars of Observability — Logs in context with metrics and traces.
• Distributed Tracing — Trace IDs that enable log correlation.
• Incident Response — Logs as the primary forensic data source.

Further reading

• Grafana Loki Documentation
• Vector Log Shipper
• 12-Factor App: Logs as Event Streams