Metrics and Monitoring

What it is

Metrics are numerical measurements of system state sampled over time. Unlike logs (discrete events) or traces (request paths), metrics are aggregated by design — they sacrifice detail for efficiency, enabling long-term retention and fast querying of system-wide trends. The four fundamental Prometheus metric types are:

• Counter: A monotonically increasing value that resets on restart. Measures cumulative counts (requests served, errors, bytes sent). Always query rate over time, not raw value: rate(http_requests_total[5m]).
• Gauge: A value that can go up or down. Measures current state (active connections, queue depth, memory usage, CPU percentage). Query the current value directly.
• Histogram: Samples observations into configurable buckets, tracking cumulative counts and the total sum. Used for latency distributions — enables p50/p90/p99 calculation with histogram_quantile(0.99, ...). Buckets must be pre-configured based on expected latency range.
• Summary: Like histogram but calculates quantiles client-side. Cannot be aggregated across instances — avoid for distributed systems; prefer histograms.

Prometheus uses a pull model: it scrapes HTTP /metrics endpoints exposed by instrumented services at configurable intervals (typically 15s). This is in contrast to push-based systems (StatsD, InfluxDB line protocol) where applications push metrics to a collection endpoint. Pull has advantages: Prometheus can detect when a target is down (missing scrape), metrics are always available for ad-hoc querying from the service, and no centralised push endpoint is needed. The Prometheus data model identifies each time series by a metric name and a set of key-value labels: http_requests_total{method="GET", status="200", service="api"} is a unique time series. Cardinality = the number of unique time series = product of unique values per label dimension.

The three primary monitoring methodologies structure which metrics to collect: USE (Brendan Gregg) for infrastructure resources: Utilisation, Saturation, Errors. RED (Tom Wilkie) for services: Rate (requests/sec), Errors (error rate), Duration (latency). Four Golden Signals (Google SRE): Latency, Traffic, Errors, Saturation — combining USE and RED with explicit latency focus. All three methodologies start from the same insight: most service degradation is visible in a small, well-chosen set of metrics.

Why it exists

Metrics are the first line of detection for system problems. A dashboard showing request rate, error rate, and p99 latency for all services — updated every 15 seconds — enables detection of degradation within seconds to minutes of occurrence. This is far faster than log search or trace analysis. Metrics are cheap to store (numbers, not text), fast to query, and naturally support time-series analysis (trends, anomaly detection, rate-of-change). The Prometheus ecosystem became dominant in cloud-native environments because its pull model integrates naturally with ephemeral container workloads, and the PromQL query language is expressive enough to compute complex derived metrics at query time.

When to use

• All production services — RED metrics (rate, errors, duration) for every service are a non-negotiable baseline.
• Infrastructure monitoring — USE metrics (utilisation, saturation, errors) for all compute, network, and storage resources.
• SLO alerting — SLI metrics are the foundation of error budget tracking and burn rate alerts.

When not to use

• High-cardinality event detail (per-user, per-request data) should go into traces and logs, not metrics. Adding unbounded labels to metrics causes cardinality explosion.

Typical architecture

PROMETHEUS ARCHITECTURE:

Services expose /metrics (Prometheus format):
  # HELP http_requests_total Total HTTP requests
  # TYPE http_requests_total counter
  http_requests_total{method="GET",status="200"} 1234
  http_requests_total{method="GET",status="500"} 12

Prometheus scrape config (prometheus.yml):
  scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true

KEY PROMQL PATTERNS:
  # Request rate (RED - Rate):
  rate(http_requests_total{job="api"}[5m])

  # Error rate (RED - Errors):
  rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m])

  # p99 latency (RED - Duration):
  histogram_quantile(0.99,
    sum(rate(http_request_duration_seconds_bucket[5m]))
    by (le, service))

  # CPU utilisation (USE):
  1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))
    by (instance)

RECORDING RULES (pre-compute expensive queries):
  groups:
  - name: api_metrics
    rules:
    - record: job:http_requests:rate5m
      expr: sum(rate(http_requests_total[5m])) by (job)
    - record: job:http_errors:rate5m
      expr: sum(rate(http_requests_total{status=~"5.."}[5m]))
            by (job)

LONG-TERM STORAGE:
  Prometheus: 15-day local retention
  Thanos / Grafana Mimir: multi-year HA retention
  Remote write: push to Thanos/Mimir from Prometheus

Pros and cons

Implementation notes

Use recording rules for any PromQL expression that is used in dashboards or alerts — recording rules pre-compute the expression on every evaluation, stored as a new time series, making dashboards fast even at high cardinality. Without recording rules, complex dashboard queries at 90-day range scan over millions of samples on every page load. Name recording rules following the convention level:metric:operation — e.g., job:http_requests:rate5m.

Manage cardinality explicitly: audit metric cardinality regularly with prometheus_tsdb_head_series and identify the top contributors. Avoid labels whose values are unbounded or proportional to traffic (e.g., user agent strings, full URL paths — use path templates instead like /users/:id not /users/12345). Use Prometheus's metric_relabel_configs to drop high-cardinality labels before they are stored.

Common failure modes

• Cardinality explosion: A single new high-cardinality label (e.g., adding IP address per request) multiplying existing time series by millions, causing Prometheus OOM and query timeouts.
• Histogram bucket misconfiguration: Default Prometheus histogram buckets (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10 seconds) are wrong for many services — a service with p99 of 200ms needs finer buckets below 500ms for accurate percentile calculation.
• No long-term storage: Default Prometheus 15-day retention means no historical data for capacity planning, trend analysis, or year-over-year comparisons — needs Thanos or Mimir from day one.
• Alerting on raw counters: Alerting http_errors_total > 0 fires on the first error ever and stays firing forever; always alert on rate() to alert on error rate, not cumulative count.

Decision checklist

• Are RED metrics (rate, errors, duration) instrumented for all services?
• Are USE metrics (utilisation, saturation, errors) instrumented for all infrastructure resources?
• Are cardinality limits defined and enforced (no unbounded labels)?
• Are recording rules defined for all dashboard and alert queries?
• Is long-term metric storage configured (Thanos/Mimir/remote write)?
• Are histogram bucket boundaries appropriate for the expected latency range?

Example use cases

• API gateway monitoring: Instrument all routes with http_requests_total{method, status, route} and http_request_duration_seconds{route} histogram; Grafana dashboard shows per-route rate, error %, and p50/p90/p99 latency; alerts fire when error rate exceeds SLO threshold.
• Database resource monitoring: USE methodology applied: postgres_exporter provides connection utilisation, replication lag (saturation), and query error rates; alerts on >80% connection pool utilisation and >10s replication lag.
• SLO burn rate alerting: Record job:api_availability:ratio1h = 1 - error_rate; alert fires when 1h burn rate × 14.4 > 1 (fast burn) or 6h burn rate × 6 > 1 (slow burn), indicating error budget consumption pace that would exhaust the 30-day budget.

Related patterns

• Three Pillars of Observability — Metrics in context with logs and traces.
• SLOs and Error Budgets — Metrics as the foundation for SLO measurement.
• Alerting Strategies — Building effective alerts on top of metrics.

Further reading

• Prometheus Metric Types
• Brendan Gregg: USE Method
• Google SRE: Four Golden Signals