Observability & Operations

Health Checks and Probes

Liveness, readiness, and startup probes; /health and /ready endpoint patterns; deep vs shallow checks; Kubernetes probe configuration; and Spring Boot Actuator health indicators.

⏱ 9 min read

What it is

Health checks are endpoints that expose the operational state of a service to its infrastructure platform. Kubernetes defines three distinct probe types with different semantics: A liveness probe answers "is this container alive and not deadlocked?" — if it fails, Kubernetes restarts the container. A readiness probe answers "is this container ready to receive traffic?" — if it fails, Kubernetes removes the pod from service endpoints but does not restart it. A startup probe answers "has this container finished initialising?" — it disables liveness/readiness probes during startup to prevent premature restarts of slow-starting applications (JVM warmup, database migrations, loading large models).

A shallow health check verifies only that the service process is alive and can respond — it returns 200 OK as long as the HTTP server is running. This is appropriate for liveness probes, where the goal is to detect deadlocks or process crashes. A deep health check verifies critical dependencies — database connectivity, cache availability, message broker connection — and is appropriate for readiness probes. A deep liveness probe is dangerous: if the database goes down, all pods fail their liveness probe simultaneously, triggering cascading restarts that amplify the incident rather than allowing graceful degradation.

The /health endpoint pattern (popularised by Spring Boot Actuator) exposes structured JSON with overall status and per-component status. The overall status aggregates components: if any required component is DOWN, the overall status is DOWN. Optional components (cache) may be DOWN without affecting overall health. Health endpoint status codes: 200 OK means healthy; 503 Service Unavailable means unhealthy (load balancers interpret 5xx as unhealthy for traffic routing). Some platforms expose `/ready` for readiness and `/live` for liveness as separate paths with different behaviour.

Why it exists

Without health checks, container orchestrators cannot distinguish a healthy pod from a deadlocked one, a pod that has just started from one ready to serve traffic, or a pod whose dependencies are unavailable from one in full health. The consequences are serving traffic to pods that cannot respond correctly, failing to route traffic to pods that have completed startup, and failing to restart genuinely broken pods. Health checks are the minimum contract between an application and its platform that enables automated remediation, traffic management, and rolling deployments to work correctly.

When to use

  • All containerised services running in Kubernetes or any container orchestrator that supports health probes.
  • Services behind load balancers that use health check status to route traffic.
  • Applications with slow startup (JVM, large ML models, database migrations) — startup probes prevent premature liveness/readiness failures during initialisation.

When not to use

  • Deep health checks as liveness probes — a dependency failure should not cause container restarts; it should cause the pod to be removed from rotation (readiness) while the service degrades gracefully.

Typical architecture

KUBERNETES PROBE CONFIGURATION:
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: checkout-api
        livenessProbe:
          httpGet:
            path: /live     # Shallow: is process alive?
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3   # Restart after 3 failures
          timeoutSeconds: 2

        readinessProbe:
          httpGet:
            path: /ready    # Deep: are deps connected?
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3   # Remove from LB after 3 failures
          successThreshold: 1   # Add back after 1 success

        startupProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 30  # 30 × 10s = 5 min to start
          periodSeconds: 10

HEALTH ENDPOINT RESPONSE:
GET /health → 200 OK (or 503 if unhealthy)
{
  "status": "UP",
  "components": {
    "db": {
      "status": "UP",
      "details": { "database": "PostgreSQL", "ping": "1ms" }
    },
    "redis": {
      "status": "UP",
      "details": { "version": "7.2" }
    },
    "diskSpace": {
      "status": "UP",
      "details": { "free": "45GB", "threshold": "1GB" }
    }
  }
}

LIVENESS (/live): always returns 200 unless deadlocked
READINESS (/ready): returns 200 only when all required
                    deps are available

SPRING BOOT ACTUATOR:
  management.endpoint.health.show-details=always
  management.health.livenessState.enabled=true
  management.health.readinessState.enabled=true
  → /actuator/health/liveness
  → /actuator/health/readiness

DEPENDENCY SENSITIVITY:
  Required (DOWN → pod not ready): database, primary cache
  Optional (DOWN → status DEGRADED, still serves traffic):
    secondary cache, analytics pipeline, notification service

Pros and cons

Pros

  • Kubernetes integration enables automated traffic removal during degradation (readiness) and automated restart on crash/deadlock (liveness) without human intervention.
  • Separate liveness and readiness semantics prevent the common failure mode of cascading restarts when a dependency is unhealthy — pods are removed from rotation but not restarted, maintaining capacity while the dependency recovers.
  • Startup probes eliminate the configuration compromise of setting long initialDelaySeconds for all environments to accommodate slow-starting instances.

Cons

  • Deep readiness checks that are too aggressive (checking a flaky dependency) can cause unnecessary traffic removal and create cascading availability issues — health check sensitivity must be tuned carefully.
  • Health endpoint itself becomes a dependency that must be maintained and tested; a health endpoint that crashes or hangs is worse than no health endpoint.
  • Over-reliance on health checks as a substitute for proper dependency management — a service that constantly flaps between ready and not-ready has a design problem, not just a health check configuration problem.

Implementation notes

The liveness endpoint should be extremely simple — only check internal state that is meaningful for the running process (connection pool health, goroutine/thread counts, internal deadlock detection). It must never call external services, as a transient network failure would then cause unnecessary container restarts. The readiness endpoint should check all dependencies required to serve requests: database connection, message broker, required caches. Each dependency check should have a very short timeout (100-500ms) — the health check itself must complete quickly or the probe times out. Run dependency checks in parallel when multiple dependencies must be verified.

Use the failureThreshold setting to require sustained probe failures before action is taken. A single probe failure due to a transient timeout should not immediately remove a pod from rotation. Three consecutive failures (with periodSeconds: 5, this means 15 seconds of sustained failure) provides adequate debouncing while still responding to real problems quickly enough. For the liveness probe, a higher failureThreshold (5-10) is appropriate since container restarts are more disruptive than brief traffic removal.

Common failure modes

  • Deep liveness probes: Checking database connectivity in the liveness probe causes all pods to restart when the database has a transient hiccup — amplifying a recoverable degradation into full service unavailability. Liveness should be shallow; readiness should be deep.
  • No startup probe for slow-starting services: JVM application with 45-second startup sets initialDelaySeconds: 60 to avoid premature liveness failures; this hardcodes a 60-second delay for all deploys even when startup completes in 15 seconds; startup probes eliminate this waste.
  • Health endpoint leaking sensitive information: Detailed health responses exposing internal hostnames, database versions, or dependency topology — health endpoints should be available only within the cluster or behind authentication in production.
  • Readiness check too strict: Including optional/non-critical dependencies in readiness check means the pod is removed from traffic for every non-critical dependency hiccup; classify dependencies as required vs optional and only fail readiness for required dependencies.

Decision checklist

  • Are liveness probes shallow (process-level only, no external calls)?
  • Are readiness probes deep (checking required runtime dependencies)?
  • Are startup probes configured for services with >10 second startup time?
  • Are probe failureThreshold values set to avoid triggering on transient failures?
  • Are optional dependencies (non-critical caches) excluded from failing readiness?
  • Is the health endpoint access-controlled or limited to internal cluster traffic?

Example use cases

  • Database connection pool exhaustion: Checkout service readiness probe checks database connectivity; connection pool exhausted during traffic spike; pods fail readiness, are removed from load balancer; new requests are distributed to less loaded pods; pool recovers; pods rejoin rotation — all without restart or manual intervention.
  • Slow JVM startup: Java microservice takes 40-90 seconds to warm up (class loading, cache pre-warming); startup probe allows up to 5 minutes before the liveness/readiness probes become active; pods are not restarted for being "slow" during normal initialisation.
  • Rolling deployment health gate: New version deployed rolling; each new pod must pass readiness probe before the next pod is updated; if new pods fail readiness (due to a config error), rollout pauses automatically; no traffic is sent to the broken version; rollout is halted before affecting all pods.

Further reading