Continuous Profiling

What it is

Profiling captures a statistical sample of what code is executing at any given moment, allowing engineers to identify which functions consume the most CPU time, allocate the most memory, or hold locks the longest. Traditional profiling is performed on-demand in development environments — attach a profiler, reproduce the issue, analyse results. Continuous profiling (sometimes called "always-on profiling") runs profiling agents at low overhead continuously in production, storing profile data as time-series alongside metrics and traces. This enables engineers to correlate "CPU spiked at 14:32" with "the `JSON.parse()` function in the cache miss path consumed 40% of CPU at 14:32" — connecting the performance signal to the exact code location.

The key profile types: CPU profile — samples the call stack at regular intervals (e.g., 100Hz) to determine which functions are on-CPU most frequently. The profile shows the percentage of samples in which each function appeared, indicating its relative CPU cost. Heap/memory profile — samples memory allocation events to show which functions are allocating the most memory, enabling identification of allocation hotspots, memory leaks, and GC pressure causes. Goroutine/thread profile (Go-specific) — shows all goroutines, their current state (running, blocked, waiting), and their call stacks — essential for diagnosing goroutine leaks and deadlocks. Mutex/contention profile — shows which mutexes are being contended most, identifying synchronisation bottlenecks under concurrent load.

A flamegraph is the standard visualisation for profile data. The x-axis represents the percentage of samples (wider = more time spent), not time elapsed. The y-axis represents the call stack depth (root at bottom, leaf functions at top). To read a flamegraph: look for wide blocks at the top of the flame — these are the functions consuming the most CPU. Click to zoom into a subtree. Look for plateaus where a parent function is wide but its children are many small slices — this indicates a function spending most of its time in work rather than delegating. A "tall thin spike" indicates a deep but rarely-taken code path.

Why it exists

Metrics and traces answer "something is slow" and "which request was slow." Profiling answers "why is it slow — which lines of code are responsible." Without profiling, performance investigations rely on guesswork: instrumenting suspected slow functions, redeploying, waiting for the problem to recur, and iterating. Continuous profiling short-circuits this loop by providing always-available code-level performance data. Google's Engineering Productivity Research found that profiling data attached to performance incidents reduced investigation time by 50%. Grafana Pyroscope and Parca (CNCF) have made continuous profiling accessible without significant infrastructure investment, positioning it as the fourth pillar of observability (alongside metrics, logs, and traces).

When to use

• Services with unexplained CPU or memory pressure — continuous profiling identifies the specific code path responsible without requiring an on-demand profiling session to reproduce the issue.
• Performance regression investigation — correlating a performance metric change with a specific commit's code changes by comparing profiles before and after.
• Cost optimisation — CPU profiling in cloud environments can identify inefficient code that drives compute costs.

When not to use

• High-overhead profilers at full sampling rate on latency-sensitive services — CPU profiling overhead is typically 1-3% with modern profilers (Pyroscope uses 10Hz sampling by default), but some profiling configurations are significantly more expensive and should be load tested before production deployment.

Typical architecture

CONTINUOUS PROFILING ARCHITECTURE:

Application → Pyroscope SDK → Pyroscope Server → Grafana

OR (pull-based):
Application (pprof endpoint) → Parca Agent → Parca Server

GO PPROF INTEGRATION:
  // Enable pprof HTTP endpoints
  import _ "net/http/pprof"

  // Exposed at: GET /debug/pprof/
  // /debug/pprof/profile?seconds=30  (CPU)
  // /debug/pprof/heap               (memory)
  // /debug/pprof/goroutine           (goroutines)
  // /debug/pprof/mutex               (lock contention)

PYROSCOPE SDK (Go):
  pyroscope.Start(pyroscope.Config{
      ApplicationName: "checkout-api",
      ServerAddress:   "http://pyroscope:4040",
      ProfileTypes: []pyroscope.ProfileType{
          pyroscope.ProfileCPU,
          pyroscope.ProfileAllocObjects,
          pyroscope.ProfileAllocSpace,
          pyroscope.ProfileInuseObjects,
          pyroscope.ProfileInuseSpace,
      },
      Tags: map[string]string{
          "version":     os.Getenv("APP_VERSION"),
          "environment": "production",
      },
  })

PYROSCOPE SDK (Python):
  pyroscope.configure(
      app_name="ml-inference",
      server_address="http://pyroscope:4040",
      tags={"service": "ml-inference", "env": "prod"}
  )

FLAMEGRAPH READING GUIDE:
  Wide block = function consumes significant CPU
  Height = call stack depth
  Look for: wide top-level blocks (CPU hotspots)

  Example flamegraph analysis:
  [HTTP handler - 80% of CPU]
    [JSON deserialise - 55% of CPU]  ← HOTSPOT
      [encoding/json - 55%]
    [DB query - 20% of CPU]
    [response marshal - 5%]

  Finding: replace encoding/json with sonic/jsoniter
  Result: 55% CPU reduction → 45% overall reduction

PROFILING OVERHEAD:
  Pyroscope (10Hz sampling): ~1% CPU overhead
  Go pprof (30s CPU profile): ~2-5% during profile
  py-spy (Python): ~1% overhead at 100Hz

Pros and cons

Implementation notes

Start with CPU and heap profiling — these are the most universally useful profile types and have the lowest interpretation barrier. Add goroutine and mutex profiling only when investigating specific concurrency issues. Tag profiles with deployment version, git commit, and environment so that profile comparisons across deployments are possible. When a performance regression occurs, the "compare profiles" view in Pyroscope or Grafana's continuous profiling view shows exactly which functions changed their CPU contribution between versions.

For Go services, the net/http/pprof import automatically exposes profiling endpoints — but ensure these endpoints are not exposed publicly; they should be accessible only within the cluster or behind authentication. Use Kubernetes NetworkPolicy to restrict access to /debug/pprof. Pyroscope and Parca both support eBPF-based profiling that works without any application-side SDK changes — this is particularly valuable for services you do not control or cannot modify, allowing cluster-wide continuous profiling with a single agent deployment.

Common failure modes

• Profiling inlined functions: Compiler optimisations inline small functions, making them invisible in profiles — the CPU appears to be in the calling function. Disabling inlining for debugging (at the cost of performance) or using frame pointer unwinding with eBPF-based profilers avoids this.
• Misreading wide base blocks: Main/root functions appear wide because all paths flow through them — width at the bottom of the flamegraph is not indicative of cost; cost is shown by width at the top of the visible flame (leaf-level functions).
• Exposing pprof endpoints publicly: Go pprof endpoints expose internal application details (function names, goroutine states) and allow triggering CPU profiles that consume resources; restrict access to internal networks only.

Decision checklist

• Is continuous profiling deployed for CPU and memory on high-traffic services?
• Are profiles tagged with deployment version and git commit for regression comparison?
• Is pprof (or equivalent) endpoint access restricted to internal cluster traffic?
• Are engineers trained on flamegraph interpretation and common hotspot patterns?
• Is profile data retention policy defined to manage storage costs?

Example use cases

• Identifying JSON serialisation bottleneck: Service CPU trending up 20% over 3 weeks with no obvious cause; continuous profile comparison shows encoding/json consuming 45% of CPU in response marshalling path; replaced with faster JSON library; CPU returns to baseline; cost reduction of ~$2,000/month in EC2 costs.
• Go goroutine leak detection: Service memory grows slowly over days until OOM kill; goroutine profile shows 50,000 goroutines, each blocked on an HTTP client request with no timeout configured; goroutines are never garbage collected; fix: add context with timeout to all outbound HTTP calls; goroutine count stabilises at 200.
• Regression bisect: p99 latency increases 40% in Tuesday's deployment; Pyroscope diff view compares Monday vs Tuesday profile; shows a regex compilation that was previously cached is now being compiled on every request due to a subtle caching regression; root cause found in 15 minutes rather than hours of instrumented investigation.

Related patterns

• Distributed Tracing — Traces identify slow requests; profiling explains why they are slow.
• Three Pillars of Observability — Profiling is often considered the fourth pillar alongside metrics, logs, and traces.
• OpenTelemetry — OTel profiling signal specification is under active development.

Further reading

• Grafana Pyroscope Documentation
• Parca Continuous Profiling (CNCF)
• Go pprof: Profiling Go Programs