Observability & Operations

Synthetic Monitoring

Scripted browser tests with Playwright, API probes, global probe locations, Pingdom, Datadog Synthetics, and Grafana Cloud Synthetic Monitoring — and how synthetic tests complement real user monitoring.

⏱ 9 min read

What it is

Synthetic monitoring uses scripted, automated tests to simulate user interactions with a service and measure availability and performance from controlled probes. Unlike Real User Monitoring (RUM), which captures data from actual users, synthetic monitors run on a predictable schedule from defined geographic locations — providing consistent baseline measurements regardless of actual user traffic. This makes synthetic monitoring the only reliability signal available during low-traffic periods (nights, weekends), for services with few users in specific regions, or for testing scenarios that are difficult to observe from real user traffic alone (payment flows, admin workflows).

The main test types are: HTTP API probes — send an HTTP request to an endpoint, verify status code, response time, and optionally response body content. Suitable for uptime monitoring and SLI measurement. Browser tests (Playwright, Selenium) — launch a headless browser and execute a scripted user journey (load homepage, search for product, add to cart, proceed to checkout). Measures end-to-end performance including JavaScript execution, third-party scripts, and rendering. Catches regressions invisible to pure API tests. Multi-step API tests — chain multiple API calls to simulate a complete workflow (authenticate → create resource → verify resource → delete resource), validating that the entire sequence completes successfully. TCP/UDP probes — verify that a port is open and accepting connections; useful for infrastructure-level availability monitoring of databases, message brokers, and other non-HTTP services.

Geographic probe distribution is a key feature: running synthetic tests from probes in North America, Europe, and APAC detects latency or availability issues affecting specific regions — CDN misconfiguration, DNS routing failures, or data centre issues. Without multi-region synthetic tests, a service might be completely unavailable in APAC while appearing fully healthy in dashboards driven by US-based probes.

Why it exists

RUM only tells you about failures that real users experience and report through your telemetry — it requires users to be present and for their telemetry data to be collected. Synthetic monitoring runs whether users are present or not, providing continuous availability verification. Critically, synthetic monitors can detect issues before real users encounter them: a canary deploy that breaks the checkout flow will be caught by a synthetic checkout test within 1-5 minutes, before significant user traffic has routed to the new version. Synthetic monitoring is also the primary tool for enforcing uptime SLAs with third-party dependencies — automated probes to third-party API endpoints provide objective evidence of SLA compliance or breach.

When to use

  • Critical user journeys (checkout, login, core business workflows) — synthetic browser tests provide end-to-end validation on a schedule.
  • Low-traffic periods or regional coverage gaps — synthetic monitoring fills observability gaps where RUM data is sparse.
  • Third-party SLA monitoring — automated probes provide objective evidence for SLA claims with vendors.

When not to use

  • As a replacement for RUM — synthetic tests simulate idealised user journeys; real users encounter different network conditions, browser versions, and usage patterns that only RUM captures. Both are needed for comprehensive coverage.

Typical architecture

SYNTHETIC MONITORING ARCHITECTURE:

Global Probe Network:
  Probes in: us-east-1, us-west-2, eu-west-1,
             ap-southeast-1, ap-northeast-1

Tests scheduled every 1-5 minutes from each probe.

TEST TYPES:

1. HTTP API Probe (Datadog Synthetics):
   GET https://api.example.com/health
   Expected: status 200, response < 200ms
   Assertions:
     - status code: 200
     - response time: < 200ms
     - body.status: "UP"
   On failure: P1 alert → PagerDuty

2. Multi-step Browser Test (Playwright):
   test('Checkout flow', async ({ page }) => {
     await page.goto('https://shop.example.com');
     await page.fill('[data-test="search"]', 'laptop');
     await page.click('[data-test="search-btn"]');
     await page.waitForSelector('[data-test="results"]');
     await page.click('[data-test="first-result"]');
     await page.click('[data-test="add-to-cart"]');
     await expect(page.locator(
       '[data-test="cart-count"]')).toHaveText('1');

     // Measure step durations
     const searchDuration = await measureStep('search');
     expect(searchDuration).toBeLessThan(500);
   });

3. Multi-step API Test:
   Step 1: POST /auth/login → save token
   Step 2: POST /orders { items: [...] } → save orderId
   Step 3: GET /orders/{orderId} → verify status: "created"
   Step 4: DELETE /orders/{orderId}

ALERTING:
  1 probe failure in 1 location → warning (might be fluke)
  2 consecutive failures → alert
  Failures in 2+ locations → P1 (likely real outage)
  Single location → P2 (might be regional issue)

METRICS PRODUCED:
  - synthetic.availability (% of successful probes)
  - synthetic.response_time (p50, p95, p99 by location)
  - synthetic.step_duration (per step in multi-step tests)

RUM vs SYNTHETIC:
  Synthetic: consistent, proactive, low volume, controlled
  RUM: real user behaviour, high volume, reactive, variable

Pros and cons

Pros

  • Detects issues before real users do — a synthetic test failing immediately after a deployment provides faster detection than waiting for real user error rates to rise through RUM data.
  • Provides consistent baseline measurements for SLO calculation that are not subject to the variability of real user traffic patterns.
  • Multi-region probes detect geographic availability issues (CDN failures, regional DNS problems) that single-region monitoring misses entirely.

Cons

  • Scripted browser tests are brittle — UI changes (selector changes, element reordering) break tests requiring maintenance; test scripts must be treated as production code with proper ownership.
  • Synthetic tests simulate idealised flows with consistent network conditions and default browser configuration; they do not represent the diversity of real user conditions (slow connections, older devices, browser extensions).
  • Cost scales with test frequency and number of probe locations; running browser tests every minute from 10 locations is significantly more expensive than simple HTTP probes.

Implementation notes

Prioritise tests for the 3-5 most critical user journeys rather than attempting to cover every page or endpoint. The checkout flow, login/authentication, and core API endpoints are the high-value targets. Write browser tests using stable test selectors (data-test attributes) rather than CSS classes or element positions — this makes tests robust to UI refactoring. Mandate that all UI components expose data-test attributes as part of your development standards. For multi-step tests, capture and report timing for each individual step — a step that degrades from 100ms to 800ms is an early warning even if the overall test still passes within the total timeout.

Set alert thresholds that require failures from multiple probe locations before escalating to a page — a single probe failure in one location is often a probe infrastructure issue or transient network blip rather than a service problem. Require 2+ consecutive failures or 2+ location failures before triggering a P1 alert. Store synthetic test results as time-series metrics and include synthetic availability in SLI calculations alongside RUM data — synthetic monitoring provides the floor of your SLI even when real user traffic is absent.

Common failure modes

  • Brittle test scripts: Tests that use CSS selectors or XPath break on any UI change; require stable data-test attributes and own tests as production code with the same review standards.
  • Too many tests: Monitoring every page and API endpoint creates maintenance burden without proportional value; focus on critical user journeys.
  • Single probe location: A single US-based probe cannot detect CDN failures or regional outages affecting APAC users; deploy probes in all regions where significant user populations exist.
  • No test data management: Browser tests that create real test orders, test users, or payment transactions in production without cleanup; use dedicated test accounts, sandbox environments, or cleanup logic to prevent data accumulation.

Decision checklist

  • Are the top 3-5 critical user journeys covered by synthetic browser tests?
  • Are probes running from multiple geographic regions matching user distribution?
  • Are alert thresholds requiring multiple consecutive or multi-location failures before paging?
  • Are test scripts using stable selectors (data-test attributes) rather than brittle CSS selectors?
  • Are synthetic test results included in SLI/SLO calculations?
  • Is there a process for keeping browser test scripts current when UI changes?

Example use cases

  • Deployment smoke test: After every production deploy, synthetic tests run automatically against the new version; a broken checkout test fires within 3 minutes of deploy; deployment is automatically rolled back before any real users encounter the failure; on-call is notified but no user-visible incident occurs.
  • Regional CDN outage detection: CDN edge node fails for APAC region; synthetic probes from Singapore and Tokyo return 504 errors; US probes continue returning 200; on-call is alerted with clear regional attribution; CDN failover is initiated.
  • SLA monitoring for third-party payment gateway: Synthetic probes hit payment gateway sandbox API every 5 minutes; monthly availability report generated automatically; used in quarterly SLA review meeting with payment vendor as objective evidence of below-SLA performance in two months.

Further reading