Change Data Capture

What it is

Change Data Capture (CDC) is an integration technique that monitors a database's transaction log (write-ahead log, binlog, redo log) to capture row-level changes — inserts, updates, and deletes — and stream them as events to downstream consumers in near real-time. Unlike polling-based approaches that periodically query tables for changed records, CDC reads the database's internal replication infrastructure, which is already maintained for high-availability purposes. This makes CDC efficient, low-overhead, and capable of capturing deletes (which timestamp-based polling cannot).

The dominant open-source tool for CDC is Debezium, a distributed platform built on Kafka Connect that provides connectors for PostgreSQL (using logical replication / WAL), MySQL (binlog), SQL Server (change tracking tables), MongoDB (oplog), Oracle (LogMiner), and others. Each change event emitted by Debezium contains the full before- and after-image of the row, metadata about the source transaction (LSN position, timestamp, transaction ID), and the DDL operation type. These events are published to Kafka topics, typically one topic per source table.

Why it exists

The fundamental problem CDC solves is bridging the gap between a stateful database and an event-driven or streaming architecture. When a service owns a relational database, writing to the DB and publishing an event to a message broker atomically is difficult — the dual-write problem. If the DB write succeeds but the event publish fails (or vice versa), downstream consumers diverge from the source of truth. The Transactional Outbox pattern solves this at the application level, but CDC solves it at the infrastructure level by treating the database log as the authoritative event source.

CDC also enables non-invasive integration with legacy systems. If you cannot modify an existing application to emit events but need to stream its data changes to a data warehouse, search index, or microservice, CDC lets you tap the database without touching the application code. This is invaluable for data migration, real-time analytics, cache invalidation, and maintaining read replicas across polyglot persistence stores.

When to use

• You need to stream database changes to Kafka for event-driven consumers but cannot modify the source application to emit events.
• Building a real-time data pipeline to a data warehouse (Snowflake, BigQuery) or data lake without periodic batch ETL jobs.
• Implementing cache invalidation: when a record changes in the DB, CDC events trigger cache eviction in Redis or similar.
• Maintaining a search index (Elasticsearch, OpenSearch) in sync with a primary relational database.
• Cross-region or cross-datacenter data replication where you need sub-second propagation with full-fidelity change events including deletes.
• As the event source in a Transactional Outbox pattern, reading the outbox table via CDC rather than application-level polling.

When not to use

• The database does not expose a transaction log to external consumers (some managed DBaaS offerings restrict this) — check that logical replication is supported and enabled.
• You need business-level events with domain semantics (OrderPlaced, CustomerUpgraded) rather than low-level row change events — CDC produces structural changes; event meaning must be interpreted by consumers.
• Tables undergo very high DDL churn (frequent schema changes) — CDC connectors can fail or require reconfiguration when table structures change significantly.
• The source table contains sensitive PII and downstream Kafka topics cannot be adequately secured — CDC exposes all column data including deleted records.

Typical architecture

  PostgreSQL (source)               Kafka                    Consumers
  ┌────────────────────┐
  │  Application DB    │
  │  ┌──────────────┐  │    ┌──────────────────┐   ┌─────────────────┐
  │  │  orders tbl  │  │    │                  │   │  Search Index   │
  │  │  INSERT/UPD  │  │    │  Topic:          │──▶│  (Elasticsearch)│
  │  │  DELETE      │  │    │  db.public.orders│   └─────────────────┘
  │  └──────────────┘  │    │                  │   ┌─────────────────┐
  │                    │    │  {before: {...}, │──▶│  Data Warehouse │
  │  WAL (pg_wal)      │    │   after: {...},  │   │  (Snowflake)    │
  │  ┌──────────────┐  │    │   op: "u",       │   └─────────────────┘
  │  │ LSN offset   │──┼───▶│   ts_ms: ...}    │   ┌─────────────────┐
  │  │ txn log      │  │    │                  │──▶│  Cache Eviction │
  │  └──────────────┘  │    └──────────────────┘   │  (Redis)        │
  └────────────────────┘                           └─────────────────┘
          ▲
  Debezium Postgres Connector
  (Kafka Connect Worker)
          

Pros and cons

Implementation notes

For PostgreSQL, enable logical replication by setting wal_level = logical in postgresql.conf and creating a replication slot for Debezium. A replication slot causes the WAL to be retained until the consumer acknowledges receipt, so monitor slot lag carefully — an idle or lagging connector will cause unbounded WAL growth and can fill your disk. Set max_slot_wal_keep_size as a safety valve. Create a dedicated replication user with REPLICATION privilege and grant SELECT on all tables to be captured. For MySQL, enable the binlog with binlog_format = ROW and binlog_row_image = FULL.

Use the Debezium Schema Registry integration to publish Avro-encoded events; this ensures schema compatibility is enforced as source schemas evolve. Configure tombstone.on.delete = true to emit null-value tombstone events on deletes, enabling Kafka log compaction to work correctly. For tables with sensitive PII, use Debezium's Single Message Transformations (SMTs) to mask or drop fields before they reach the Kafka topic. Monitor connector lag with the kafka.connect:type=connector-metrics JMX metrics and set alerts on offset-lag exceeding acceptable thresholds.

Common failure modes

• Replication slot starvation: The Debezium connector falls behind and the PostgreSQL WAL grows unboundedly until disk fills, causing DB crash; always monitor slot lag and set max_slot_wal_keep_size.
• DDL breaks connector: A developer adds a NOT NULL column to a source table without telling the CDC team; the connector fails to deserialize new events because the schema changed without a new Avro schema version.
• Initial snapshot race: During the initial table snapshot, new writes arrive that CDC captures as change events; consumers must handle the overlap between snapshot and streaming correctly to avoid duplicates.
• PII in Kafka topics: Developers forget that CDC events contain all columns and publish a topic with Social Security Numbers to a Kafka cluster that security scanners flag months later.
• Consumer lag misinterpretation: Consumers treat CDC events as reliable domain events and build business logic on field-level deltas, creating tight coupling to DB schema rather than domain semantics.

Decision checklist

• Does your database support and expose its transaction log for external consumption (logical replication for PostgreSQL, binlog for MySQL)?
• Can you afford the WAL/binlog retention overhead and connector infrastructure (Kafka Connect workers)?
• Have you planned for schema evolution governance so DDL changes don't silently break CDC pipelines?
• Have you assessed PII exposure and planned field-level masking for sensitive columns?
• Do consumers need domain events with business semantics, or are row-level changes sufficient?
• Have you considered the Transactional Outbox pattern as an application-level alternative that produces cleaner domain events?

Example use cases

• Real-time search sync: An e-commerce platform uses Debezium on its PostgreSQL products table to stream every product insert/update/delete to Kafka, from which an Elasticsearch sink connector indexes changes within 200ms, keeping search results current without batch reindexing.
• Data warehouse loading: A SaaS company streams CDC events from its operational MySQL database to Kafka and uses a Kafka Connect JDBC sink to load incremental changes into Snowflake, replacing nightly batch ETL jobs and reducing data freshness from 24 hours to under 5 minutes.
• Microservice synchronization: During a database-per-service migration, CDC events from the monolith's order table feed a new Order microservice's replica database, keeping them in sync until the migration cutover is complete.

Related patterns

• Transactional Outbox — Application-level alternative to CDC for publishing events atomically with DB writes.
• Streaming Integration — CDC events are often the source for Kafka Streams or Flink streaming pipelines.
• ETL vs ELT — CDC provides the Extract step for real-time ELT pipelines into data warehouses.
• Anti-Corruption Layer — Wrapping CDC events in an ACL to translate structural row events to domain events.

Further reading

• Debezium Documentation — Official reference for CDC connectors
• PostgreSQL Logical Replication documentation
• Change Data Capture at LinkedIn — Martin Kleppmann