Data Contracts

What it is

A data contract is a formal specification, owned by the data producer, that defines what a dataset guarantees to its consumers: the schema (field names, types, nullability), semantics (what each field means), quality commitments (freshness SLA, completeness thresholds), and versioning policy (how breaking changes are managed). It is the data equivalent of an API contract — just as a REST API contract specifies what endpoints accept and return, a data contract specifies what a database table, Kafka topic, or event stream produces.

Data contracts are typically expressed using schema formats: Apache Avro (binary, compact, widely used in Kafka ecosystems); Protocol Buffers (Protobuf) (binary, strongly typed, used in gRPC and event streaming); JSON Schema (human-readable, flexible, used for REST APIs and lightweight event schemas). These schemas are versioned and stored in a Schema Registry (Confluent Schema Registry, AWS Glue Schema Registry, Apicurio) which enforces compatibility rules when producers attempt to publish new schema versions.

The three schema compatibility modes enforced by schema registries are: BACKWARD — consumers using the new schema can read data written with the old schema (add optional fields, remove fields); FORWARD — consumers using the old schema can read data written with the new schema (add fields with defaults, remove optional fields); FULL — both directions are compatible simultaneously (most restrictive — add optional fields with defaults only).

Why it exists

In a typical organization, data flows from producers (operational databases, event streams, SaaS APIs) through multiple transformation layers to consumers (dashboards, ML models, analytics). Without contracts, a producer engineer renames a column in a table; 3 downstream pipelines silently break; dashboards show NULL where numbers should appear; an ML model begins returning incorrect predictions trained on malformed features. Nobody notices for days because all the pipelines ran without error — the data was just wrong. Data contracts replace implicit, undocumented assumptions with explicit, versioned, enforced specifications that make breaking changes visible before they reach consumers.

When to use

• Event-driven architectures: Kafka or event-streaming producers and consumers in different teams — schema registry enforcement prevents incompatible schema changes from reaching consumers in production.
• Data mesh implementations: Each domain team publishes their data products with formal contracts — consumers can trust the data without direct coordination with the producing team.
• Cross-team data dependencies: When the team consuming data is different from the team producing it — contracts provide a stable interface and explicit change notification obligations.
• ML feature pipelines: Feature tables used for model training and serving must not change silently — contracts specify the feature schema and null/range quality constraints models depend on.
• Regulatory reporting pipelines: Financial, compliance, or health data flowing into regulatory reports — contracts document lineage and semantics required for audit evidence.

When not to use

• Don't impose formal contracts on all internal datasets — for exploratory analytics, sandbox tables, or developer-owned scratch data, contracts add overhead without benefit; reserve them for stable, multi-consumer datasets.
• Don't treat schema validation alone as a complete data contract — a schema validates shape but not semantics, quality levels, or SLAs; a complete contract includes all four dimensions.
• Don't use FULL compatibility mode universally — it's the most restrictive and prevents many legitimate schema evolutions; use BACKWARD for most Kafka consumer scenarios where new consumers need to read historical messages.
• Don't start with contracts and hope adoption follows — contracts require producer buy-in and often organizational mandate; start with the highest-value cross-team data flows rather than a bottom-up rollout.

Typical architecture

  Schema Registry (Confluent / AWS Glue / Apicurio):
  ┌─────────────────────────────────────────────────┐
  │  Subject: orders-value                          │
  │  Version 1: { order_id: int, amount: double }  │
  │  Version 2: { order_id: int, amount: double,   │ ← BACKWARD compatible
  │               currency: string = "USD" }        │   (new optional field w/ default)
  │  Version 3: ❌ REJECTED                         │ ← Rename amount → total_amount
  │             (breaks backward compatibility)     │   blocks registry registration
  └─────────────────────────────────────────────────┘

  Producer flow:
  App ──▶ Serialize with Avro schema
      ──▶ Schema Registry: "Is version 2 compatible with BACKWARD rule?"
      ──▶ Registry returns schema ID (or rejects)
      ──▶ Kafka message: [schema_id][avro_bytes]

  Consumer flow:
  Kafka message ──▶ Extract schema_id
                ──▶ Fetch schema from Registry by ID
                ──▶ Deserialize with writer's schema, project to reader's schema
                ──▶ Process with consumer's expected fields

  Full data contract specification (YAML):
  name: "orders"
  version: "2.1.0"
  owner: "payments-team"
  schema:
    format: avro
    registry_subject: orders-value
  quality:
    freshness_sla: "5 minutes"
    completeness:
      - field: order_id
        max_null_rate: 0.0
      - field: amount
        min_value: 0.0
  semantics:
    order_id: "Globally unique order identifier. Never reused."
    amount: "Pre-tax order total in USD. Excludes shipping."
  change_policy:
    notice_period: "14 days"
    breaking_change_approval: "consumers must ACK"
          

Pros and cons

Implementation notes

For Kafka-based event streaming, use Confluent Schema Registry with Avro schemas as the baseline contract enforcement mechanism. Register subject-level compatibility mode as BACKWARD for most topics — this allows consumers to be upgraded before producers, the safest upgrade order. Set the default compatibility to BACKWARD at the registry level and only override per-subject when there's a specific reason for FULL or NONE. Use schema ID embedding: Confluent serializers prepend the schema ID (4 bytes) to each message, enabling consumers to always deserialize with the exact writer's schema even when their local schema differs — this is the "schema on read" pattern that enables independent consumer evolution.

For batch data contracts (warehouse tables, data lake files), use dbt tests as contract assertions: not_null, unique, accepted_values, and relationships tests encode quality constraints directly in the dbt model definition. Combine with dbt-expectations (Great Expectations integration) for range checks and statistical assertions. Run these tests as part of your CI pipeline on every dbt model change — contract failures block deployment. Publish the dbt model's manifest.json and test results to your data catalog (DataHub, Atlan) to surface contract status alongside the table metadata, giving consumers visibility into current contract health.

Common failure modes

• Schema registry bypass: A team publishes Kafka messages without the Schema Registry serializer; messages contain unversioned JSON; downstream consumers can't enforce compatibility; a field rename breaks consumers silently.
• NONE compatibility mode: A registry is configured with compatibility NONE (no enforcement) "temporarily" to unblock an urgent release; the temporary exception becomes permanent; schema drift accumulates unchecked.
• Contract without consumer notification: A producer makes a BACKWARD-compatible change (adds optional field) and considers it safe; a consumer was relying on the field being absent to detect a data category; semantically breaking change passes technical compatibility check.
• Avro field rename migration failure: A developer renames a field in Avro by changing the name in the schema (technically incompatible); the correct approach is adding the new field with an alias; the breaking change is caught in staging but not documented as a known migration pattern.
• Quality SLA not monitored: A data contract specifies "freshness: 1 hour" but no monitoring alerts when the producing pipeline is delayed 4 hours; consumers continue using stale data until someone notices incorrect dashboard values.

Decision checklist

• Is there a schema registry deployed and configured with an appropriate default compatibility mode (BACKWARD for most Kafka use cases)?
• Are all Kafka producers using registry-aware serializers that register and embed schema IDs in messages?
• Are contract quality assertions (not-null, uniqueness, range checks) automated in CI pipelines so that violations block deployment?
• Is there a process for producers to notify consumers before making schema changes, even backward-compatible ones?
• Does the contract include semantic documentation (field descriptions, business meaning) not just structural schema definitions?
• Is contract health (freshness, quality assertion pass rate) monitored and visible in dashboards alongside the data itself?

Example use cases

• Kafka payment events: A payments team produces payment.completed events to Kafka with Avro schema registered in Confluent Schema Registry. When the team adds a new optional payment_method_type field, the registry validates BACKWARD compatibility and issues a new schema version. Downstream fraud detection, analytics, and notifications consumers receive the new field automatically and can opt in to using it without a coordinated release.
• dbt feature table contract: A data platform team publishes a fct_user_features table for ML model training. The dbt model includes not_null and accepted_values contract tests for all required ML features. When an upstream pipeline bug introduces NULLs in a key feature column, the dbt CI test fails and blocks the table from being deployed to the feature store — the ML model training job is not triggered with malformed data.
• Data mesh product contract: The "Order" domain team in a Data Mesh organization publishes their orders data product with a full YAML contract specifying schema, quality SLAs, and a 14-day notice policy for breaking changes. The analytics team builds a quarterly revenue report consuming this product with confidence that the contract will be honored — and receives automated notifications when the producing team plans a change.

Related patterns

• Data Cataloging — Catalogs publish what data exists; contracts define the obligations of data producers about how it will behave.
• Contract Testing — The same consumer-driven contract principle applied to APIs; data contracts extend this to asynchronous data pipelines.
• Data Mesh — Data contracts are fundamental to Data Mesh, enabling domain teams to own and evolve their data products independently.

Further reading

• Confluent Schema Registry Documentation
• DataContract.com — Open Standard for Data Contracts
• Data Contracts in Data Mesh Architecture