Communication PatternsMessaging

Claim Check Pattern

Store large message payloads in external storage (S3, Azure Blob, GCS) and pass only a lightweight reference token through the message broker — eliminating payload size limits and reducing broker memory and bandwidth costs.

⏱ 9 min read

What it is

The claim check pattern (also called reference-based messaging) splits a large message into two parts: the payload is stored in an external store (S3 bucket, Azure Blob Storage, GCS), and a lightweight claim check — a reference token containing the storage URL or object key — is sent through the message broker. The consumer receives the claim check, fetches the payload from the external store using the reference, processes it, and optionally deletes it. The broker never sees the large data.

The pattern is named after a luggage claim check: you drop your bag (payload) at the baggage counter (blob store) and receive a ticket (claim check token). The ticket travels with you (through the broker) and you redeem it at the destination to retrieve your luggage.

Why it exists

Message brokers impose message size limits that are typically measured in kilobytes to single-digit megabytes: SQS default limit is 256 KB, RabbitMQ default is 128 MB but performance degrades sharply above a few MB, Kafka default is 1 MB. When payloads exceed these limits — scanned documents, images, audio, large JSON reports, binary files — the message cannot be sent at all or degrades broker performance for all other tenants. The claim check pattern removes this constraint by keeping the broker responsible only for routing small reference tokens and moving bulk data to cheap, high-capacity object storage.

When to use

  • Large payloads exceeding broker limits — any message over ~64 KB that would exceed or stress your broker's limits (256 KB for SQS, custom for RabbitMQ/Kafka).
  • Binary data — images, PDFs, audio, video thumbnails, serialised ML model outputs that are impractical to base64-encode into a message.
  • Fan-out with large shared data — in a pub/sub topology, one stored payload can be referenced by hundreds of subscriber messages rather than duplicating the data in each message.
  • Audit and replay requirements — the external store becomes a durable, queryable archive of all large payloads; the broker message provides lightweight routing metadata.

When not to use

  • Small payloads (<10 KB) — the extra round-trip to the external store adds latency and complexity for no benefit.
  • Ultra-low latency pipelines — every consumer must make an extra HTTP call to fetch the payload; this adds tens of milliseconds per message.
  • When the external store is unavailable — a claim check pointing at an expired or deleted object causes consumer failures; the claim check is only as reliable as the backing store.

Typical architecture

Producer stores the payload in the external store and receives a reference URL or key. Producer constructs a small message containing the reference, metadata, and routing headers. Producer publishes the small message to the broker. Consumer receives the message, extracts the reference, fetches the payload from external storage, processes it, and ACKs the message. After processing, the consumer may delete the payload or leave it for the TTL to expire.

Producer
  1. PUT large-payload.pdf → S3 Bucket
     ← Returns: s3://my-bucket/jobs/2024-01/abc123.pdf

  2. Publish to broker:
     {
       "claimCheckUrl": "s3://my-bucket/jobs/2024-01/abc123.pdf",
       "contentType": "application/pdf",
       "correlationId": "abc123",
       "metadata": { "pages": 42, "jobType": "ocr" }
     }

Consumer
  3. Receive small message from broker (≪ 1 KB)
  4. GET s3://my-bucket/jobs/2024-01/abc123.pdf
     ← Returns: 4.2 MB PDF bytes
  5. Process payload (OCR extraction)
  6. ACK message to broker
  7. DELETE s3://my-bucket/jobs/2024-01/abc123.pdf

Pros and cons

Pros

  • Removes broker size limits — payloads of arbitrary size are supported.
  • Reduces broker memory and bandwidth — the broker handles only tiny reference messages.
  • Enables payload sharing — a single stored object can be referenced by many messages.
  • Natural audit trail — the external store serves as a durable record of all processed payloads.

Cons

  • Extra latency — consumer must make an additional round-trip to fetch the payload.
  • Additional operational dependency — the external store (S3/blob) becomes a required component in the message pipeline.
  • Reference lifecycle management — payloads must be deleted after processing to avoid unbounded storage growth; TTL must be chosen carefully.
  • Security surface — the object reference must be protected; an unauthorised party that intercepts the message can retrieve the payload if they can access the store.

Implementation notes

Pre-signed URLs vs object keys: There are two strategies for the claim check token. Option A: store the object key (e.g., jobs/2024-01/abc123.pdf) and grant consumers IAM access to the bucket. This is simpler but requires consumers to have permanent S3 read permissions. Option B: include a pre-signed URL with a short TTL (e.g., 30 minutes). This allows consumers without bucket credentials to fetch the payload — useful for cross-account or third-party consumers — but the URL must be generated at publish time, and expiry must align with maximum consumer processing latency.

Storage lifecycle and TTL: Configure S3 lifecycle rules to delete objects after N days based on the prefix or tag. A common approach: set a TTL of 7 days on the payload bucket. If the consumer is healthy, it processes messages well within this window. If the consumer is down for more than 7 days, the payload is gone and the message is a broken reference — handle this as a dead-letter case.

Security: Never store sensitive payloads without server-side encryption (S3 SSE-KMS or SSE-S3). Ensure the bucket is not publicly accessible. If using pre-signed URLs, generate them with the minimum TTL necessary and validate the content hash on retrieval to detect tampering. Treat the claim check message as sensitive if the URL reveals information about the stored data.

Observability impact: Log the claim check reference at every stage (publish, fetch, process, delete) so that distributed traces can link log lines to specific payloads. Without this, debugging a failed consumer is difficult because the log message only contains a reference URL, not the payload content.

Common failure modes

  • Expired pre-signed URL: Consumer is slow to process; by the time it tries to fetch the payload, the pre-signed URL has expired — HTTP 403. Set URL TTL to at least 10× the maximum expected processing time.
  • Payload already deleted: A consumer crashes after fetching but before ACKing; the message is redelivered but the payload was already deleted by a previous attempt. Ensure consumers do not delete payloads during processing — delete only after ACK.
  • Storage unavailable: S3 or blob storage is temporarily unavailable; the consumer cannot fetch the payload and fails. Implement exponential backoff with dead-lettering for storage fetch failures.
  • Reference not stored atomically: Producer stores the payload but crashes before publishing the message; the payload is orphaned in storage. Implement periodic orphan cleanup or use the transactional outbox pattern for the publish step.

Decision checklist

  • Is the payload large enough to justify the extra round-trip (rule of thumb: >64 KB)?
  • Is server-side encryption enabled on the payload store?
  • Is the payload store bucket/container private (not public)?
  • Is storage lifecycle/TTL configured to prevent unbounded object accumulation?
  • Does the consumer delete the payload only after successfully ACKing the message?
  • Is the claim check URL/key logged at each processing stage for observability?
  • If using pre-signed URLs, is the TTL set to at least 10× the maximum processing latency?

Example use cases

  • Document processing pipeline: Users upload contracts (multi-MB PDFs); producer stores in S3 and enqueues a claim check; OCR consumer fetches and extracts text; search indexer consumer fetches and indexes content — both consumers access the same stored payload.
  • Event-driven video transcoding: Upload service stores raw video in S3, publishes a claim check message; transcoding workers fetch the video, produce variants, and store outputs back to S3 under new keys.
  • Cross-account data sharing: Publisher in Account A stores data in S3 and issues a pre-signed URL; message with URL is published to an SNS topic shared with Account B consumers who have no direct S3 access.

Further reading