Message Broker Selection

The decision

Message brokers are the backbone of asynchronous architectures, providing the persistent queues and event streams that decouple services in time. The major options—Apache Kafka, RabbitMQ, Amazon SQS/SNS, Apache Pulsar, and Redis Streams—represent very different design philosophies, and choosing the wrong one for your workload creates operational burden and performance ceilings that are expensive to escape later.

Kafka is a distributed log optimized for high throughput, long-term message retention, and event replay. RabbitMQ is a traditional message broker with flexible routing patterns, designed for complex message topologies where messages are consumed and discarded. SQS is a fully managed AWS service that excels at simple, durable queuing with minimal operational overhead. Pulsar extends Kafka's model with tiered storage and multi-tenancy. Redis Streams provides millisecond latency for real-time use cases where in-memory processing is acceptable.

These systems differ along several axes that matter profoundly: throughput (messages per second), latency (time from publish to consumer processing), retention (how long messages are kept), routing complexity (how messages are directed to specific consumers), multi-tenancy (isolation between teams or applications), and operational complexity (how much expertise is required to run it reliably). No single broker excels on all axes simultaneously.

Why it matters

Message brokers sit at the center of event-driven architectures. Getting this selection wrong has cascading consequences. Choosing Kafka for a simple task queue wastes engineering effort on operational complexity (ZooKeeper/KRaft cluster management, partition rebalancing, consumer group coordination) that provides no benefit over SQS. Choosing SQS for a high-throughput event streaming use case hits throughput limits (SQS processes ~3,000 messages/second per queue for standard queues) and lacks the log retention and replay capabilities needed for event sourcing patterns.

Message ordering, delivery guarantees, and consumer group semantics differ significantly between brokers. Kafka guarantees ordering within a partition; ordering across partitions requires careful key selection. SQS standard queues guarantee at-least-once delivery but not ordering; SQS FIFO queues add ordering but at higher cost and lower throughput. RabbitMQ delivers messages in order but removes them on acknowledgment, making replay impossible. These guarantees directly affect what application-level logic you need to implement for idempotency, deduplication, and reordering.

Operational cost is often underestimated. Kafka clusters require expertise in partition management, consumer lag monitoring, replication configuration, and broker recovery procedures. A misconfigured Kafka cluster—wrong retention settings, insufficient replicas, unbalanced partitions—silently degrades performance or loses data. Managed services (Confluent Cloud, Amazon MSK for Kafka; CloudAMQP for RabbitMQ; AWS SQS for queuing) trade cost for operational simplicity, and for most teams the managed service premium pays for itself in engineering hours saved.

Choose Kafka when

• Throughput requirements exceed 100,000 messages per second and horizontal scaling is needed
• Message retention and replay are required—event sourcing, audit logs, reprocessing after bugs
• Multiple independent consumer groups need to read the same stream at different offsets
• Stream processing with Kafka Streams or ksqlDB is planned as part of the architecture
• The system needs to handle data ingestion pipelines from many producers to many consumers
• Strong ordering guarantees within a partition are required for event sequencing

Choose alternatives when

• Simple task queuing with minimal operational overhead—use SQS
• Complex routing logic (topic exchanges, header-based routing, dead letter workflows)—use RabbitMQ
• Sub-millisecond latency with in-memory processing is acceptable—use Redis Streams
• Multi-tenant platform with strict isolation between teams and workloads—consider Pulsar
• AWS-native architecture with no dedicated broker operations team—use SQS/SNS

Comparison

THROUGHPUT VS COMPLEXITY COMPARISON
════════════════════════════════════════════════════════════════

Throughput (msgs/sec)
   1M ┤                                      ● Kafka
      │                               ● Pulsar
 100K ┤
      │
  10K ┤                       ● RabbitMQ
      │
   1K ┤   ● Redis Streams (per instance)
      │           ● SQS (per queue)
   └──┴─────────────────────────────────────────
      Simple                             Complex
                  Operational Complexity

FEATURE COMPARISON
═══════════════════════════════════════════════════════════════════════

Feature           Kafka        RabbitMQ     SQS          Pulsar       Redis Streams
──────────────────────────────────────────────────────────────────────────────────
Max throughput    Very High    High         Moderate     Very High    High (in-mem)
Latency           Low (ms)     Low (ms)     Medium       Low          Ultra-low (μs)
Message retention Log (days+)  Until ACK    Up to 14d    Tiered (∞)   Memory/RDB
Replay            ✓ (offsets)  ✗            ✗ (FIFO only)✓            ✓ (limited)
Ordering          Per partition Optional     FIFO only    Per topic    Per stream
Routing           Topic/key    Exchanges    SNS fanout   Topic/ns     Stream keys
Multi-tenancy     Manual       vhosts       IAM/queues   Namespaces   DB/prefix
Managed cloud     MSK/Confluent CloudAMQP   AWS native   StreamNative Redis Cloud
Operational cost  High         Medium       None (SaaS)  Medium       Low-Medium
Protocol          Kafka wire   AMQP         HTTP/SQS     Pulsar/AMQP  RESP

USE CASES:
Kafka       → Event streaming, CDC, analytics pipelines, event sourcing, log aggregation
RabbitMQ    → Task queues, complex routing, work distribution, priority queues
SQS         → Simple work queues, Lambda triggers, microservice decoupling on AWS
Pulsar       → Multi-tenant platforms, tiered storage for infinite retention, geo-replication
Redis Streams→ Real-time leaderboards, IoT telemetry, live notifications, sub-ms pipelines
          

Trade-offs

Implementation considerations

For Kafka, partition count is one of the most important configuration decisions—it cannot be decreased after the fact without data migration, and increasing it redistributes data and triggers consumer rebalancing. A common guideline is to provision 10–30 partitions per topic for moderate throughput, allowing consumer parallelism to match partition count. Replication factor should be 3 for production (data survives two broker failures). Set min.insync.replicas=2 with acks=all for producer durability to prevent data loss when a broker fails. Use Schema Registry with Avro or Protobuf to enforce message schema compatibility as producers and consumers evolve independently.

For RabbitMQ, choose the exchange type carefully as it determines routing behavior: direct exchanges route by exact routing key (task queues); topic exchanges route by pattern-matching (logs.*.error); fanout exchanges broadcast to all bound queues (notifications); headers exchanges route by message headers (complex criteria). Lazy queues (stored on disk rather than in memory) are mandatory for queues that may grow large—default queues that grow beyond available RAM crash the broker. Enable publisher confirms and consumer acknowledgments for at-least-once delivery; unacknowledged messages are requeued on consumer failure.

For SQS, understanding the difference between Standard and FIFO queues is essential. Standard queues offer maximum throughput but with at-least-once delivery (occasional duplicates) and best-effort ordering. FIFO queues guarantee exactly-once processing and strict FIFO within a message group, but throughput is limited to 3,000 messages/second with batching or 300 without. Use SQS with Dead Letter Queues (DLQ) for every production queue—messages that fail processing after a configurable number of attempts are moved to the DLQ, preventing poison pills from blocking the main queue indefinitely. Set the visibility timeout at 6x the maximum expected processing time to prevent premature message redelivery.

Common mistakes

• Kafka for simple task queues: Using Kafka where SQS would suffice adds cluster management overhead for no benefit; choose the simplest tool that satisfies requirements.
• Under-partitioned Kafka topics: Starting with 3 partitions on a high-throughput topic creates a bottleneck that requires a disruptive repartitioning operation to resolve later.
• Missing Dead Letter Queues: Without DLQ configuration, a single malformed message that causes consumer exceptions blocks queue processing indefinitely as the message is repeatedly redelivered and failed.
• No idempotency in consumers: At-least-once delivery (all major brokers except SQS FIFO) means consumers must handle duplicate messages; non-idempotent consumers cause double charges, duplicate notifications, or corrupted state.
• Large message payloads: Storing large binary payloads (images, PDFs, large JSON documents) directly in message bodies degrades broker performance; use the claim-check pattern—store the payload in S3/object storage and include only the reference in the message.

Decision checklist

• What is the required throughput (messages per second) at peak load, and does it exceed what SQS can handle?
• Do consumers need to replay historical messages, or is consume-and-discard sufficient?
• Do multiple independent consumer groups need to read the same stream simultaneously?
• Is the routing logic complex (pattern matching, header-based routing, priority queues)?
• What is your operational capacity for managing a Kafka or RabbitMQ cluster versus using a fully managed service?
• Are you on AWS and need minimal operational overhead? SQS/SNS with Lambda or ECS consumers is the default choice.
• What are the latency requirements? Sub-millisecond suggests Redis Streams; milliseconds to seconds is fine for any broker.

Real-world examples

• LinkedIn (Kafka origin): LinkedIn created Kafka to handle its activity stream—user interactions, page views, search queries—at a scale that no existing message broker could sustain. The log retention and replay model was specifically designed for LinkedIn's need to feed both real-time processing and batch analytics from the same event stream.
• Slack (RabbitMQ to Kafka migration): Slack ran on RabbitMQ for years to handle message delivery between clients. As scale grew, they migrated portions of the architecture to Kafka for the throughput and retention capabilities, demonstrating that systems often start with simpler brokers and migrate to Kafka as requirements outgrow the simpler model.
• AWS Lambda (SQS): Virtually every AWS Lambda-based microservice uses SQS as its trigger mechanism. The fully managed nature, automatic scaling, DLQ support, and native Lambda integration make SQS the de-facto standard for simple asynchronous task processing in AWS-native architectures without a dedicated broker operations team.

Related decisions

• Synchronous vs Asynchronous Communication — the prior decision: do you need a message broker at all?
• Communication Patterns — pub/sub, competing consumers, and message routing patterns
• Event-Driven Architecture — event streaming as an architectural style
• Event Sourcing — using Kafka as an event store for event sourcing patterns

Further reading

• Apache Kafka Documentation — kafka.apache.org
• RabbitMQ Documentation — rabbitmq.com
• Amazon SQS Developer Guide — AWS Documentation