Kafka vs RabbitMQ vs SQS: Which Message Broker Should You Choose?

"Use Kafka" has become a reflex in system design answers, and it is often wrong. Kafka, RabbitMQ and SQS solve overlapping but different problems, and the difference comes down to one design choice: is the broker a log that consumers read from, or a queue that hands out messages and forgets them? This guide explains each architecture, compares them on the dimensions that matter, shows producer and consumer code for each, and ends with a decision table.

The Core Difference: A Log vs a Queue

Once this distinction is clear, most choices make themselves. "Order events consumed by billing, analytics, notifications and fraud, each at their own pace" is a log. "Resize these 10,000 uploaded images using a pool of workers" is a queue.

How Kafka Works

• Topics are split into partitions; each partition is an append-only, ordered log stored as segment files on disk. Each record gets an increasing offset.
• Producers choose a partition by key hash (same key → same partition) or spread keyless records across partitions.
• Each partition has a leader and followers. With acks=all, a write is acknowledged once all in-sync replicas have it; combined with min.insync.replicas=2 and replication factor 3, you tolerate one broker failure without losing acknowledged data.
• Consumer groups: within a group, each partition is assigned to one consumer. Max useful parallelism = number of partitions. Different groups are fully independent.
• Consumers commit offsets (stored in an internal topic). Crash before commit → messages since last commit are reprocessed.
• Retention by time or size (for example 7 days), or log compaction to keep the latest value per key forever (useful for changelogs and state).
• Metadata is managed by KRaft, a built-in Raft controller quorum; Kafka 4.0 removed ZooKeeper.

Why it is fast: sequential disk writes, heavy batching and compression, the OS page cache, and zero-copy transfer from file to socket. Consumers pull in large batches. For a full design walkthrough, see design a distributed message queue like Kafka.

from confluent_kafka import Producer, Consumer
import json

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,     # no duplicates/reordering from producer retries
    "acks": "all",
})

def place_order(order):
    producer.produce("orders", key=order["order_id"], value=json.dumps(order))
    producer.poll(0)                # serve delivery callbacks

producer.flush()

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "billing",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,    # commit only after successful processing
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    handle_billing(json.loads(msg.value()))      # must be idempotent
    consumer.commit(message=msg, asynchronous=False)

Kafka's weak spots

• Not a task queue. No per-message ack, no per-message retry or delay, no priorities. A poison message blocks its partition unless you build retry and dead-letter topics yourself.
• Parallelism is capped by partitions, and increasing partitions changes key-to-partition mapping for new messages, which can break per-key ordering during the change.
• Operational weight. Brokers, storage, rebalancing, partition reassignment, monitoring consumer lag. Managed offerings (Confluent Cloud, Amazon MSK, Aiven) reduce this at a price.
• Kafka 4.0 introduced share groups ("queues for Kafka", KIP-932) as an early-access feature to support queue-like per-record consumption; check its status for your version before relying on it.

How RabbitMQ Works

RabbitMQ implements AMQP 0-9-1 (plus MQTT, STOMP and AMQP 1.0 via plugins or natively in newer versions). Producers never publish to queues directly; they publish to exchanges, which route messages to queues via bindings.

• Consumers receive pushed messages up to a prefetch limit (basic.qos), process them and ack. Unacked messages are redelivered if the consumer dies. Nack/reject can requeue or dead-letter.
• Dead letter exchanges, per-message and per-queue TTL, priorities, and request-reply via reply_to are built in.
• Quorum queues (Raft-replicated) are the recommended durable, replicated queue type; classic mirrored queues were removed in RabbitMQ 4.0.
• RabbitMQ Streams (since 3.9) add an append-only, replayable log type, narrowing the gap with Kafka for some use cases.

import pika, json

conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.exchange_declare(exchange="events", exchange_type="topic", durable=True)
ch.queue_declare(queue="billing", durable=True,
                 arguments={"x-queue-type": "quorum",
                            "x-dead-letter-exchange": "events.dlx"})
ch.queue_bind(queue="billing", exchange="events", routing_key="order.created")

# producer
ch.confirm_delivery()                    # publisher confirms: broker acks the publish
ch.basic_publish(exchange="events", routing_key="order.created",
                 body=json.dumps({"order_id": "o-91", "amount": 450}),
                 properties=pika.BasicProperties(delivery_mode=2))  # persistent

# consumer
def on_message(channel, method, props, body):
    try:
        handle_billing(json.loads(body))           # must be idempotent
        channel.basic_ack(method.delivery_tag)
    except Exception:
        channel.basic_nack(method.delivery_tag, requeue=False)  # to DLX

ch.basic_qos(prefetch_count=20)
ch.basic_consume(queue="billing", on_message_callback=on_message)
ch.start_consuming()

RabbitMQ's weak spots

• Classic queues are designed to be short. Millions of backed-up messages hurt memory and performance (lazy mode and quorum queues help, but a huge backlog is still a smell).
• No replay for classic and quorum queues: once acked, a message is gone. A new consumer cannot read history (Streams can).
• Per-queue throughput is bounded by one queue's leader; scaling means sharding across queues (consistent hash exchange, sharding plugin, super streams).

How Amazon SQS Works

SQS is a managed queue with an HTTP API. There are no brokers, partitions or clusters to manage, and you pay per request.

1. Producer calls SendMessage.
2. Consumer calls ReceiveMessage (use long polling with WaitTimeSeconds up to 20 to avoid empty responses and cost).
3. The received message becomes invisible to others for the visibility timeout (default 30 seconds, up to 12 hours).
4. After processing, the consumer calls DeleteMessage. If it does not delete in time, the message becomes visible again and is redelivered.
5. After maxReceiveCount failed receives, a redrive policy moves it to a dead-letter queue.

Other facts worth knowing: retention is configurable from 1 minute to 14 days (default 4 days); delay queues and per-message timers up to 15 minutes; payloads larger than the message size limit go to S3 with a pointer (the Extended Client Library does this). For pub/sub fan-out, publish to an SNS topic subscribed by multiple SQS queues. Limits change over time, so check the current SQS quotas page.

import boto3, json

sqs = boto3.client("sqs", region_name="ap-south-1")
url = sqs.get_queue_url(QueueName="billing-jobs")["QueueUrl"]

# producer
sqs.send_message(QueueUrl=url, MessageBody=json.dumps({"order_id": "o-91"}))

# consumer
while True:
    resp = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10,
                               WaitTimeSeconds=20, VisibilityTimeout=60)
    for m in resp.get("Messages", []):
        try:
            handle_billing(json.loads(m["Body"]))      # must be idempotent
            sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])
        except Exception:
            pass   # not deleted: reappears after the visibility timeout, then DLQ

SQS's weak spots

• No replay after delete; no multiple independent consumer groups (use SNS fan-out to several queues).
• AWS-only; per-request pricing can become significant at very high volume compared with a self-run cluster.
• Per-message latency is typically higher than an in-datacentre broker because every operation is an HTTPS API call.
• For a Kafka-like stream on AWS, use Kinesis Data Streams or Amazon MSK instead.

Ordering Guarantees Compared

"Is ordering guaranteed?" is a favourite interview question, and the honest answer for all three is "only within a scope you define".

• Kafka: strict order within a partition. Use the entity ID (order ID, user ID) as the key, so all events for that entity are ordered. With the idempotent producer (default in recent clients), retries do not reorder. Adding partitions remaps keys for new messages.
• RabbitMQ: a single queue is FIFO, but with multiple consumers, redeliveries and prefetch, processing order is not guaranteed. For strict order, use one consumer per queue (the single active consumer feature helps), or partition entities across queues with the consistent hash exchange.
• SQS: Standard has no ordering guarantee. FIFO orders within a message group; set MessageGroupId to the entity ID. Different groups are processed in parallel.

Delivery Semantics: At-Most-Once, At-Least-Once, Exactly-Once

What each offers:

• Kafka has idempotent producers and transactions, which together give exactly-once semantics for read-from-Kafka, process, write-to-Kafka pipelines (Kafka Streams uses this). Once your consumer writes to an external database or calls an API, you are back to at-least-once and need idempotency.
• RabbitMQ: publisher confirms + persistent messages + quorum queues + manual consumer acks give at-least-once.
• SQS FIFO: deduplicates sends within 5 minutes, but a consumer that exceeds the visibility timeout causes redelivery.

So in every case: design consumers to be idempotent (processed-message table, conditional updates, unique constraints). The patterns are covered in saga pattern, outbox and idempotent consumers.

Throughput, Latency, Replay and Operational Cost

Benchmarks vary enormously with message size, replication and acknowledgement settings, so treat any single "X messages/second" claim with suspicion and test your own workload.

Retries, Dead-Letter Queues and Poison Messages

Every consumer eventually meets a message it cannot process: malformed JSON, a reference to a deleted entity, a downstream API returning errors for hours. How each broker handles this is one of the most practical differences between them, and one of the most common causes of production incidents.

SQS: built in

Set a redrive policy on the queue, for example maxReceiveCount = 5. Each failed attempt simply lets the visibility timeout expire; after the fifth receive the message moves to the DLQ. Backoff between attempts is the visibility timeout, which you can extend per message with ChangeMessageVisibility to implement exponential backoff. SQS also supports redriving messages from the DLQ back to the source queue after you fix the bug.

RabbitMQ: dead-letter exchanges plus TTL

A basic.nack with requeue=false, a message TTL expiry, or exceeding a quorum queue's delivery-limit routes the message to the queue's dead-letter exchange. Delayed retries are typically built with a "wait" queue that has a TTL and dead-letters back to the work queue, or with the delayed message exchange plugin. Never nack with requeue=true in a tight loop: a poison message will spin at full speed and starve everything behind it.

Kafka: you build it

Kafka has no per-message failure state. If the consumer does not commit past a bad record, it re-reads it forever and the partition stalls. The standard pattern is retry topics:

• On failure, publish the record to a retry topic with attempt count and error in headers, then commit the original offset so the partition keeps flowing.
• Retry consumers wait until the record is old enough before processing (pause the partition rather than sleeping inside the poll loop, or the consumer gets kicked from the group for exceeding max.poll.interval.ms).
• After the last retry, the record goes to a dead-letter topic.
• Trade-off: moving a record to a retry topic breaks per-key ordering. If order matters, stop processing that key (or the whole partition) until the failure is resolved instead.

Spring Kafka (@RetryableTopic) and Kafka Connect's errors.deadletterqueue.topic.name implement these patterns so you do not hand-roll them.

Decision Table: Which Broker for Which Job

Many large systems use more than one: Kafka as the event backbone, plus SQS or RabbitMQ for job queues where per-message retry and delay matter.

How Message Brokers Come Up in Interviews

Almost every system design answer includes a queue somewhere: notifications, video processing, feed fan-out, payments, order pipelines. Interviewers listen for whether you chose deliberately. Kafka-specific internals come up in backend rounds too (see Kafka interview questions).

Questions to prepare

1. "Why Kafka here and not SQS?" Multiple consumers, replay, ordering per key, throughput. Or the reverse: "This is a job queue with retries; SQS is simpler."
2. "How do you guarantee order for a user's events?" Key by user ID; per-partition order; idempotent producer.
3. "A consumer crashes after processing but before committing. What happens?" Redelivery; idempotent consumer.
4. "One bad message keeps failing." Retry with backoff, then DLQ (or retry topics in Kafka) with alerting.
5. "Consumer lag is growing." Scale consumers up to the partition count, increase partitions (aware of key remapping), batch processing, find slow downstream calls.
6. "How do you publish an event reliably after a DB write?" Transactional outbox; never dual-write.

Model answer snippet

"Order events go to a Kafka topic keyed by order ID with 48 partitions, replication factor 3 and acks=all. Billing, notifications and analytics are separate consumer groups. Notification sending itself is a job with per-message retries and delays, so the notification service enqueues sends into SQS with a DLQ. Every consumer dedupes by event ID, since delivery is at-least-once."

Common mistakes

• Using Kafka as a task queue and then struggling with per-message retries and poison messages.
• Claiming "Kafka gives exactly-once" without qualifying that it applies to Kafka-to-Kafka processing.
• Needing global ordering and not noticing it limits you to one partition.
• Using SQS Standard where order matters.
• Forgetting the DLQ and alerting, so failed messages vanish or loop forever.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.