Interview Questions · Apache Kafka

Apache Kafka Interview Questions & Answers: 50 Questions from Basics to Exactly-Once

Fifty Kafka questions that backend and data engineering interviewers ask, answered precisely: how partitions and consumer groups scale, what acks and ISR really guarantee, how exactly-once works, and how to debug lag and rebalances in production.

✍️ Pranjal Jain, Ex-Microsoft · IIT Kanpur 📅 10 October 2026 ⏱ 24 min read
Parallel conveyor lanes of numbered parcels representing Kafka partitions, replicas and consumer groups
TL;DR
  • Kafka is a distributed, replicated, append-only log. Almost every answer follows from that sentence.
  • Partitions are the unit of parallelism and ordering. Within a consumer group, each partition is read by exactly one consumer.
  • Durability comes from acks=all + min.insync.replicas=2 + replication factor 3, with unclean leader election off.
  • Exactly-once in Kafka means idempotent producers plus transactions for read-process-write within Kafka; side effects in external systems still need idempotency on your side.
  • 2026-relevant facts: Kafka 4.0 removed ZooKeeper (KRaft only), the new consumer rebalance protocol (KIP-848) is generally available, and tiered storage moves old segments to object storage.

Kafka shows up in interviews in two ways: as a technology question ("how do consumer groups work?") and as a building block in system design ("put the events on Kafka, partitioned by what?"). The second is where candidates get exposed. Saying "we'll use Kafka" is easy; explaining how ordering, retries and duplicates behave when a consumer crashes mid-batch is what interviewers actually score.

The 50 questions below go from fundamentals to production operations. Code is in Java (the reference client) and Python (confluent-kafka). For the system design side, pair this with our walkthrough on designing a distributed message queue like Kafka.

How Kafka Shows Up in Interviews

RoundWhat gets askedLevel
Resume deep dive"You used Kafka. What was the partition key? How did you handle duplicates? What was consumer lag at peak?"2+ years backend
Concepts roundPartitions, consumer groups, acks, ISR, offsets, compactionSDE-2, data engineer
System designEvent pipelines for orders, payments, notifications, clickstream; choosing keys and delivery semanticsSDE-2 and above
Debugging scenario"Lag keeps growing", "consumers keep rebalancing", "we saw duplicate payments"Senior

Core Kafka Concepts: Questions 1-10

Q1. What is Apache Kafka?

Kafka is a distributed event streaming platform. Producers append records to topics; Kafka stores them durably as replicated, ordered logs; consumers read them at their own pace, tracking their position with an offset. Unlike a traditional queue, reading does not delete data. Records are retained by time or size, so many independent consumers can read the same stream and replay history.

Q2. Explain topic, partition, offset, broker and cluster.

  • Topic: a named stream of records, like orders or payment-events.
  • Partition: a topic is split into partitions, each an ordered, append-only log stored on brokers. Partitions let a topic scale beyond one machine.
  • Offset: a record's sequential position within a partition. Offsets are per partition, not per topic.
  • Broker: a Kafka server that stores partitions and serves clients.
  • Cluster: a set of brokers plus the controller quorum that manages metadata.

Q3. Why is Kafka so fast?

Five design choices. Sequential I/O: appends to the end of a log file, which disks and SSDs handle far better than random writes. OS page cache: Kafka relies on the kernel's cache instead of its own heap cache, so recent data is served from memory. Zero-copy: on Linux, data goes from page cache to the network socket via sendfile without copying through user space (when TLS is not in use). Batching: producers send and brokers write records in batches. Compression: batches are compressed end to end (lz4, zstd, snappy, gzip).

Q4. What is a consumer group?

A set of consumers sharing a group.id that divide a topic's partitions among themselves. Each partition is assigned to exactly one consumer in the group, so the group processes the topic in parallel without duplicating work. Different groups are independent: an email-service group and an analytics group each receive every record. This is how Kafka offers both queue semantics (within a group) and publish-subscribe (across groups).

Q5. What happens if a group has more consumers than partitions?

The extra consumers sit idle. With 6 partitions, at most 6 consumers in one group do useful work. That is why the partition count caps consumer parallelism, and why it needs planning upfront.

Topic "orders" (4 partitions) Consumer group "billing" P0 ─────────────────────────────► consumer-1 P1 ─────────────────────────────► consumer-1 P2 ─────────────────────────────► consumer-2 P3 ─────────────────────────────► consumer-3 consumer-4 (idle if added beyond 4) Consumer group "analytics" reads P0..P3 independently with its own offsets

Q6. Does Kafka guarantee ordering?

Only within a partition. Records with the same key go to the same partition (by default, a murmur2 hash of the key modulo the partition count), so all events for one order ID or one user stay ordered. There is no global order across partitions. If you truly need total order, you need a single partition, which caps throughput at one consumer.

Q7. How do you choose a partition key?

Pick the entity whose events must be processed in order: orderId for order lifecycle events, accountId for wallet debits and credits, userId for a user's activity. Then check for skew: if one key is very hot (a large merchant on a payments platform, a celebrity's account), its partition becomes a bottleneck. Mitigations include a composite key (merchant plus a bucket number) when strict per-merchant order is not needed, or a dedicated topic for the hot entity.

Q8. What is the role of ZooKeeper, and what is KRaft?

Historically, ZooKeeper stored cluster metadata (brokers, topic configs, partition leadership) and handled controller election. KRaft (Kafka Raft) replaces it with a quorum of Kafka controller nodes that store metadata in an internal replicated log. KRaft was declared production-ready in 3.3, and Kafka 4.0 removed ZooKeeper mode completely. Benefits: one system to operate, faster controller failover, and support for many more partitions per cluster.

Q9. What is a record made of?

A key (optional), a value, a timestamp, optional headers (metadata such as a trace ID or schema version), plus the partition and offset assigned by the broker. Keys drive partitioning and compaction; headers are useful for routing and tracing without parsing the payload.

Q10. Kafka vs a traditional message queue like RabbitMQ?

AspectKafkaRabbitMQ (classic queues)
ModelDistributed log; consumers track offsetsBroker pushes messages; deletes on ack
ReplayYes, within retentionNo, once acknowledged it is gone
OrderingPer partitionPer queue, weakened by redelivery and multiple consumers
ThroughputVery high, batching-orientedLower, per-message routing
RoutingTopic and partition onlyExchanges with rich routing rules
Best forEvent streams, logs, CDC, analytics, event sourcingTask queues, RPC-style work, per-message priorities

More detail, including SQS, in our Kafka vs RabbitMQ vs SQS comparison.

Producers: Questions 11-18

Q11. Walk through what happens when a producer sends a record.

  1. Serialize key and value with the configured serializers (String, Avro, Protobuf, JSON).
  2. Partition: the partitioner picks a partition from the key, or uses the sticky partitioner if the key is null.
  3. Batch: the record is added to an in-memory batch for that partition in the record accumulator.
  4. Send: a background sender thread ships batches to the partition leaders when a batch is full (batch.size) or linger.ms has elapsed.
  5. Acknowledge: the leader writes the batch and responds according to acks; the producer retries retriable errors and finally completes the callback or future.

Q12. What do acks=0, acks=1 and acks=all mean?

  • acks=0: fire and forget. The producer does not wait. Fastest; records can be lost silently.
  • acks=1: the leader has written it to its log. Lost if the leader dies before followers copy it.
  • acks=all (or -1): all in-sync replicas have it. Combined with min.insync.replicas (Q31), this is the durable setting.

Since Kafka 3.0, the Java producer defaults to acks=all and enable.idempotence=true. Interviewers like candidates who know defaults changed; older blog posts still say acks=1.

Q13. What is an idempotent producer?

Retries can create duplicates: the broker writes a batch, the acknowledgement is lost, the producer retries, and the batch is written twice. With idempotence enabled, the broker assigns the producer a producer ID, and the producer attaches a sequence number to each batch per partition. The broker discards a batch whose sequence number it has already written. This gives exactly-once writes per partition for a single producer session, and it preserves ordering with up to 5 in-flight requests per connection.

Q14. How do batch.size and linger.ms affect throughput and latency?

batch.size is the maximum bytes per partition batch (default 16 KB). linger.ms is how long the producer waits to fill a batch before sending. Larger batches and a few milliseconds of linger dramatically improve throughput and compression ratio at the cost of a little latency. The historical default for linger.ms was 0; Kafka 4.0 raised it to 5 ms, reflecting that small batching delays are almost always worth it. For high-volume pipelines, values like 64-256 KB batches and 10-50 ms linger are common starting points, but measure.

Q15. What happens when the key is null?

Since Kafka 2.4, the default partitioner uses a sticky strategy: it sends records to one partition until the batch is full or sent, then switches. That produces bigger batches than strict round-robin while still spreading load over time. Later versions refined it to account for broker performance. Null keys mean no per-entity ordering.

Q16. What happens if you increase the number of partitions on a topic?

New records with an existing key may hash to a different partition, because the mapping is hash modulo partition count. Events for one key can then be split across two partitions, briefly breaking per-key ordering, and any consumer state keyed by partition becomes wrong. You cannot decrease partitions at all. So pick the count with headroom, and if you must expand a key-ordered topic, do it during a quiet period or migrate to a new topic.

Q17. Which compression codec should you choose?

lz4 and zstd are common choices: lz4 for low CPU cost and good speed, zstd for a better compression ratio at moderate CPU. Snappy is similar to lz4; gzip compresses well but uses the most CPU. Compression happens on whole batches, so it works far better with batching enabled. Set it on the producer; brokers keep the producer's compression when the topic's compression.type is producer.

Q18. Show a durable producer configuration.

Java
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.LINGER_MS_CONFIG, "10");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, String.valueOf(64 * 1024));
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, "120000");   // total time incl. retries

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    ProducerRecord<String, String> record =
            new ProducerRecord<>("payment-events", paymentId, payloadJson);
    producer.send(record, (metadata, ex) -> {
        if (ex != null) {
            log.error("send failed for {}", paymentId, ex);   // alert, or write to a fallback store
        } else {
            log.debug("written to {}-{}@{}", metadata.topic(), metadata.partition(), metadata.offset());
        }
    });
}

Point out the callback: a send can fail after all retries, and ignoring the result is a common source of silent data loss.

Consumers, Offsets and Rebalancing: Questions 19-28

Q19. How does a consumer read data?

Consumers pull: the application calls poll() in a loop, which fetches batches of records from the partitions assigned to it, starting at the committed offset. Pull lets slow consumers go at their own pace and lets Kafka batch efficiently. poll() also drives group membership and offset commits, so a consumer that stops polling is considered dead.

Q20. Where are offsets stored, and what does committing mean?

Committed offsets are stored in an internal compacted topic, __consumer_offsets, keyed by group, topic and partition. Committing offset N means "this group has processed everything before N; resume from N after a restart". The committed offset is the position of the next record to read, not the last one processed.

Q21. Auto-commit vs manual commit?

With enable.auto.commit=true (the default), the consumer commits the offsets returned by previous polls every auto.commit.interval.ms (5 seconds by default) as part of later poll() calls. It is convenient, but if records are handed off to other threads, offsets can be committed before processing finishes (possible loss), and a crash can replay up to one interval of records (duplicates). Manual commit after processing gives at-least-once with clear control.

Java
props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-processor");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("payment-events"));
    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) {
            processIdempotently(r.key(), r.value());   // safe to see a record twice
        }
        if (!records.isEmpty()) {
            consumer.commitSync();                     // commit only after processing the batch
        }
    }
}
Python (confluent-kafka)
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "broker1:9092",
    "group.id": "payment-processor",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["payment-events"])
try:
    while running:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print("consumer error:", msg.error())
            continue
        process_idempotently(msg.key(), msg.value())
        consumer.commit(message=msg, asynchronous=False)   # commit this record's offset + 1
finally:
    consumer.close()

Committing per message in Python is simple but slow at high volume; batching commits every N records or every few hundred milliseconds is the usual compromise.

Q22. What does auto.offset.reset do?

It decides where to start when the group has no committed offset for a partition (a new group) or the committed offset no longer exists (deleted by retention). earliest starts from the oldest available record, latest (the default) from new records only, and none throws an error. A new analytics consumer that should process history needs earliest; forgetting this is a classic "why is my new consumer missing data?" bug.

Q23. What is a rebalance and what triggers it?

A rebalance reassigns partitions among the members of a group. Triggers: a consumer joins, leaves or is considered dead; the subscribed topics change partition count; or a consumer subscribed by pattern sees a new matching topic. A consumer is considered dead if it misses heartbeats for session.timeout.ms (45 seconds by default since 3.0), or if it does not call poll() within max.poll.interval.ms (5 minutes by default).

Q24. Why do consumers keep rebalancing, and how do you fix it?

The most common cause is slow processing: a batch of up to max.poll.records (500 by default) takes longer than max.poll.interval.ms, so the consumer is kicked out, its partitions move, it rejoins, and the cycle repeats, often reprocessing the same records. Fixes: reduce max.poll.records, speed up processing, or move slow work off the poll thread while pausing partitions. Other causes are long GC pauses, frequent deployments (fix with static membership) and network issues causing missed heartbeats.

Q25. Eager vs cooperative rebalancing, and what is KIP-848?

In the classic eager protocol, every consumer gives up all its partitions during a rebalance, so the whole group stops ("stop the world"). Cooperative incremental rebalancing (CooperativeStickyAssignor) moves only the partitions that need to move, in multiple rounds, while others keep processing. KIP-848, the next-generation consumer group protocol generally available in Kafka 4.0, moves assignment logic to the broker-side group coordinator and makes rebalances incremental by design, avoiding group-wide synchronization barriers.

Q26. What is static group membership?

Setting group.instance.id gives a consumer a persistent identity. If it restarts within the session timeout (a rolling deploy, say), it gets the same partitions back with no rebalance. Useful for stateful consumers with large local caches or state stores.

Q27. Is KafkaConsumer thread-safe? How do you scale processing?

No. A KafkaConsumer must be used from one thread (only wakeup() is safe from another thread). To scale, either run one consumer per thread, up to the partition count, or have one consumer thread hand records to a worker pool. The second approach needs care: you must track which offsets have completed per partition and commit only the highest contiguous completed offset, and you lose per-partition ordering unless workers are assigned by key.

Q28. What is consumer lag and how do you monitor it?

Lag is the difference between the partition's latest offset (log end offset) and the group's committed offset: how many records are waiting. Check it with kafka-consumer-groups.sh --describe --group <name>, export it to your metrics system (consumer client metrics such as records-lag-max, or external lag exporters), and alert on lag that keeps growing rather than on an absolute number. Growing lag means consumers are slower than producers: add consumers up to the partition count, optimize processing, or add partitions.

Replication and Durability: Questions 29-35

Q29. How does replication work in Kafka?

Each partition has a configured replication factor, typically 3. One replica is the leader and handles all writes (and by default all reads); the others are followers that fetch from the leader, just like consumers. Replicas are placed on different brokers and, with rack awareness, in different racks or availability zones.

Q30. What is the ISR?

The in-sync replica set is the leader plus followers that are fully caught up, meaning they have fetched up to the leader's log end within replica.lag.time.max.ms (30 seconds by default). A follower that falls behind is removed from the ISR and added back when it catches up. With acks=all, the leader waits for every member of the current ISR, not every replica.

Q31. What does min.insync.replicas do?

It sets the minimum ISR size for a write with acks=all to succeed. With replication factor 3 and min.insync.replicas=2, a write is acknowledged only when at least two replicas have it. If the ISR shrinks to 1, producers using acks=all get NotEnoughReplicasException: Kafka chooses consistency over availability for that partition. Without this setting (default 1), acks=all can degrade into acks=1 when followers drop out of the ISR.

💡
The durable trio to quote in interviewsReplication factor 3, min.insync.replicas 2, producer acks=all. You can lose one broker with no data loss and no write outage. Lose two, and writes stop rather than silently becoming unsafe.

Q32. What is the high watermark?

The high watermark is the offset up to which all ISR replicas have replicated. Consumers can read only up to the high watermark, so they never see a record that could disappear if the leader failed. The log end offset (LEO) on the leader can be ahead of it with not-yet-replicated records.

Q33. What happens when a leader broker fails?

The controller detects the failure and elects a new leader from the partition's ISR, then propagates the new metadata. Producers and consumers refresh metadata and continue with the new leader. Records that were acknowledged with acks=all are on every ISR member, so they survive. With KRaft, controller failover and leader election are faster than they were with ZooKeeper, particularly for clusters with many partitions.

Q34. What is unclean leader election?

If all ISR replicas are down, Kafka can either wait for one to return (partition unavailable) or elect an out-of-sync replica (available but loses the records it never received). unclean.leader.election.enable controls this and has defaulted to false since 0.11. Keep it false for anything like payments; consider true only for data where availability beats completeness, such as some metrics.

Q35. What is the preferred leader and rack awareness?

The preferred leader is the first replica in the assignment list. After failures, leadership can concentrate on a few brokers; Kafka periodically moves leadership back to preferred replicas (auto.leader.rebalance.enable) to balance load. With broker.rack set, Kafka spreads replicas across racks or zones, so losing a whole zone does not lose all replicas. Consumers can also be configured to fetch from the closest replica (KIP-392) to save cross-zone network cost.

Delivery Semantics and Exactly-Once: Questions 36-40

Q36. Explain at-most-once, at-least-once and exactly-once.

SemanticsHow you get itFailure outcome
At-most-onceCommit offset before processing; acks=0 producerCrash after commit loses the record
At-least-onceCommit after processing; producer retriesCrash before commit reprocesses: duplicates
Exactly-onceIdempotent producer + transactions + read_committed consumers; or at-least-once + idempotent processingEach record's effect happens once

At-least-once with idempotent consumers is the most common production choice. The honest framing for interviews: exactly-once delivery across arbitrary systems is not achievable; exactly-once processing is, by combining Kafka transactions inside Kafka or deduplication at the sink.

Q37. How do Kafka transactions work?

A producer configured with a transactional.id can write to multiple partitions and commit consumer offsets atomically. A transaction coordinator on the broker tracks it in the internal __transaction_state topic and writes commit or abort markers into each partition. Consumers with isolation.level=read_committed only see records from committed transactions. The transactional.id also fences "zombie" instances: if an old instance of the same producer comes back after a pause, its writes are rejected.

Q38. Show a consume-transform-produce loop with exactly-once.

Java
producer.initTransactions();
while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    if (records.isEmpty()) continue;
    producer.beginTransaction();
    try {
        Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
        for (ConsumerRecord<String, String> r : records) {
            String enriched = enrich(r.value());
            producer.send(new ProducerRecord<>("orders-enriched", r.key(), enriched));
            offsets.put(new TopicPartition(r.topic(), r.partition()),
                        new OffsetAndMetadata(r.offset() + 1));
        }
        // offsets are committed as part of the same transaction as the output records
        producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException e) {
        producer.close();          // another instance took over; stop this one
        break;
    } catch (KafkaException e) {
        producer.abortTransaction();
        // rewind to the last committed offsets so the batch is reprocessed
        for (TopicPartition tp : consumer.assignment()) {
            OffsetAndMetadata committed = consumer.committed(Set.of(tp)).get(tp);
            consumer.seek(tp, committed != null ? committed.offset() : 0L);
        }
    }
}

The consumer here must have auto-commit disabled and isolation.level=read_committed if its input is itself transactional. This is exactly what Kafka Streams does when you set processing.guarantee=exactly_once_v2.

Q39. Does Kafka exactly-once cover writing to a database?

No. Kafka transactions cover Kafka-to-Kafka flows. If your consumer writes to PostgreSQL or calls a payment gateway, a crash between the side effect and the offset commit can repeat the side effect. Solutions: make the sink idempotent (upsert by a unique event ID, or a processed-events table with a unique constraint), or store the consumer offset in the same database transaction as the result and seek to it on startup.

Q40. How do you guarantee an event is published when a database row changes?

Writing to the database and then to Kafka is a dual write: either can fail independently. The transactional outbox pattern writes the event into an outbox table in the same database transaction as the business change; a relay or a CDC tool such as Debezium reads the outbox (or the database log) and publishes to Kafka. This is the standard answer in system design rounds involving orders and payments, and the backbone of the saga pattern.

Storage, Retention and Log Compaction: Questions 41-44

Q41. How does Kafka store data on disk?

Each partition is a directory of segment files. The active segment receives appends; when it reaches segment.bytes (1 GB by default) or segment.ms, it is closed and a new one opens. Each segment has an offset index and a time index, sparse indexes that let Kafka find a given offset or timestamp by binary search and a short scan. Retention and compaction operate on closed segments.

Q42. How does retention work?

With cleanup.policy=delete (the default), whole closed segments are deleted once all their records are older than retention.ms (7 days by default) or the partition exceeds retention.bytes (unlimited by default). Because deletion is per segment, data can live somewhat longer than the configured retention.

Q43. What is log compaction and when do you use it?

With cleanup.policy=compact, Kafka keeps at least the latest record for each key and removes older records with the same key in the background. A record with a null value is a tombstone that marks the key for deletion; tombstones themselves are removed after delete.retention.ms (24 hours by default) so that consumers have time to see them. Use compaction for changelogs and current-state topics: user profiles, product prices, Kafka Streams state store changelogs, and __consumer_offsets itself. A new consumer can rebuild the full current state by reading the compacted topic from the beginning.

Compaction example
Before compaction (key: value)          After compaction
offset 0  user42: {city: "Pune"}
offset 1  user7:  {city: "Delhi"}       offset 1  user7:  {city: "Delhi"}
offset 2  user42: {city: "Bengaluru"}   offset 2  user42: {city: "Bengaluru"}
offset 3  user9:  {city: "Chennai"}     offset 4  user9:  null (tombstone, removed later)
offset 4  user9:  null

Offsets are never renumbered; compaction leaves gaps, which consumers handle transparently.

Q44. What is tiered storage?

Tiered storage (KIP-405, generally available since Kafka 3.9) lets brokers offload closed log segments to remote storage such as S3, keeping only recent data on local disks. Consumers reading old offsets transparently fetch from the remote tier. It decouples retention from broker disk size, making months of retention affordable and making broker replacement faster, since less data needs to be re-replicated.

Operations and Performance Tuning: Questions 45-50

Q45. How do you decide the number of partitions?

Measure the throughput one partition supports on the producer side (p) and on the consumer side (c), which depends heavily on your processing logic. For target throughput T, you need at least max(T/p, T/c) partitions. Example: target 300 MB/s, one partition sustains 30 MB/s on produce, and one consumer processes 10 MB/s: max(10, 30) = 30 partitions, then add headroom for 1-2 years of growth (say 48 or 60). Each partition costs broker memory, file handles and leader election time, so do not create thousands "just in case".

Q46. How do you tune a consumer for throughput?

Increase fetch.min.bytes and fetch.max.wait.ms so the broker returns larger batches; raise max.partition.fetch.bytes for large records; process records in batches (bulk database writes instead of one insert per record); commit asynchronously with a periodic synchronous commit; and keep per-batch processing time well under max.poll.interval.ms. Most consumer slowness is in the processing code, not in Kafka.

Q47. How do you handle poison messages and retries?

A record that always fails (bad schema, a bug) blocks its partition if you retry forever. The common pattern: retry in-process a few times with backoff for transient errors; then publish the record to a retry topic (possibly several with increasing delays) and finally to a dead-letter topic with the error in headers; commit the original offset and move on. Alert on DLQ volume and build a tool to replay DLQ records after fixes. Note that moving records to a retry topic gives up strict ordering for that key, which you must accept or handle.

Q48. What are Kafka Connect, Kafka Streams and Schema Registry?

  • Kafka Connect: a framework for moving data between Kafka and external systems with reusable connectors (JDBC, Debezium CDC, S3, Elasticsearch), handling offsets, scaling and retries for you.
  • Kafka Streams: a Java library for stream processing (filtering, joins, windowed aggregations) with local state stores backed by compacted changelog topics and exactly-once support. No separate cluster; it runs inside your application.
  • Schema Registry (most widely used is Confluent's): stores Avro, Protobuf or JSON schemas, gives each a version ID embedded in messages, and enforces compatibility rules (backward, forward, full) so a producer cannot deploy a change that breaks consumers.

Q49. How do you secure a Kafka cluster?

Encryption in transit with TLS; authentication with SASL (SCRAM, OAUTHBEARER, Kerberos) or mutual TLS; authorization with ACLs per principal on topics, groups and cluster operations; plus quotas to stop one client from saturating brokers. Note that TLS disables the zero-copy optimization because data must be encrypted in user space, which costs some CPU.

Q50. What are share groups (queues for Kafka)?

KIP-932 introduces share groups, which let multiple consumers cooperatively consume from the same partition with per-record acknowledgement, more like a traditional work queue, so consumer parallelism is no longer capped by partition count. It arrived as an early-access feature in Kafka 4.0 and has been maturing in subsequent releases; check its status in the release notes for your version before relying on it. Mentioning it shows you follow the project, but do not build an interview design around it unless asked.

Kafka Design Scenarios for Interviews

Use these to practise applying the answers above. Each is a common system design sub-problem.

Scenario 1: Payment events for a UPI-style platform

Topic payment-events, keyed by paymentId so each payment's lifecycle (initiated, debited, credited, settled) stays ordered. Replication factor 3, min ISR 2, acks=all, idempotent producer. Events are written through the outbox pattern from the payments database. Consumers (ledger, notifications, fraud) are separate consumer groups, each committing after idempotent processing keyed by event ID. See designing a UPI payment system for the wider design.

Scenario 2: Clickstream analytics at high volume

Very high write volume, tolerance for small loss, no strict ordering needs. Use null or user-ID keys, aggressive batching (larger linger.ms and batch.size), zstd or lz4 compression, many partitions sized from measured throughput, and consumers that write in bulk to the warehouse. acks=1 may be acceptable here; say explicitly that you are trading durability for throughput. The ad click aggregation design goes deeper.

Scenario 3: "We are seeing duplicate notifications"

Diagnose: consumers commit after processing, so a rebalance or crash between sending and committing replays records. Check logs for rebalances caused by exceeding max.poll.interval.ms. Fix the cause (smaller batches, faster processing, static membership) and make the effect idempotent by storing sent notification IDs with a unique constraint or a short-TTL key in Redis.

⚠️
Answers that hurt you"Kafka guarantees ordering" (only per partition). "Kafka gives exactly-once so we don't need idempotency" (not for external side effects). "Add more consumers to fix lag" (not beyond the partition count). "Kafka deletes messages after they are consumed" (it retains by time or size). "acks=all means all replicas" (it means all in-sync replicas, which is why min.insync.replicas matters).

Kafka often sits at the centre of event-driven architectures; if your interviews go there, continue with event sourcing and CQRS and our microservices interview questions.

Frequently Asked Questions

What are the most common Kafka interview questions?▾
How partitions and consumer groups provide parallelism, how ordering is guaranteed only within a partition, what acks=all and min.insync.replicas do, how offsets are committed, what causes rebalances, how exactly-once semantics work with idempotent producers and transactions, and when to use log compaction.
Does Kafka guarantee message ordering?▾
Only within a single partition. Messages with the same key go to the same partition and are therefore ordered relative to each other. There is no ordering guarantee across partitions of a topic.
Can Kafka lose messages?▾
Yes, with weak settings: acks=0 or acks=1 with a leader failure, unclean leader election, or a consumer that commits offsets before processing. With acks=all, min.insync.replicas of 2 on a replication factor of 3, unclean election disabled, and commit-after-processing consumers, acknowledged messages survive a single broker failure.
Is ZooKeeper still needed for Kafka?▾
No. KRaft mode, where Kafka manages its own metadata through a Raft-based controller quorum, became production ready in Kafka 3.3, and Kafka 4.0 removed ZooKeeper support entirely. Older clusters may still run ZooKeeper until they migrate.
How many partitions should a Kafka topic have?▾
Enough to meet target throughput and consumer parallelism: divide the target throughput by the measured throughput of one partition on the producer side and on the consumer side, and take the larger number, with headroom for growth. Too many partitions increase metadata, recovery time and open file handles, so do not overshoot by orders of magnitude.
Kafka or RabbitMQ: which should I pick?▾
Kafka suits high-throughput event streams that several consumers read independently, replay, and retain for days. RabbitMQ suits task queues with per-message acknowledgement, flexible routing and priority. Many systems use both for different jobs.
Is Kafka asked for freshers?▾
Rarely as a standalone topic. It appears for backend roles with a couple of years of experience, data engineering roles, and in system design rounds where an event stream or message queue is part of the design, such as notifications, payments or analytics pipelines.

Master Event-Driven System Design

Prepflix's system design track uses Kafka in real designs, from payments to analytics pipelines, so you can defend every partitioning and delivery choice.

Explore the Prepflix Course →
Pranjal Jain
Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.