The fastest way to lose a Kafka interview is to say Kafka guarantees message ordering. It doesn’t. It guarantees ordering inside a single partition and nowhere else. Once a topic has more than one partition, two messages written a millisecond apart can be read in either order unless they share a partition key. Interviewers push on partitions, consumer groups, and delivery guarantees precisely because those three ideas are where the clean “distributed log” picture meets what actually happens when a broker dies mid-write.
Most Kafka rounds live inside a system design or data engineering loop, usually 45 minutes, sometimes a dedicated streaming round at shops that run large Kafka deployments. The questions open conceptual and then lean on the edges: what happens to the consumer that owned partition 3 when it crashes, can you get exactly-once from Kafka into a Postgres table, why is your consumer lag climbing on one partition while the rest stay flat. Strong answers name the config knobs and the tradeoffs behind them. Weak answers describe the happy path and stop.
Why a topic with twelve partitions has no global order
A partition is an append-only log with its own offset sequence. A topic is a name over a set of partitions, nothing more. When a producer sends a record, the partitioner picks a partition: by hash of the key if you set one, sticky batching if you don’t. That single decision drives most of Kafka’s behavior downstream.
Ordering holds within a partition and never above it. If you need every event for a given user to stay in order, the user id has to be the partition key. The partition key is the only ordering lever you have, so a careless choice surfaces later as out-of-order processing that’s miserable to debug. A common follow-up: you keyed by user id and now a single user drives 40% of traffic, what breaks? That’s a hot partition. One broker does most of the work, the consumer that owns it falls behind, and adding partitions doesn’t help because the hot key still hashes to exactly one of them.
Partition count is also your parallelism ceiling. You can’t run more active consumers in a group than there are partitions, so a topic with six partitions caps useful consumers at six. People overcorrect and create thousands of partitions, which inflates end-to-end latency and slows recovery after a failure. A sane answer to “how many partitions” sizes for peak throughput per partition and target consumer parallelism, then leaves headroom, instead of reciting a magic number.
Consumer groups and the one-consumer-per-partition ceiling
A consumer group splits a topic’s partitions across workers that share a group.id. Each partition is owned by exactly one consumer in the group at any moment. Two consumers in the same group never read the same partition; two consumers in different groups each read everything independently. That model answers most “how do I scale consumers” questions on its own: add consumers up to the partition count, then horizontal scaling within that group is done.
A group’s read position lives in its committed offsets. Kafka keeps them in an internal topic called __consumer_offsets, one position per partition per group. Restart a consumer and it resumes from the last committed offset for the partitions it gets assigned. That resume behavior is also where a lot of production bugs hide, which is exactly why interviewers ask about it.
Rebalancing, and why it used to freeze the pipeline
When a consumer joins, leaves, or misses enough heartbeats to be presumed dead, Kafka reassigns partitions across the group. Under the old eager protocol every consumer stopped, surrendered all of its partitions, and waited for a fresh assignment. That stop-the-world pause got worse as the group grew. Anyone who ran Kafka at scale before 2024 has a story about a rolling deploy setting off back-to-back rebalances while the group spent more time reassigning than consuming.
KIP-848, the new consumer group protocol that became the default in Kafka 4.0 (March 2025), moves assignment logic to the broker-side group coordinator and makes rebalances incremental. Consumers keep the partitions they still own and hand over only the ones being moved. If your interviewer is current, noting that the coordinator now computes assignments and that cooperative rebalancing avoids the global pause is the kind of specific, dated detail that signals you’ve kept up with the project.
The three delivery guarantees and what each one costs
Every messaging interview converges here: at-most-once, at-least-once, exactly-once. The words are simple. The config sitting behind them is where people fumble.
At-least-once is the default you usually want. The producer retries on failure, so a record can land twice, and the consumer commits its offset after processing. If the consumer crashes after processing but before committing, it reprocesses on restart. Fine when your downstream write is idempotent (an upsert keyed by event id), a real problem when it isn’t (incrementing a balance, firing an email).
Exactly-once is real in Kafka, with a sharp edge. The idempotent producer (enable.idempotence=true, on by default since 3.0) stamps each record with a producer id and sequence number so the broker drops duplicates created by retries. That removes producer-side dupes within a partition. For end-to-end exactly-once you add transactions: the producer wraps its output writes and the consumer offset commit into one atomic transaction, and downstream consumers set isolation.level=read_committed so they never see records from an aborted transaction. This is what Kafka Streams runs under processing.guarantee=exactly_once_v2.
The catch every interviewer waits for: exactly-once only holds inside Kafka. The moment your consumer writes to a system that isn’t part of the Kafka transaction (Postgres, S3, a REST endpoint), you are back to at-least-once unless that sink dedupes on its own. Saying “we use exactly-once, so I don’t need idempotent writes to the database” is the sentence that ends the round.
Durability sits under all of it. acks=all makes the leader wait for every in-sync replica before acknowledging a write; pair it with min.insync.replicas=2 so a write is rejected rather than silently accepted when replicas fall behind. acks=1 acknowledges on the leader alone and loses data if that leader fails before the record replicates. acks=all with min.insync.replicas=2 on a replication factor of 3 is the standard “how do you avoid data loss” answer.
| Delivery guarantee | Producer configuration | Consumer offset commit | How you still lose or duplicate |
|---|---|---|---|
| At-most-once | acks=0 or acks=1, no retries | Commit offset before processing | Consumer dies after commit, before processing: message never handled |
| At-least-once | acks=all, retries enabled, enable.idempotence=true | Commit offset after processing | Consumer dies after processing, before commit: message reprocessed as a duplicate |
| Exactly-once (Kafka to Kafka) | Transactional producer, enable.idempotence=true | Offsets committed inside the transaction; consumers read with isolation.level=read_committed | Writing to a non-transactional external sink drops you back to at-least-once |
Where offsets bite: at-least-once by accident
enable.auto.commit=true is the default, and it commits offsets on a 5-second timer whether or not you finished processing. Say auto-commit fires right after poll() hands you a batch but before your handler writes the results, and then the consumer dies. On restart it resumes past records it never actually processed. That’s silent data loss wearing an at-least-once costume. The fix is to turn auto-commit off and commit manually once the work is done.
while (running) {
var records = consumer.poll(Duration.ofMillis(200));
for (var record : records) {
process(record); // finish the work first
}
consumer.commitSync(); // then commit the whole batch
}Commit after processing and a crash costs you reprocessing, not lost data. That’s the trade you want for anything that matters, and it’s why “at-least-once with an idempotent downstream” beats chasing exactly-once for most pipelines.
What changed once ZooKeeper was gone
If your Kafka knowledge froze a couple of years back, this is the update that dates you in the room. Kafka 4.0 removed ZooKeeper completely. Cluster metadata, controller election, and topic configuration now live in KRaft, Kafka’s own Raft-based consensus running on dedicated controller nodes. You can’t upgrade a ZooKeeper cluster straight to 4.0; you migrate it to KRaft on a 3.x release first, then upgrade.
Interviewers care because KRaft removes an entire second distributed system from the operational picture and makes controller failover faster, which feeds directly into the transaction coordinator recovery that exactly-once depends on. Asked “what does ZooKeeper do in Kafka,” the current answer is that in 4.0 it does nothing because it’s gone, followed by what replaced it. Reciting ZooKeeper’s old responsibilities as if they’re live is a quiet tell that you haven’t touched a recent cluster.
Reading consumer lag like an operator
Lag is the gap between a partition’s log-end offset and the group’s committed offset for it. Rising lag means you’re producing faster than you consume. The question that separates people who have run this from people who have only read about it: your lag is climbing on partition 7 alone while everything else stays flat, what’s happening? Even lag across all partitions points at under-provisioned consumers, so add more up to the partition count. Lag on one partition is almost always a hot key skewing traffic to it, or a single slow or poisoned record blocking the batch behind it. Which one it is changes the fix, and knowing to look at per-partition lag instead of the aggregate is the tell that you’ve debugged a real stall at 2am rather than memorized the definition.
Keep sharpening your system design:
