“Design something like Kafka, or SQS, pick whichever you know better.” That’s how the prompt usually lands, and how you scope it in the first three minutes says more than the forty that follow. Most candidates draw three boxes (producer, broker, consumer), connect them with arrows, and freeze the moment the interviewer asks what happens when a broker dies with a half-written batch. The whole question lives in that failure detail, so head there instead of ending there.
Pin down what “delivered” means before drawing anything
There are three delivery semantics and they are nowhere near equal in difficulty. At-most-once means the broker sends and forgets, so if the consumer crashes the message is gone. At-least-once means the broker keeps redelivering until the consumer confirms, so nothing is lost but duplicates are possible. Exactly-once means every message lands and takes effect one time, which sounds like the obvious thing to want and is the option that quietly eats the rest of your interview if you promise it casually.
The answer an interviewer wants to hear is at-least-once with idempotent consumers. You accept that a message can arrive twice, and you make the consumer’s handling safe to repeat, usually by keying the side effect on a message id or a business key so the second attempt is a no-op. That combination gives you the practical effect of exactly-once without a distributed transaction spanning your broker and every downstream store. Say “exactly-once” without saying “idempotency” and you’ll spend ten minutes explaining a two-phase commit you never needed.
While you’re scoping, get numbers on the table. Ask for target throughput (say 1M messages/sec at peak, 1KB each, so roughly 1GB/sec of writes), retention (hours, days, replayable for a week?), and how many independent consumer groups read the same stream. Those three answers decide whether you’re building a durable log or a transient work queue, and they move the storage bill by an order of magnitude.
The log and the queue are two different data structures
This is the fork that separates a real answer from box-drawing. A classic broker like RabbitMQ or SQS treats a message as a row that gets deleted once a consumer acknowledges it. A log-based system like Kafka or Pulsar treats the topic as an append-only file: consumers read by position, and the broker deletes nothing until a time or size threshold passes. Same word, “queue,” two very different machines underneath.
The delete-on-ack model is easier to reason about and hands you per-message visibility timeouts and dead-letter routing for free, which is why SQS is the default glue for a lot of AWS backends. The append-only log wins the moment you have multiple readers or want replay: three teams can each consume the same order-events topic at their own pace, and a consumer that shipped a bug can rewind its offset to yesterday and reprocess. The cost is that you, not the broker, track what you’ve read, and that bookkeeping is where most of the follow-up questions go.
Partitions are both the ordering unit and the scaling unit
You can’t keep total order across a topic spread over twenty machines without funneling every write through one of them, which caps throughput at whatever a single broker can do. So you give up global order and keep order where it actually matters. A topic splits into partitions, each partition is an ordered log on one broker (plus its replicas), and you route a message to a partition by hashing a key. Pick user_id as the key and every event for a given user stays ordered on one partition while different users spread across the cluster.
That one decision answers a stack of follow-ups at once. Throughput scales by adding partitions. Ordering holds within a key, not across keys. And the partition count sets a ceiling on parallelism for a consumer group, because a partition is read by at most one consumer in the group at a time. Interviewers push here: “You have 10 partitions and 15 consumers, what happens?” Five consumers sit idle. Repartitioning later is painful because it changes the hash routing, so you overprovision partitions up front.
| Message system | Storage model | Ordering it guarantees | Default delivery semantics | What you add to scale throughput |
|---|---|---|---|---|
| Apache Kafka | Append-only log, retained by time or size | Per partition (per key) | At-least-once | More partitions and brokers |
| Apache Pulsar | Append-only log on tiered storage (BookKeeper) | Per partition | At-least-once (effectively-once available) | More partitions and bookies |
| RabbitMQ (classic queue) | Delete on consumer ack | Per queue (FIFO) | At-least-once | More consumers, sharded queues |
| AWS SQS Standard | Hosted, delete on ack | None (best-effort) | At-least-once | Automatic, no ordering |
| AWS SQS FIFO | Hosted, delete on ack | Per message group id | Exactly-once within a 5-minute dedup window | More message groups |
Replication, and what “acked” is allowed to mean
A partition lives on one broker, but it can’t only live there, or a disk failure loses data. Each partition has replicas: one leader that takes all reads and writes, and followers that copy the leader’s log. Kafka calls the set of replicas that are caught up the in-sync replicas, or ISR. The knob the interviewer wants you to name is how many replicas must confirm a write before the producer calls it done.
Confirm after the leader alone writes and you’re fast, but you lose acknowledged messages if that leader dies before a follower copies them. Require the full ISR and you survive a broker loss with zero data loss, at the price of higher write latency and a stall whenever a follower falls behind. The middle setting, leader plus a majority, is what most production clusters run. Put the numbers into words: acks=all with a minimum ISR of 2 on a replication factor of 3 tolerates one broker down while still confirming writes.
At-least-once is a promise about your consumers too
Back to delivery, now with the machinery to make it concrete. A consumer reads a batch, processes it, then commits its offset (its read position) back to the broker. Commit the offset before processing and a crash means you skip messages, which is at-most-once. Process first, then commit, and a crash after processing but before the commit reprocesses the batch on restart, which is at-least-once. There is no third ordering that hands you exactly-once for free, which is the whole reason idempotency carries the weight.
When someone insists on real exactly-once, the straight answer is that it exists inside a single system’s boundary and not across it. Kafka can do exactly-once from topic to topic using an idempotent producer plus transactional offset commits, because the read, the write, and the offset update all sit in Kafka. The moment the side effect is a row in Postgres or a charge on a credit card, the broker can’t make that atomic with its own offset commit, and you’re back to at-least-once plus a dedup key on the consumer side.
Consumer groups and the rebalance that stalls everyone
A consumer group is how you scale reading: partitions get divided among the group’s members, each partition assigned to exactly one member. Add a consumer and the group rebalances, reassigning partitions. That rebalance is a real operational cost people forget. Under a naive protocol every consumer stops, gives up its partitions, and waits for the new assignment before anyone reads again, so a rolling deploy of your consumers can pause the whole group repeatedly. Cooperative rebalancing hands off only the partitions that actually move, and saying you’d reach for it is the kind of specific that reads as experience rather than memorization.
Where do the committed offsets live? Not in the consumer, or a crash loses them. They go back into the broker, in Kafka’s case a compacted internal topic keyed by group and partition. That keeps the source of truth for “what has this group read” inside the same durable, replicated system as the data itself, so recovery after a consumer restart is just reading the last committed position.
Where messages go when nothing works
Two failure modes close out most rounds. A poison message that throws every time it’s processed will, under at-least-once, redeliver forever and wedge the partition behind it. You cap the retry count and route the message to a dead-letter queue for a human or a separate job to inspect. And when producers outrun consumers, the log grows; retention policy is your release valve, dropping messages older than the window or past a size cap. If a lagging consumer group needs data that already aged out, that’s genuine data loss you surface as a tradeoff, not a bug you paper over. Setting retention to cover your worst realistic consumer outage, a few days rather than a few hours, is cheap insurance.
Follow-ups worth rehearsing before you walk in:
- “A consumer processes a message then crashes before committing. What does its next poll return?”
- “How do you keep every event for one user ordered while scaling to a hundred brokers?”
- “Producers write 1GB/sec, consumers read 600MB/sec. What happens over the next hour, and what breaks first?”
- “You need to reprocess yesterday’s traffic after a bug shipped. Walk me through it.”
The people who struggle in this round rarely fail because they forgot a component. They fail because they promised exactly-once in the second minute and spent the rest of the hour trying to pay for it.
Keep sharpening your system design:
