The whole course ▾
System Design · Lesson 16 of 39 · Week 4
Kafka and event streaming
One QuickBite order now matters to four teams: notifications, analytics, fraud and rewards. Today: Kafka. One event, many readers.
Got a doubt? Ask in the comments under the video.Subscribe
One order event feeds analytics, notifications and fraud checks.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
Four teams, one order
Yesterday, a queue handed each order message to one notification worker, and deleted it once it was done. Today, more teams want the same message.
The order service puts each order on the queue, and the notifications worker takes it and deletes it. Now three more teams want every single order: analytics, to count sales; fraud checks, to spot stolen cards; and rewards, to give loyalty points. But a queue hands each message to one consumer, and then it's gone.
We could make the order service send four copies to four queues. But then every new team means changing the order service, and if one of the four sends fails, the teams disagree about what happened. Worse, when analytics finds a bug in last week's numbers, last week's messages are long gone. What we want is messages that stay, and that anyone can read.
A log, not a queue
Kafka's idea is simple, and it changes everything: don't delete a message when it's read. Keep it in a log.
Kafka keeps events in a log: an append-only list, where new events go on the end and nothing is ever edited. The order service appends each order to the orders topic. Each team reads the log at its own pace, and remembers its own position, called an offset. Notifications is at event one thousand and forty-two, analytics a few behind, fraud checks up to date, and rewards further back after a deploy. Reading removes nothing.
In one line: a queue gives each message to one reader and deletes it; a log keeps it, and every reader keeps its own place. Adding a fifth team now means nothing to the order service. The new team just starts reading. Under the hood, each partition is a file that only grows at the end, and disks are very fast at that, like the LSM trees from lesson ten.
Partitions and consumer groups
One log on one machine would soon be a bottleneck, so Kafka splits every topic.
A topic is split into partitions: separate logs that can live on different machines. When the order service writes an event, Kafka picks the partition from the event's key, here the customer ID, hashed, just like sharding in lesson twelve. So all of one customer's orders land in the same partition, in order. On the reading side, the fraud team runs a consumer group: three fraud workers sharing the work, and Kafka gives each partition to exactly one of them.
Three rules follow. Order is kept within a partition, never across them, which is why the key matters. Each partition is read by one worker per group, so a group can never have more busy workers than partitions. So to go faster, add more partitions, then more workers. And the analytics group reads the very same partitions, independently, from its own offsets. And Kafka copies each partition to other servers, usually three copies in all, so losing one server loses nothing.
Rewind the tape
Because nothing is deleted when it's read, Kafka can do something a queue can't. You can go back in time.
Kafka keeps events for a set time, seven days by default, or up to a set size, and then removes old ones in bulk. That unlocks two superpowers. A brand-new team can start from the beginning and catch up on a week of orders. And when analytics ships a bug and counts wrong, they fix the bug, rewind their offset to Monday, and replay the week. The other teams don't even notice.
This changes how services talk. The order service no longer calls notifications, analytics, fraud and rewards. It publishes a fact: order placed. Each team decides what to do about it, and deploys on its own schedule. That's event streaming, and it's how large companies keep dozens of teams from tripping over each other.
Queue or log?
So is Kafka just a better queue? Not quite. They're built for different jobs.
A queue, like Amazon SQS, gives each message to one worker and forgets it once it's done, with simple retries and dead letters per message. Perfect for a pile of independent jobs, like sending texts. A log, like Kafka, lets every group read everything, keeps it for days so you can replay, and keeps order within a partition. For one event that many teams care about, the log wins. Many systems use both: Kafka for events, and queues for jobs.
The price: a worker reads its partition in order, so one event it can't process blocks everything behind it in that partition. You need a plan for it, like moving it to a separate dead-letter topic and carrying on. And running Kafka well is real work, which is why many teams pay for a managed version.
In the interview
Kafka shows up whenever one event fans out to many services, or whenever you need a history you can replay.
Say it like this: we publish order events to a Kafka topic, partitioned by customer ID so each customer's events stay in order, and each team reads with its own consumer group. Then mention retention and replay, because that's the feature that makes a log worth it. If they ask about numbers: one partition handles thousands of events a second, and big clusters handle millions.
Then expect these. How do you keep events in order? Same key, same partition. A consumer falls behind: now what? Watch its lag, the gap between its offset and the end of the log, and add partitions and workers. And why Kafka and not a queue? Many readers, replay, and order per key.
The common mistake is promising that all events are processed in the order they happened. Kafka only promises order within a partition, so choose the key that keeps the events that matter together.
Quick check
A topic has 4 partitions. One consumer group runs 6 workers. How many are busy?
Show the answer
Four: one per partition. The other two sit idle.
Analytics deployed a bug two days ago and counted wrong. How do they fix the numbers?
Show the answer
Fix the bug, rewind the group's offset two days, and replay.
Remember: a log keeps events, and each reader keeps its own offsets. Partitions give you order per key, and scale per partition. And retention means you can replay. Tomorrow, the bug that at-least-once delivery leaves behind: duplicates.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.