The whole course ▾

System Design · Lesson 16 of 39 · Week 4

Kafka and event streaming

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

One QuickBite order now matters to four teams: notifications, analytics, fraud and rewards. Today: Kafka. One event, many readers.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

One order event feeds analytics, notifications and fraud checks.

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

Four teams, one order

Four teams, one order
Order service
Queue
Notifications
Analytics
Fraud checks
Rewards
order placed
taken, deleted
me too?
me too?
me too?
send it four times?

Yesterday, a queue handed each order message to one notification worker, and deleted it once it was done. Today, more teams want the same message.

The order service puts each order on the queue, and the notifications worker takes it and deletes it. Now three more teams want every single order: analytics, to count sales; fraud checks, to spot stolen cards; and rewards, to give loyalty points. But a queue hands each message to one consumer, and then it's gone.

We could make the order service send four copies to four queues. But then every new team means changing the order service, and if one of the four sends fails, the teams disagree about what happened. Worse, when analytics finds a bug in last week's numbers, last week's messages are long gone. What we want is messages that stay, and that anyone can read.

A log, not a queue

A log, not a queue
Order
service
Orders
topic
append-only
log
Notifications
Analytics
Fraud
checks
Rewards
append
at 1,042
at 1,038
at 1,042
at 990
IN ONE LINE
Aqueuegiveseachmessage
toonereaderanddeletes
it;alogkeepsit,andevery
readerkeepsitsownplace.

Kafka's idea is simple, and it changes everything: don't delete a message when it's read. Keep it in a log.

Kafka keeps events in a log: an append-only list, where new events go on the end and nothing is ever edited. The order service appends each order to the orders topic. Each team reads the log at its own pace, and remembers its own position, called an offset. Notifications is at event one thousand and forty-two, analytics a few behind, fraud checks up to date, and rewards further back after a deploy. Reading removes nothing.

In one line: a queue gives each message to one reader and deletes it; a log keeps it, and every reader keeps its own place. Adding a fifth team now means nothing to the order service. The new team just starts reading. Under the hood, each partition is a file that only grows at the end, and disks are very fast at that, like the LSM trees from lesson ten.

Partitions and consumer groups

Partitions and consumer groups
Order
service
Partition
0
Partition
1
Partition
2
Fraud worker
reads 0
Fraud worker
reads 1
Fraud worker
reads 2
hash(customer)
1
Fraud consumer group
The rules
1
Order is kept within a partition
2
Each partition: one worker per
group
3
More partitions -> more workers

One log on one machine would soon be a bottleneck, so Kafka splits every topic.

A topic is split into partitions: separate logs that can live on different machines. When the order service writes an event, Kafka picks the partition from the event's key, here the customer ID, hashed, just like sharding in lesson twelve. So all of one customer's orders land in the same partition, in order. On the reading side, the fraud team runs a consumer group: three fraud workers sharing the work, and Kafka gives each partition to exactly one of them.

Three rules follow. Order is kept within a partition, never across them, which is why the key matters. Each partition is read by one worker per group, so a group can never have more busy workers than partitions. So to go faster, add more partitions, then more workers. And the analytics group reads the very same partitions, independently, from its own offsets. And Kafka copies each partition to other servers, usually three copies in all, so losing one server loses nothing.

Rewind the tape

Rewind the tape
Retention
Events kept for days: 7 by default
A new team starts from the beginning
A buggy reader rewinds and replays
EVENTS, NOT CALLS
Producerspublishfacts
("orderplaced");each
consumerdecideswhatto
doaboutthem.

Because nothing is deleted when it's read, Kafka can do something a queue can't. You can go back in time.

Kafka keeps events for a set time, seven days by default, or up to a set size, and then removes old ones in bulk. That unlocks two superpowers. A brand-new team can start from the beginning and catch up on a week of orders. And when analytics ships a bug and counts wrong, they fix the bug, rewind their offset to Monday, and replay the week. The other teams don't even notice.

This changes how services talk. The order service no longer calls notifications, analytics, fraud and rewards. It publishes a fact: order placed. Each team decides what to do about it, and deploys on its own schedule. That's event streaming, and it's how large companies keep dozens of teams from tripping over each other.

Queue or log?

Queue or log?
Queue, like SQS
Each message reaches one worker
Gone once it's done
No replay
Log, like Kafka
Every group reads everything
Kept for days: replay
Ordered within a partition
THE PRICE
Onestuckeventblocks
itswholepartition,and
orderholdsonlywithina
partition.

So is Kafka just a better queue? Not quite. They're built for different jobs.

A queue, like Amazon SQS, gives each message to one worker and forgets it once it's done, with simple retries and dead letters per message. Perfect for a pile of independent jobs, like sending texts. A log, like Kafka, lets every group read everything, keeps it for days so you can replay, and keeps order within a partition. For one event that many teams care about, the log wins. Many systems use both: Kafka for events, and queues for jobs.

The price: a worker reads its partition in order, so one event it can't process blocks everything behind it in that partition. You need a plan for it, like moving it to a separate dead-letter topic and carrying on. And running Kafka well is real work, which is why many teams pay for a managed version.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
PublisheventstoaKafka
topic,partitionedbykeyfor
order;eachteamreadswith
itsownconsumergroup.
Follow-ups to expect
->
How do you keep events in
order?
->
A consumer falls behind. Now
what?
->
Why Kafka and not a queue?
THE COMMON MISTAKE
Expectingglobal
order:Kafkaorders
eventsonlywithina
partition.

Kafka shows up whenever one event fans out to many services, or whenever you need a history you can replay.

Say it like this: we publish order events to a Kafka topic, partitioned by customer ID so each customer's events stay in order, and each team reads with its own consumer group. Then mention retention and replay, because that's the feature that makes a log worth it. If they ask about numbers: one partition handles thousands of events a second, and big clusters handle millions.

Then expect these. How do you keep events in order? Same key, same partition. A consumer falls behind: now what? Watch its lag, the gap between its offset and the end of the log, and add partitions and workers. And why Kafka and not a queue? Many readers, replay, and order per key.

The common mistake is promising that all events are processed in the order they happened. Kafka only promises order within a partition, so choose the key that keeps the events that matter together.

Quick check

Quick check
?
A topic has 4 partitions.
One consumer group runs 6
workers. How many are busy?
Four: one per partition. The
other two sit idle.
?
Analytics deployed a bug two
days ago and counted wrong.
How do they fix the numbers?
Fix the bug, rewind the group's
offset two days, and replay.
Remember
A log keeps events; readers
keep offsets
Partitions: order per key,
scale per partition
Retention means you can
replay
Quick check

A topic has 4 partitions. One consumer group runs 6 workers. How many are busy?

Show the answer

Four: one per partition. The other two sit idle.

Quick check

Analytics deployed a bug two days ago and counted wrong. How do they fix the numbers?

Show the answer

Fix the bug, rewind the group's offset two days, and replay.

Remember: a log keeps events, and each reader keeps its own offsets. Partitions give you order per key, and scale per partition. And retention means you can replay. Tomorrow, the bug that at-least-once delivery leaves behind: duplicates.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.