The whole course ▾

System Design · Lesson 37 of 39 · Week 8

Stream processing: an ad-click aggregator

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

Restaurants now pay QuickBite for every click on their promoted listing: a billion clicks a day, each one money. Today: stream processing.

The video for this lesson is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

Every click is money

Every click is money
Customer app
Click
service
Counts DB
one row per ad
click on ad
count + 1
a popular ad: one hot
row
The numbers
1B clicks a day ÷ 100,000 s =
10,000/s
Peak ~ 3 × = 30,000 clicks/s
Counts per ad, per minute, within a
minute
Billing: no click lost, none counted
twice

QuickBite starts selling ads. Restaurants can promote their listing to the top of the search results, and pay a few rupees for every click. Advertisers want to watch their clicks live, and the finance team wants every single one counted exactly.

A billion clicks a day is ten thousand a second, with a peak around thirty thousand. Advertisers want counts per ad, per minute, showing up on their dashboard within about a minute. And for billing, every click must be counted exactly once: a lost click is lost money, and a doubled one is an angry restaurant.

The naive design: the customer app reports a click, and the click service adds one to that ad's row in a counts DB. Thirty thousand updates a second, and a popular ad's row becomes the hot row from mock interview one: every click waiting for the one before it. And there's no history: if a count is wrong, there's nothing to recount from.

Clicks as a stream

Clicks as a stream
Customer
app
Click
service
Clicks
topic
Kafka,
keyed
by ad
Stream
processor
like Flink
Counts
store
ad, minute ->
clicks
Advertiser
dashboard
click
1
append
event
2
3
one total
a minute
4
read counts
WHY A STREAM
Eventsarekept,so
countscanberecomputed;
countinghappensin
memory,notinhotrows.

The fix is to stop counting at the moment of the click. Record the click as an event, and count the stream of events.

The click service now only records each click as an event, with a unique click ID and the time it happened, appended to a clicks topic in Kafka, partitioned by ad ID, from lesson sixteen. A stream processor, a system like Apache Flink, reads the topic and keeps a running count per ad, per minute, in memory. Once a minute it writes the totals to a counts store, and the advertiser dashboard reads from there. Thirty thousand clicks a second become a few thousand writes a minute.

Two wins. The raw events are kept in Kafka, and copied to cheap storage, so any count can be recomputed later, which billing will need. And counting happens in the stream processor's memory, spread across machines by ad ID, so a popular ad is just a bigger number in memory, not a row everyone queues for.

Windows

Windows
Kinds of window
->
Tumbling: fixed 1-minute buckets
->
Sliding: the last 5 minutes, every minute
->
Session: activity separated by gaps
STATE, SPLIT BY KEY
Theprocessorkeepsone
counterperadper
window,splitacross
machinesbyadID.

Counting a never-ending stream needs a way to cut it into pieces. Those pieces are called windows.

Tumbling windows are fixed, back-to-back buckets: twelve o'clock to twelve-oh-one, then twelve-oh-one to twelve-oh-two. Perfect for clicks per minute. Sliding windows overlap: the last five minutes, updated every minute, for a smoother live graph. And session windows group activity separated by quiet gaps, like one user's browsing session. Ad billing uses tumbling windows: every click belongs to exactly one minute.

Inside the processor, the state is simple: one counter per ad, per window. Because the topic is partitioned by ad ID, each machine owns the counters for its own ads, and never has to talk to the others. Adding machines spreads the ads further. It's lesson twelve's sharding, applied to a computation instead of a table.

Counted exactly once

Counted exactly once
Checkpoints
1
Every few seconds: save counters and offsets
together
2
Crash: restore the last checkpoint
3
Rewind Kafka to the saved offsets, replay
4
Write totals idempotently, by ad and minute
EXACTLY-ONCE, IN PRACTICE
Replayableinput+
checkpointedstate+
idempotentoutput=
countedexactlyonce.

Now the billing requirement. Machines crash. How does a stream processor avoid losing counts, or doubling them, when it restarts?

Every few seconds, the processor saves a checkpoint: all its counters, and its position in each Kafka partition, together, as one snapshot. If a machine crashes, the processor restores the last checkpoint, rewinds Kafka to the offsets saved with it, and replays the clicks since. The counters and the input go back to the same moment, so every click is counted once. And totals are written idempotently, keyed by ad and minute, so writing the same minute twice changes nothing.

That's lesson seventeen's honest exactly-once, built into a whole system: input you can replay, state you can restore to the same moment, and output that's safe to write twice. Events may be processed more than once after a crash. Their effect on the counts happens exactly once.

Late clicks

Late clicks
Late events
1
Count by when it happened, not arrived
2
A watermark: all clicks up to 12:04 are probably
in
3
Close a window once the watermark passes it
4
Very late? Correct the total later
A NIGHTLY RECOUNT
Thestreamgivesfast
counts;anightlybatch
recountfromrawevents
givesthebillingnumbers.

One more problem, and it's the subtle one. Clicks don't always arrive in order, or on time.

A customer clicks an ad in a lift, and the click reaches us two minutes later. Count it by when it happened, its event time, not when it arrived. The processor tracks a watermark: a moving line that says all clicks up to twelve-oh-four have probably arrived. When the watermark passes the end of a minute, that minute's window is closed, and its total written. And clicks that arrive later still are used to correct the total afterwards, or counted separately, never silently dropped.

And for money, add a safety net, like the reconciliation in mock interview five. Every night, a batch job recounts the whole day from the raw click events, after removing duplicate click IDs and clicks from known bots. The stream gives advertisers fast numbers during the day. The nightly recount gives the numbers restaurants are actually billed for.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
ClicksintoKafka,countedin
windowsbyastreamprocessor,
withcheckpoints,watermarks,
andanightlyrecount.
Follow-ups to expect
->
How do you avoid double
counting?
->
What about late clicks?
->
Bots clicking ads?
THE COMMON MISTAKE
Countingwithdatabase
incrementsperclick,and
noraweventsto
recountfrom.

Design an ad-click aggregator, or any real-time counter, is a classic senior question about streams.

Say it like this: click events go into Kafka, partitioned by ad. A stream processor like Flink counts them in one-minute tumbling windows by event time, using watermarks for late clicks, and checkpoints for exactly-once counting. Totals are written idempotently to a counts store. And a nightly batch recount from raw events produces the billing numbers.

Then expect these. How do you avoid double counting? Checkpoints, idempotent writes, and deduplication by click ID. What about late clicks? Event time, watermarks, and corrections. And bots clicking ads to drain a rival's budget? Filter known bots in the stream, and remove the rest in the nightly recount, before anyone is billed.

The common mistake: adding one to a database row for every click. It melts on popular ads, and when a count is questioned, there's nothing to recount from. Keep the events, and count the stream.

Quick check

Quick check
?
A click happened at 12:03:58
but arrived at 12:05:10.
Which minute is it counted in?
12:03: count by event time, not
arrival time.
?
The stream processor
crashes and restarts. Why
aren't clicks counted twice?
Checkpoints: counters and Kafka
offsets are restored to the
same moment, then replayed.
Remember
Keep the events; count the
stream
Windows, by event time, with
watermarks
Checkpoints, idempotent
output, nightly recount
Quick check

A click happened at 12:03:58 but arrived at 12:05:10. Which minute is it counted in?

Show the answer

12:03: count by event time, not arrival time.

Quick check

The stream processor crashes and restarts. Why aren't clicks counted twice?

Show the answer

Checkpoints: counters and Kafka offsets are restored to the same moment, then replayed.

Remember: keep the events, and count the stream, never with per-click increments. Count in windows by event time, with watermarks for late clicks. And checkpoints plus idempotent output give exactly-once counts, with a nightly recount for billing. Tomorrow, QuickBite goes international.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.