The whole course ▾
System Design · Lesson 34 of 39 · Week 7
Design a notification system
Order confirmed, rider assigned, food arriving: QuickBite sends 100 million notifications a day, by push, text and email. Today: notifications.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
A hundred million a day
Back in lesson fifteen, a slow SMS gateway took down QuickBite's checkout, and a queue fixed it. Today we grow that one queue into a complete notification system, the kind every big app runs.
The requirements. Send by push, text message, or email. Fill each message from a template, in the user's own language. Respect each user's preferences: some want texts, some don't, and nobody wants a discount at two in the morning. And never send the same notification twice, because a doubled your order is confirmed looks broken, and a doubled text costs money.
A hundred million notifications a day averages about a thousand a second. But they aren't spread evenly: dinner time, and a big sale announced to every user at once, create peaks ten times higher. And the channels differ in price: push notifications are free to send, while every text message costs money, so texts are kept for what truly matters.
And notice whose job is what. We don't deliver anything ourselves: Apple and Google deliver push, an SMS gateway delivers texts, and an email provider delivers email. Our system's job is deciding what to send, to whom, through which channel, at what time, and making sure it happens exactly once.
From event to message
Notifications start as events from other services. Here's the path from an event to a message ready to send.
Other services publish events, like rider assigned for order four-two-one, to Kafka, and the notification service reads them. For each one, it checks the user's settings in the preferences DB: which channels they allow, their quiet hours, their language. Then it fills the right message from templates: your rider Ravi is on the way, in English, Hindi or Tamil. Other services never need to know how notifications work; they just publish what happened.
So, for every event: first, have we seen this event before? If so, skip it, because Kafka delivers at least once. Then check preferences and quiet hours. Pick the channels: push for most things, a text as well for a one-time password. Then fill the template, and put the message on a queue for each channel it should go to.
A queue per channel
Why separate queues? Because the three providers fail and slow down at different times.
Each channel gets its own queue and its own workers: a push queue feeding Apple's and Google's push services, APNs and FCM; an SMS queue feeding the SMS gateway; and an email queue feeding the email provider. When the SMS gateway slows down at dinner, only the SMS queue grows. Push notifications keep flowing, untouched. It's the bulkhead from lesson twenty-one, built from queues.
And each channel's workers can be tuned to their provider. The SMS gateway might allow a hundred messages a second on our plan, so SMS workers never send faster than that. Push can go much faster. Separate queues let each channel scale, retry and slow down on its own terms.
Delivered, and only once
Now reliability: every message delivered, and none delivered twice.
Four rules. Dedupe each notification on its event, its user and its type, so a repeated event becomes one message: lesson seventeen's idempotency. If a provider errors, retry with backoff and jitter, from lesson twenty-one. For critical messages, like a one-time password, if push isn't delivered in time, fall back to a text. And messages to addresses that keep failing, like a deleted email account, go to dead letters, and the address is marked invalid.
How do we know it arrived? The providers tell us, through webhooks, like the payment provider in mock interview five: delivered, bounced, opened. Store those against each message. They power the fallback to text, clean up bad addresses, and tell the marketing team which messages people actually open.
Don't annoy people
A notification system can be perfectly reliable, and still make users switch notifications off forever. The last part is about respect.
Limit offers per user, say at most three a day, using a counter per user, like lesson nineteen's rate limiter. During quiet hours, hold offers until the morning, instead of dropping them, or waking anyone up. And bundle: five new offers from restaurants you follow becomes one notification, not five buzzes in a minute.
Most important of all, two lanes. Transactional messages, like a one-time password or your food has arrived, go through their own high-priority queues, are never rate limited, and ignore quiet hours. Marketing goes through separate queues, with limits. So a sale announced to fifty million users can never delay the password someone is waiting for.
In the interview
Design a notification system is a popular question, because it combines queues, idempotency, retries and rate limits in one design.
Say it like this: services publish events; a notification service checks preferences and quiet hours, fills templates, and deduplicates; then a queue per channel feeds workers that call each provider, with retries, backoff and fallbacks. Transactional and marketing messages travel in separate lanes, and marketing is rate limited per user.
Then expect these. The SMS provider goes down? Its queue grows while retries back off, a circuit breaker stops hammering it, and you can switch to a second provider. Fifty million users get one offer? A batch job puts them on the marketing queue gradually, at a rate the providers allow. And how do you avoid duplicates? Dedupe on event, user and type.
The common mistake: one queue for everything. The day marketing sends fifty million offers, every one-time password waits behind them, and nobody can log in. Separate lanes, always.
Quick check
The SMS gateway slows down. Do push notifications slow down too?
Show the answer
No: each channel has its own queue and workers.
The "rider assigned" event arrives twice. How does the customer get only one push?
Show the answer
Dedupe on the event ID, user and type before sending.
Remember: a queue per channel, so one provider's trouble stays contained. Dedupe, retry with backoff, and fall back for critical messages. And separate lanes, so marketing never delays a password. That's case study week: YouTube, Dropbox, a crawler, autocomplete, and notifications.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.