The whole course ▾
System Design · Week 4 revision
Week 4 revision: Async
Async week in ten minutes: queues, Kafka, idempotency, live updates and rate limits, then six questions to test yourself.
Got a doubt? Ask in the comments under the video.Subscribe
Lessons 15 and 16Queues and logs
This week was about the work that doesn't have to happen while the customer waits, starting with a text message that took down checkout.
Lesson fifteen moved slow work off the request. The order service drops a job on a queue, answers the customer straight away, and workers do the slow part. Delivery is at least once: a job that isn't acknowledged in time reappears, retries back off, and after a few failures it moves to dead letters. Lesson sixteen added a log. The orders topic in Kafka keeps every event for days, partitioned by customer, and each team's consumer groups read all of it, at their own pace, and can rewind.
The interview lines: queue the work the user doesn't wait for, with retries, backoff and a dead-letter queue. And when many services need the same events, publish them to Kafka, partitioned by key so each customer's events stay in order.
Lesson 17Duplicates made harmless
At-least-once delivery has a price, and on Wednesday a double-tapped Pay button showed us exactly what it is.
Lesson seventeen, in three lines. Duplicates are guaranteed: a lost reply makes every sender retry. Idempotency keys make client retries safe: the payment service stores each key with its result, and a repeated key gets the saved result, not a second charge. And an outbox, saving the event in the same transaction as the data, plus consumers that skip event IDs they've already seen, makes events safe from end to end.
The interview line: exactly-once is at-least-once delivery, plus idempotent processing. Say that, and you've answered the follow-up before it's asked. Kafka's own exactly-once feature covers reading and writing inside Kafka; the moment you charge a card or send an email, you're back to idempotency keys.
Lesson 18Live updates
Then QuickBite's map came alive, and the server had to talk first.
Lesson eighteen compared four ways to push news: short polling, long polling, server-sent events and WebSockets. For live tracking with chat, WebSockets. The rider app sends GPS to the location service, which publishes on the order's channel in pub/sub. The connection servers that subscribed push it down the open socket to the customer app. Sockets live on one server, and pub/sub finds it.
The interview line: WebSockets through dedicated connection servers, with pub/sub routing each update to the right server; server-sent events when updates only flow one way; and polling when they're rare. And plan for reconnects: mobile connections drop all the time, so on reconnect the app fetches the latest state, then carries on live.
Lesson 19Rate limiting
And on Friday, a bot hammered the login page at three in the morning.
Lesson nineteen, in three lines. A token bucket per caller: a few tokens for bursts, refilled at a steady rate that caps the average. Check it at the API gateway, with the counts shared in Redis, because ten servers counting alone allow ten times the limit. And answer four-twenty-nine with Retry-After, while logins also get limits per account, and a CAPTCHA, because attackers use thousands of IPs.
The interview line: a token bucket per user or IP, checked at the gateway, counted in Redis, answering four-twenty-nine with Retry-After. And if Redis goes down, fail open, with a rough local limit on each gateway as a backstop, rather than locking every customer out.
What's nextNext week: architecture
QuickBite is now one big service doing everything. Next week, we split it, and learn how a system of many services survives its own failures.
Microservices: when to split a monolith, and how services find each other. Timeouts, retries and circuit breakers, for the night the payment provider slows down. Sagas, for an order that spans three services. Unique IDs across many shards. And logs, metrics and traces, to answer: why did that order take nine seconds?
Test yourself
Checkout sends the receipt email itself, and slows down whenever email is slow. Fix?
Show the answer
Queue it: put the receipt on a queue and answer the customer straight away.
A Kafka topic has 8 partitions, and a consumer group has 3 workers. How is the work split?
Show the answer
Two or three partitions each: every partition goes to one worker.
The rewards worker receives the same "order placed" event twice. How do you avoid double points?
Show the answer
Record each event ID with the work, and skip IDs already seen.
Why use an outbox, instead of saving the order and then publishing the event?
Show the answer
A crash between the two loses the event; the outbox saves both in one transaction.
Updates flow only from server to browser, every few seconds. WebSockets, or something simpler?
Show the answer
Server-sent events: one-way is all it needs.
Ten gateways each allow 100 requests a minute per user, counted locally. What's the real limit?
Show the answer
Ten times higher: up to 1,000 a minute. Share the counters in Redis.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.