The whole course ▾
System Design · Lesson 19 of 39 · Week 4
Rate limiting
At 3 AM, a bot tries ten thousand passwords a minute on QuickBite's login. Today: rate limiting, and how to stop a flood.
Got a doubt? Ask in the comments under the video.Subscribe
A bot hammers QuickBite's login.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
A flood at the front door
Everything we built this week makes QuickBite faster and sturdier. Today we protect it, starting at the front door: the login page.
At three in the morning, a bot starts guessing passwords: ten thousand login attempts a minute, taken from a list of leaked passwords, through the load balancer to the login service. Each attempt checks a password hash from the users DB, and password hashing is slow on purpose, so every guess costs real processor time. Real customers trying to log in for a late-night order wait, then time out.
Two things are at risk: the service, which is drowning, and the accounts, because one of those guesses may be right. And floods aren't always attacks. A buggy app stuck in a retry loop, or a partner calling our API too eagerly, can do the same damage by accident. So we need a rule: no single caller gets more than its fair share, and we enforce it before the expensive work starts.
The token bucket
That rule is called rate limiting. The most common way to do it is a bucket of tokens.
Picture each caller, say each IP address, holding a bucket of ten tokens. Every request takes one token. Tokens refill at a steady rate, one a second, until the bucket is full again. And if the bucket is empty, the request is rejected straight away, with status four hundred and twenty-nine: too many requests. A real customer never notices. A bot hits the wall within seconds.
It's popular because it matches how people behave. A quick burst of requests is fine, because the bucket was full. But over time, nobody can go faster than the refill rate. There are only two numbers to tune, the bucket size and the refill rate, and very little to store: for each caller, a token count and the time it was last updated.
Counting in windows
The other family of limiters counts requests in windows of time. They're simpler to explain, and one of them has a sneaky flaw.
A fixed window counts per clock minute: at most sixty in the minute starting at twelve o'clock, then the count resets. Simple. But a client can send sixty at eleven fifty-nine and fifty-nine seconds, and sixty more right at twelve: a hundred and twenty in two seconds. A sliding window always counts the last sixty seconds from right now, so there's no edge to exploit. A common trick estimates it cheaply, from the current and the previous minute's counts.
So which one? A token bucket for most APIs, because it allows natural bursts. A sliding window when the limit must be strict, like a paid plan's quota. And a fixed window when simple is good enough. In an interview, name one, and say why it fits.
Where the limit lives
Next: where to check the limit, and how ten servers agree on one count.
Check limits at the edge, in the API gateway that sits in front of every service, so rejected requests cost almost nothing and never reach the services. But QuickBite runs ten gateway servers, and a bot's requests are spread across all of them. If each kept its own count, the bot would get ten times the limit. So the counts live in one shared place: Redis, the same in-memory store we used for caching in lesson eight.
Each check is one quick Redis operation, under a millisecond: take a token for this caller. It must be a single atomic step, or a small script that runs as one, so two gateways can't both take the last token. And if Redis itself is slow or down? Most systems fail open: they let requests through, and lean on other defences, rather than lock every customer out because a limiter broke.
And choose what you count per. Per IP address for anonymous traffic, like the login page. Per user, or per API key, once someone is logged in, so a whole office behind one IP isn't punished. And per endpoint: login gets a much stricter limit than browsing menus, because one is expensive and dangerous, and the other is cheap and cached.
Saying no, politely
A rate limiter says no a lot. How it says no matters, for honest clients and for attackers.
Answer with status four hundred and twenty-nine, so clients know exactly what happened. Add a Retry-After header, wait thirty seconds, so honest apps back off instead of hammering, with the exponential backoff from lesson fifteen. And many APIs add headers saying how many requests are left in the current window, so well-behaved clients can slow down before they ever hit the limit.
One warning. A serious attacker uses thousands of IP addresses, a few guesses each, and slips under every per-IP limit. So logins also get limits per account: five failures, then slow down or lock for a while. And after repeated failures, a CAPTCHA, which is cheap for a human and expensive for a bot.
In the interview
Rate limiting comes up as a full question, design a rate limiter, and as a detail in almost every public API.
Say it like this: a token bucket per user, or per IP for anonymous traffic, checked at the API gateway, with counters in Redis so every gateway server shares them, and a four-twenty-nine with Retry-After when the bucket is empty. That covers the algorithm, the place, the shared state, and the client.
Then expect these. The bot uses ten thousand IP addresses? Limits per account, and CAPTCHAs. What if Redis is down? Fail open, with a rough local limit on each gateway as a backstop. And different limits per plan, free versus paid? Key the bucket by API key, and give each plan its own bucket size and refill rate.
The common mistake: each server counting on its own. With ten servers behind a load balancer, the limit you thought you set is ten times too generous. Shared counters, or nothing.
Quick check
A limit of 100 a minute, in fixed windows. What's the most a client can send in 2 seconds?
Show the answer
200: 100 at the end of one minute, and 100 at the start of the next.
A bucket of 10 tokens, refilling 1 a second, is full. 15 requests arrive at once. How many pass?
Show the answer
10: one per token. The other 5 get a 429.
Remember: a token bucket allows bursts and caps the average. Check limits at the gateway, with shared counts in Redis. And say no politely: four-twenty-nine, with Retry-After. That's week four: queues, Kafka, idempotency, live updates, and rate limiting.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.