The whole course ▾
System Design · Lesson 21 of 39 · Week 5
When things fail: timeouts, retries and circuit breakers
At 8 PM, QuickBite's payment provider slows down, and within minutes the whole app is frozen. Today: designing for failure.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
The payment provider slows down during QuickBite's dinner rush.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
One slow dependency
Yesterday we split QuickBite into services that call each other over the network. Tonight, one of those calls gets slow.
Eight in the evening, the dinner rush. The mobile app goes through the API gateway to the orders service, which calls the payments service, which calls the outside payment provider. Normally the provider answers in about two hundred milliseconds. Tonight, it starts taking thirty seconds. It hasn't crashed. It's just slow. And at dinner time, thousands of payments a minute flow through it.
Watch it spread. Each payment request now holds a thread for thirty seconds, so the payments service runs out of threads, and every new request is stuck. The orders service waits on payments, so its threads fill up too, including the ones serving menus and order history, which have nothing to do with payments. Within minutes, the whole app is frozen. That's a cascading failure. And notice: slow did more damage than dead would have.
Timeouts everywhere
The first rule of calling anything over a network: never wait forever.
Every network call gets a timeout, with no exceptions: database queries, other services, outside providers. Set it from real numbers, just above the slowest normal answer: if the provider's ninety-ninth percentile, from lesson two, is one second, then two seconds is reasonable, not thirty. Inner calls get shorter timeouts than the calls waiting on them. And when time's up, give up, return an error, and free the thread for someone else.
A timeout turns a slow failure into a fast one. The customer sees: payment couldn't be completed, please try again, after two seconds, instead of a spinner for thirty. And our threads stay free for everyone browsing menus. Many libraries have no timeout by default, or a very long one, so setting it is your job. Python's popular requests library, for example, waits forever unless you pass a timeout.
Retry, carefully
Many failures are brief: one lost packet, one server restarting. So we retry. But careless retries make things much worse.
Four rules. Retry only what's safe to repeat: idempotent operations, or requests carrying an idempotency key, from lesson seventeen. Back off exponentially: wait a hundred milliseconds, then two hundred, then four hundred. Add jitter, a random spread to each wait, so a thousand clients that failed at the same moment don't all retry at the same moment. And cap it: two or three tries, then give up.
Here's why the cap matters. If each of three layers retries three times, one customer's request can become twenty-seven calls to the payment provider, exactly when it's struggling. Retries meant to help become the attack. So retry at one layer only, and keep a retry budget: for example, no more than one retry for every ten normal requests.
The circuit breaker
Timeouts and retries still keep calling a provider that's clearly down. A circuit breaker notices, and stops.
A circuit breaker wraps calls to one dependency, like the breaker in your home's fuse box. Closed: calls flow normally, and it counts failures. When too many fail, say half of the last fifty, it trips open: for the next thirty seconds, calls fail instantly, without even trying. Then it goes half-open, and lets a few calls through as a test. If they succeed, it closes again. If not, it opens for another thirty seconds.
On QuickBite's board, the breaker lives inside the payments service, around the provider. When the provider's failures trip it, payments answers in a millisecond: provider unavailable. No threads pile up. The provider gets room to recover, instead of a flood of retries. And the rest of the app stays fast, because nothing is waiting anymore. You rarely write one yourself: libraries like Resilience4j, or a service mesh, provide breakers ready-made.
Contain it, and degrade
Two more ideas complete the toolkit: walls between failures, and a plan B for when something is down.
A bulkhead is named after the watertight walls inside a ship, which stop one flooded section from sinking the whole thing. In software, give each dependency its own pools of threads and connections. Payment calls can use at most fifty threads; menus have their own. When the provider slows down, only the payments pool fills up, and menus and order history stay unaffected. Fifty threads waiting on a dead provider is a loss you can afford. Every thread in the app isn't.
And decide what to do when something is down, before it happens. If recommendations are down, show the most popular dishes instead. If ratings are slow, show the menu without stars. If online payments are down, offer cash on delivery. A smaller app that works beats a complete app that doesn't, and customers barely notice the difference. Agree these with the product team in advance, and test them, because the night you need them is too late.
In the interview
Interviewers love to point at one box on your diagram and ask: what if this gets slow?
Have this answer ready: timeouts on every call, set from real latency; retries with backoff and jitter, capped, only where safe; a circuit breaker and a bulkhead around each dependency; and a fallback, so the user still gets something useful. Then say what the user sees, because that's what the business cares about.
Then expect these. What timeout would you choose? Just above the dependency's normal slowest, measured, not guessed. Won't retries make it worse? Not if they're capped, jittered and done at one layer, with a budget. And what does the user see? A clear message, or a fallback, within a couple of seconds, never a frozen screen.
The common mistake is designing only for the day everything works. Point at every arrow on your own diagram and ask: what if this is slow? Interviewers will, and the candidates who ask first stand out.
Quick check
Three layers each try 3 times. One failing request becomes how many calls at the bottom?
Show the answer
27: 3 × 3 × 3. Retry at one layer only.
The provider is down. Calls wait 2 seconds to time out, 500 times a second. What helps most?
Show the answer
A circuit breaker: fail at once, stop calling, and let it recover.
Remember: slow is worse than dead, so put timeouts on everything. Retry with backoff and jitter, and a cap. And contain failures with circuit breakers, bulkheads and fallbacks. Tomorrow, an order that touches three services, and has to either finish everywhere or nowhere.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.