The whole course ▾
System Design · Lesson 5 of 39 · Week 2
Scaling from one server to many
Every day at 12:30, QuickBite's one server melts. Today: how to grow from one server to many, the way interviewers expect.
Got a doubt? Ask in the comments under the video.Subscribe
Lunch rush: QuickBite's single server melts.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
The lunch rush
Last week, QuickBite was one phone, one app server and one database, and our numbers said that was plenty for an average day. Today, lunch happens.
Here's QuickBite as it stands. Customers send requests to one app server, and it reads and writes the orders DB. At three in the afternoon this is fine: the server's processor sits at about twenty percent, and a menu loads in two hundred milliseconds.
Then lunch arrives. Between twelve thirty and one thirty, QuickBite gets five hundred requests a second, and this one server can handle about two hundred. The processor hits a hundred percent, requests wait in line, and a menu that took two hundred milliseconds now takes eight seconds. The server melts, and hungry people close the app and order somewhere else.
In an interview, describe it the way an engineer on call would see it. The processor is pinned at a hundred percent. Requests pile up in a queue, waiting for a free worker. Then the timeouts start: the phone gives up after ten seconds, the user taps retry, and that retry adds even more load. That spiral is why a melting server rarely recovers on its own.
Scale up or scale out
There are exactly two ways out of this, and interviewers expect you to know both.
Scale up means buying a bigger machine: say, going from eight processor cores to sixty-four, and from thirty-two gigabytes of memory to five hundred. It needs no code changes, which makes it a great first move. But it has a ceiling, the biggest machines cost many times more than ordinary ones, and it's still one machine, so one crash takes QuickBite down. Scale out means running many ordinary servers side by side, and adding more as traffic grows. Every large system works this way, but it asks one thing of your code.
So the rule of thumb is simple. Scale up first, because it's quick and cheap to try. Scale out when one machine isn't enough, or when one machine failing would take the whole business down. For QuickBite at lunch, both are true.
Stateless servers
Let's scale QuickBite out with three servers, and watch closely what breaks, because the thing that breaks here breaks in real companies too.
Customers now reach a load balancer, which hands each request to one of three copies of the app: app servers one, two and three. All three share the same orders DB. How the load balancer picks a server is tomorrow's lesson. Today's question is what the servers must stop doing.
Here's the catch. QuickBite keeps your login session in the server's memory. You log in through server one, and your next click lands on server two, which has never heard of you. You're logged out in the middle of your order. And if server one crashes, every session on it is gone too.
The fix is to make the servers stateless. Sessions move out of the servers into a shared session store, usually Redis. Now any server can answer any request, and losing a server loses nothing but its share of the work. Stateless servers are what make scaling out possible.
Sessions aren't the only state a server can hold. Anything that lives on just one server breaks scaling out. Sessions go to a shared store like Redis. Uploaded files go to object storage, which is lesson nine. And data cached in one server's memory moves to a shared cache, in lesson eight. The goal is a server you could throw away at any moment without anyone noticing.
The next bottleneck
We fixed the app tier. So where does QuickBite break next?
Count the arrows. Every app server still talks to one database. Each new server adds more load to it, and a single database usually tops out at a few thousand queries a second. We didn't remove the bottleneck. We moved it.
Much of this course is about that database. Caching takes reads off it, in lessons seven and eight. Replication keeps copies of it, in lesson eleven. And sharding splits it across machines, in lesson twelve.
In the interview
Scaling comes up in almost every system design interview. Here's how to handle it.
Say this: I'll keep the app servers stateless, with sessions in a shared store, so we can scale out behind a load balancer. One sentence, and the interviewer knows you understand why scaling out works.
Then expect these. Why not just buy a bigger server? Because it has a ceiling and it's still one point of failure. Where do sessions live now? In a shared store like Redis. And what breaks next? The database, which is exactly where you take the conversation.
Interviewers love one more step: how many servers? Use last week's math. At peak, QuickBite sees five hundred requests a second, and one server handles about two hundred. Five hundred divided by two hundred is two and a half, so round up to three. Then add one spare, so a single failure doesn't put you back at lunch-time meltdown. Four servers. That's called N plus one.
The common mistake is drawing ten app servers and one database, and calling it scaled. Interviewers notice. Always say what the extra servers do to the database.
Quick check
You add a second app server, and users get logged out on their next click. What went wrong?
Show the answer
Sessions lived in one server's memory. Move them to a shared session store.
Peak traffic is 900 requests a second, and each server handles 200. How many servers?
Show the answer
Five to carry the load, plus one spare: six.
Three things to remember. Scale up first, and scale out to grow. Keep app servers stateless, with sessions in a shared store. And the database is always the next bottleneck. QuickBite now survives lunch.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.