The whole course ▾

System Design · Lesson 5 of 39 · Week 2

Scaling from one server to many

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

Every day at 12:30, QuickBite's one server melts. Today: how to grow from one server to many, the way interviewers expect.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

Lunch rush: QuickBite's single server melts.

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

The lunch rush

The lunch rush
Customers
App server
Orders DB
requests
SQL
500 requests/s at lunch
CPU at 100%
Signs a server is melting
CPU pinned at 100%
Requests waiting in a queue
Timeouts and errors climbing

Last week, QuickBite was one phone, one app server and one database, and our numbers said that was plenty for an average day. Today, lunch happens.

Here's QuickBite as it stands. Customers send requests to one app server, and it reads and writes the orders DB. At three in the afternoon this is fine: the server's processor sits at about twenty percent, and a menu loads in two hundred milliseconds.

Then lunch arrives. Between twelve thirty and one thirty, QuickBite gets five hundred requests a second, and this one server can handle about two hundred. The processor hits a hundred percent, requests wait in line, and a menu that took two hundred milliseconds now takes eight seconds. The server melts, and hungry people close the app and order somewhere else.

In an interview, describe it the way an engineer on call would see it. The processor is pinned at a hundred percent. Requests pile up in a queue, waiting for a free worker. Then the timeouts start: the phone gives up after ten seconds, the user taps retry, and that retry adds even more load. That spiral is why a melting server rarely recovers on its own.

Scale up or scale out

Scale up or scale out
Scale up (vertical)
One bigger machine
No code changes
Has a ceiling, and a price cliff
Still one machine to fail
Scale out (horizontal)
Many ordinary machines
Add more as you grow
Survives one machine dying
Needs stateless app servers
THE RULE OF THUMB
Scaleupfirst:it'squick.
Scaleoutwhenone
machineisn'tenough,or
can'tbeallowedtofail.

There are exactly two ways out of this, and interviewers expect you to know both.

Scale up means buying a bigger machine: say, going from eight processor cores to sixty-four, and from thirty-two gigabytes of memory to five hundred. It needs no code changes, which makes it a great first move. But it has a ceiling, the biggest machines cost many times more than ordinary ones, and it's still one machine, so one crash takes QuickBite down. Scale out means running many ordinary servers side by side, and adding more as traffic grows. Every large system works this way, but it asks one thing of your code.

So the rule of thumb is simple. Scale up first, because it's quick and cheap to try. Scale out when one machine isn't enough, or when one machine failing would take the whole business down. For QuickBite at lunch, both are true.

Stateless servers

Stateless servers
Customers
Load
balancer
App
server 1
App
server 2
App
server 3
Orders
DB
your login lives here
Who are you?
Session
store
Redis
Make each server forgettable
Sessions: a shared store like
Redis
Uploaded files: object storage,
lesson 9
Cached data: a shared cache,
lesson 8

Let's scale QuickBite out with three servers, and watch closely what breaks, because the thing that breaks here breaks in real companies too.

Customers now reach a load balancer, which hands each request to one of three copies of the app: app servers one, two and three. All three share the same orders DB. How the load balancer picks a server is tomorrow's lesson. Today's question is what the servers must stop doing.

Here's the catch. QuickBite keeps your login session in the server's memory. You log in through server one, and your next click lands on server two, which has never heard of you. You're logged out in the middle of your order. And if server one crashes, every session on it is gone too.

The fix is to make the servers stateless. Sessions move out of the servers into a shared session store, usually Redis. Now any server can answer any request, and losing a server loses nothing but its share of the work. Stateless servers are what make scaling out possible.

Sessions aren't the only state a server can hold. Anything that lives on just one server breaks scaling out. Sessions go to a shared store like Redis. Uploaded files go to object storage, which is lesson nine. And data cached in one server's memory moves to a shared cache, in lesson eight. The goal is a server you could throw away at any moment without anyone noticing.

The next bottleneck

The next bottleneck
What comes next
->
Fewer reads: caching, lessons 7 and 8
->
Copies of the data: replication, lesson 11
->
Split the data: sharding, lesson 12
Stateless servers
Customers
Load
balancer
App
server 1
App
server 2
App
server 3
Orders
DB
your login lives here
Who are you?
Session
store
Redis
every request ends here
Make each server forgettable
Sessions: a shared store like
Redis
Uploaded files: object storage,
lesson 9
Cached data: a shared cache,
lesson 8

We fixed the app tier. So where does QuickBite break next?

Count the arrows. Every app server still talks to one database. Each new server adds more load to it, and a single database usually tops out at a few thousand queries a second. We didn't remove the bottleneck. We moved it.

Much of this course is about that database. Caching takes reads off it, in lessons seven and eight. Replication keeps copies of it, in lesson eleven. And sharding splits it across machines, in lesson twelve.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
I'llkeeptheappservers
stateless,withsessionsina
sharedstore,sowecanscale
outbehindaloadbalancer.
Follow-ups to expect
->
Why not just buy a bigger server?
->
Where do sessions live now?
->
What breaks next?
How many servers?
Peak: 500 requests a second
One server: about 200 a second
500 ÷ 200 = 2.5, so 3 servers
One spare for failures: 4 (N + 1)
THE COMMON MISTAKE
Addingappserversand
forgettingthedatabase
behindthem.

Scaling comes up in almost every system design interview. Here's how to handle it.

Say this: I'll keep the app servers stateless, with sessions in a shared store, so we can scale out behind a load balancer. One sentence, and the interviewer knows you understand why scaling out works.

Then expect these. Why not just buy a bigger server? Because it has a ceiling and it's still one point of failure. Where do sessions live now? In a shared store like Redis. And what breaks next? The database, which is exactly where you take the conversation.

Interviewers love one more step: how many servers? Use last week's math. At peak, QuickBite sees five hundred requests a second, and one server handles about two hundred. Five hundred divided by two hundred is two and a half, so round up to three. Then add one spare, so a single failure doesn't put you back at lunch-time meltdown. Four servers. That's called N plus one.

The common mistake is drawing ten app servers and one database, and calling it scaled. Interviewers notice. Always say what the extra servers do to the database.

Quick check

Quick check
?
You add a second app server, and
users get logged out on their
next click. What went wrong?
Sessions lived in one server's
memory. Move them to a shared
session store.
?
Peak traffic is 900 requests a
second, and each server handles
200. How many servers?
Five to carry the load, plus one
spare: six.
Remember
Scale up first, scale out to
grow
Stateless servers, shared
sessions
The database is the next
bottleneck
Quick check

You add a second app server, and users get logged out on their next click. What went wrong?

Show the answer

Sessions lived in one server's memory. Move them to a shared session store.

Quick check

Peak traffic is 900 requests a second, and each server handles 200. How many servers?

Show the answer

Five to carry the load, plus one spare: six.

Three things to remember. Scale up first, and scale out to grow. Keep app servers stateless, with sessions in a shared store. And the database is always the next bottleneck. QuickBite now survives lunch.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.