The whole course ▾

System Design · Lesson 6 of 39 · Week 2

Load balancers

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

QuickBite now runs three app servers. But who decides which one gets your order? Today: load balancers, from first request to failover.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

QuickBite puts a load balancer in front of three app servers.

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

One address in front

One address in front
Customers
Load balancer
api.quickbite.com
App server
1
App server
2
App server
3
every
request
What it gives you
One address for the whole fleet
Load spread across servers
Dead servers skipped
automatically
WHAT IT COSTS
Aboutonemillisecondper
request,andoneboxcan
passontensofthousands
ofrequestsasecond.

Yesterday we scaled QuickBite out to three stateless app servers and drew a box in front of them. Today, we open that box.

Customers never talk to an app server directly. DNS points api dot quickbite dot com at the load balancer, so every request arrives there first. Its job is to pass each request to one healthy server: app servers one, two and three. The servers behind it can change at any time, and the customers never notice.

That one box gives you three things. A single address, so the app never needs to know how many servers exist. Load spread evenly, so no server melts while another sits idle. And dead servers skipped automatically, which we'll come back to in a minute.

And what does it cost? One extra hop adds about a millisecond to each request, which nobody notices. A single load balancer can pass on tens of thousands of requests a second, so for QuickBite's five hundred at lunch, it's barely awake.

Three ways to choose

Three ways to choose
How it picks a server
1
Round robin: each server in
turn
2
Least connections: the least
busy one
3
Hashing: same key, same server
Least connections, live
Server 1: 12 requests in flight
Server 2: 3 requests in flight
Server 3: 9 requests in flight
Next request -> server 2
WHICH ONE, WHEN
Roundrobinwhenrequestsare
alike,leastconnectionswhen
someareslow,hashingwhena
usermuststicktooneserver.

So how does the load balancer pick a server for each request? There are three answers you should know by name.

First, round robin: server one, then two, then three, then back to one. It's simple and works well when every request costs about the same. Second, least connections: send the request to the server with the fewest requests in flight right now, which handles a mix of quick and slow requests far better. Third, hashing: compute a hash of something like the user's ID, so the same user always lands on the same server.

Here's least connections at one instant during lunch. Server one has twelve requests in flight, server two has three, and server three has nine. The next request goes to server two, the least busy, even if it was server two's turn a moment ago. The load balancer counts open connections, so this costs it almost nothing.

For QuickBite, browsing a menu takes milliseconds, but checking out with a payment can take seconds. Requests aren't alike, so least connections is the better default. Hashing matters when a server holds something for a user, like a cache, and we'll see its smarter version, consistent hashing, in lesson thirteen.

Health checks

Health checks
A GOOD HEALTH CHECK
Checkswhatmatters:canthisserver
reachthedatabaseandanswerareal
request?
One address in front
Customers
Load balancer
api.quickbite.com
App server
1
App server
2
App server
3
every
request
GET /health every 5 s
3 checks failed: out
What it gives you
One address for the whole fleet
Load spread across servers
Dead servers skipped
automatically
WHAT IT COSTS
Aboutonemillisecondper
request,andoneboxcan
passontensofthousands
ofrequestsasecond.

Every five seconds, the load balancer asks each server for slash health, and a healthy server answers two hundred OK. When app server two fails three checks in a row, maybe it crashed, maybe its disk filled up, the load balancer takes it out of rotation. Its traffic goes to servers one and three. When it passes checks again, it's added back. Nobody gets paged for a single bad server.

A health check that only says the process is running isn't enough. A good one checks what matters: can this server reach the database, and answer a real request in time? Otherwise the load balancer keeps sending orders to a server that can't save them.

Packets or requests

Packets or requests
Layer 4 (transport)
Sees IP addresses and ports
Very fast, very cheap
Can't route by URL
Layer 7 (application)
Reads the HTTP request
Routes /orders and /riders apart
Can end TLS and add headers
THE USUAL ANSWER
Layer7forwebAPIs;
layer4whenyouneedraw
speedorthetrafficisn't
HTTP.

Load balancers come in two kinds, and interviewers love to ask which one you'd use.

A layer four load balancer works on connections: it sees IP addresses and ports, and forwards bytes without reading them. That makes it extremely fast and cheap, but it can't make decisions about what's inside. A layer seven load balancer reads the HTTP request itself. It can send slash orders to the orders service and slash riders to the riders service, end the TLS handshake for the servers, and add headers. It does more work per request, but for a web app like QuickBite, that's usually worth it.

So the usual answer is layer seven for web APIs, and layer four when you need raw speed, or when the traffic isn't HTTP at all, like a database connection or a game server.

No single point of failure

No single point of failure
Customers
Active
LB
Standby
LB
App servers
1, 2, 3
heartbeat
takes the address over
RENTING ONE
Cloudloadbalancerscomeas
apairalready,withhealth
checksbuiltin:youpayby
thehourandbytraffic.

There's one problem left, and it's hiding in plain sight.

We made the app servers redundant, but every request still passes through one load balancer. If it dies, QuickBite is down, no matter how many servers sit behind it. The fix is a pair. The active load balancer carries the traffic, and a standby watches it with a heartbeat every second. Behind them, the same app servers.

If the heartbeat stops, the standby takes over the active one's IP address, called a floating IP, within a few seconds, and traffic flows again. Cloud load balancers do this for you behind the scenes, which is one reason teams rent them instead of running their own.

In practice, most teams rent their load balancer from their cloud provider. It comes as a redundant pair already, with health checks built in, and scales itself up when traffic grows. You pay by the hour and by traffic, and in an interview it's perfectly fine to say: a managed load balancer.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
Alayer7loadbalancer
withleastconnections
andhealthchecks,inan
activeandstandbypair.
Follow-ups to expect
->
How does it know a server is
dead?
->
What if the load balancer
dies?
->
How does a user stick to one
server?
THE COMMON MISTAKE
Drawingoneload
balancerand
forgettingitcanfail
too.

Here's how to bring all of this into an interview in thirty seconds.

Say this: I'll put a layer seven load balancer in front of the stateless app servers, using least connections, with health checks, and an active and standby pair so it isn't a single point of failure. That one sentence answers four follow-up questions before they're asked.

Expect these follow-ups. How does it know a server is dead? Health checks, with a few failures in a row before it acts. What if the load balancer dies? A standby takes over its address. And how does a user stick to one server? Hashing, or sticky sessions, though with stateless servers you rarely need either.

The common mistake is drawing one load balancer box and moving on. The interviewer will point at it and ask what happens when it fails. Draw the pair before they ask.

Quick check

Quick check
?
Some QuickBite requests take 50
ms and some take 5 seconds.
Round robin or least connections?
Least connections: new work goes
to the server with the fewest
requests in flight, so slow requests
don't pile up on one server.
?
App server 2 crashes at
lunch. What do customers
notice?
Almost nothing: health checks
take it out within seconds, and
the other servers take its
traffic.
Remember
Least connections for mixed
requests
Health checks remove dead
servers
Two load balancers, never one
Quick check

Some QuickBite requests take 50 ms and some take 5 seconds. Round robin or least connections?

Show the answer

Least connections: new work goes to the server with the fewest requests in flight, so slow requests don't pile up on one server.

Quick check

App server 2 crashes at lunch. What do customers notice?

Show the answer

Almost nothing: health checks take it out within seconds, and the other servers take its traffic.

Remember three things. Least connections for mixed requests. Health checks remove dead servers before customers notice. And two load balancers, never one.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.