The whole course ▾
System Design · Lesson 6 of 39 · Week 2
Load balancers
QuickBite now runs three app servers. But who decides which one gets your order? Today: load balancers, from first request to failover.
Got a doubt? Ask in the comments under the video.Subscribe
QuickBite puts a load balancer in front of three app servers.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
One address in front
Yesterday we scaled QuickBite out to three stateless app servers and drew a box in front of them. Today, we open that box.
Customers never talk to an app server directly. DNS points api dot quickbite dot com at the load balancer, so every request arrives there first. Its job is to pass each request to one healthy server: app servers one, two and three. The servers behind it can change at any time, and the customers never notice.
That one box gives you three things. A single address, so the app never needs to know how many servers exist. Load spread evenly, so no server melts while another sits idle. And dead servers skipped automatically, which we'll come back to in a minute.
And what does it cost? One extra hop adds about a millisecond to each request, which nobody notices. A single load balancer can pass on tens of thousands of requests a second, so for QuickBite's five hundred at lunch, it's barely awake.
Three ways to choose
So how does the load balancer pick a server for each request? There are three answers you should know by name.
First, round robin: server one, then two, then three, then back to one. It's simple and works well when every request costs about the same. Second, least connections: send the request to the server with the fewest requests in flight right now, which handles a mix of quick and slow requests far better. Third, hashing: compute a hash of something like the user's ID, so the same user always lands on the same server.
Here's least connections at one instant during lunch. Server one has twelve requests in flight, server two has three, and server three has nine. The next request goes to server two, the least busy, even if it was server two's turn a moment ago. The load balancer counts open connections, so this costs it almost nothing.
For QuickBite, browsing a menu takes milliseconds, but checking out with a payment can take seconds. Requests aren't alike, so least connections is the better default. Hashing matters when a server holds something for a user, like a cache, and we'll see its smarter version, consistent hashing, in lesson thirteen.
Health checks
Every five seconds, the load balancer asks each server for slash health, and a healthy server answers two hundred OK. When app server two fails three checks in a row, maybe it crashed, maybe its disk filled up, the load balancer takes it out of rotation. Its traffic goes to servers one and three. When it passes checks again, it's added back. Nobody gets paged for a single bad server.
A health check that only says the process is running isn't enough. A good one checks what matters: can this server reach the database, and answer a real request in time? Otherwise the load balancer keeps sending orders to a server that can't save them.
Packets or requests
Load balancers come in two kinds, and interviewers love to ask which one you'd use.
A layer four load balancer works on connections: it sees IP addresses and ports, and forwards bytes without reading them. That makes it extremely fast and cheap, but it can't make decisions about what's inside. A layer seven load balancer reads the HTTP request itself. It can send slash orders to the orders service and slash riders to the riders service, end the TLS handshake for the servers, and add headers. It does more work per request, but for a web app like QuickBite, that's usually worth it.
So the usual answer is layer seven for web APIs, and layer four when you need raw speed, or when the traffic isn't HTTP at all, like a database connection or a game server.
No single point of failure
There's one problem left, and it's hiding in plain sight.
We made the app servers redundant, but every request still passes through one load balancer. If it dies, QuickBite is down, no matter how many servers sit behind it. The fix is a pair. The active load balancer carries the traffic, and a standby watches it with a heartbeat every second. Behind them, the same app servers.
If the heartbeat stops, the standby takes over the active one's IP address, called a floating IP, within a few seconds, and traffic flows again. Cloud load balancers do this for you behind the scenes, which is one reason teams rent them instead of running their own.
In practice, most teams rent their load balancer from their cloud provider. It comes as a redundant pair already, with health checks built in, and scales itself up when traffic grows. You pay by the hour and by traffic, and in an interview it's perfectly fine to say: a managed load balancer.
In the interview
Here's how to bring all of this into an interview in thirty seconds.
Say this: I'll put a layer seven load balancer in front of the stateless app servers, using least connections, with health checks, and an active and standby pair so it isn't a single point of failure. That one sentence answers four follow-up questions before they're asked.
Expect these follow-ups. How does it know a server is dead? Health checks, with a few failures in a row before it acts. What if the load balancer dies? A standby takes over its address. And how does a user stick to one server? Hashing, or sticky sessions, though with stateless servers you rarely need either.
The common mistake is drawing one load balancer box and moving on. The interviewer will point at it and ask what happens when it fails. Draw the pair before they ask.
Quick check
Some QuickBite requests take 50 ms and some take 5 seconds. Round robin or least connections?
Show the answer
Least connections: new work goes to the server with the fewest requests in flight, so slow requests don't pile up on one server.
App server 2 crashes at lunch. What do customers notice?
Show the answer
Almost nothing: health checks take it out within seconds, and the other servers take its traffic.
Remember three things. Least connections for mixed requests. Health checks remove dead servers before customers notice. And two load balancers, never one.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.