The whole course ▾
System Design · Lesson 11 of 39 · Week 3
Replication: copies of your data
QuickBite's one database answers every read and write, and lunchtime is too much for it. Today: replication, copies that share the load.
Got a doubt? Ask in the comments under the video.Subscribe
QuickBite's database can't keep up with reads.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
One database, every read
Yesterday, indexes made each query fast. But at lunch, QuickBite's database gets so many queries that even the fast ones wait in line.
Here's QuickBite at lunch: many app servers, all talking to one orders DB. For every order placed, there are about twenty reads: menus the cache missed, order status checks, history pages, riders' apps asking for new jobs. Writes are easy for the database. It's the reads, thousands a second, that push it to its limit.
At the lunch peak, its processor sits at ninety-five percent, and queries that took a millisecond take a hundred. Worse, it's one machine. If it dies, QuickBite stops taking orders, completely, until someone fixes it. Two problems: too many reads, and a single point of failure. One idea fixes both.
Copies of the data
The idea is to keep copies of the same database on several machines. That's called replication.
One machine stays in charge: the leader. Every write still goes to it. Next to it we add two followers. They hold full copies of the data, and they stay up to date by replaying the leader's change log: every insert, update and delete, in the same order. Followers never take writes from the app. They only copy what the leader did.
Now the app sends its writes to the leader, and its reads to the followers. Twenty reads for every write, spread across two followers, or five, or ten: you add followers until the reads fit. These copies are called read replicas, and adding them is the classic first step in scaling a read-heavy database.
In one line: writes go to the leader, and reads spread across followers that replay its change log. And as a bonus, we now have spare copies of everything, which will matter the night the leader dies.
Wait for the copies, or not?
Here's the decision that matters most. When the leader tells the app a write is done, have the followers got it yet?
Synchronous: the leader waits until the followers confirm they have the change, and only then tells the app it's done. Nothing is lost if the leader dies, but every write waits for the slowest follower, and a dead follower can block writes completely. Asynchronous: the leader confirms as soon as it has the change itself, and the followers catch up a moment later. Writes stay fast. Most systems choose asynchronous, often with one synchronous follower kept as a safety copy.
But asynchronous has a price, and it has a name: replica lag. Followers are always a little behind the leader. Usually by milliseconds. Under heavy load, sometimes by seconds. And a few seconds is plenty of time for a very confusing bug. Watch it as a number on your dashboards, because it grows exactly when you're busiest.
Where did my order go?
Here's that bug. Almost every app that adds read replicas meets it, usually on its busiest day.
A customer places an order. The write goes to the leader, which confirms it, and the app says: order placed. A moment later, the app opens the orders page, and that read goes to a follower, which is two hundred milliseconds behind and hasn't received the order yet. So the customer sees: no orders. They panic, and order again. Two biryanis.
The fix has a name: read your own writes. Right after a customer writes something, send that customer's reads to the leader, which is never behind, for a few seconds. Everyone else keeps reading from followers. A finer version remembers where the customer's last write sits in the change log, and reads from any follower that has caught up past it. Either way, nobody watches their own change disappear.
So decide feature by feature. A menu that's a second old? Nobody notices. A restaurant's rating? Fine. But your own order, or a payment you just made, must come from the leader. Saying that distinction out loud is exactly what interviewers want to hear. The rule of thumb: if a user would notice their own change missing, read it from the leader.
When the leader dies
Replication fixed the reads. It also gives us a way to survive the leader dying.
At two in the morning, the leader's disk fails, and it dies. Writes stop. The fix is called failover: a follower is promoted to be the new leader, the app's writes are pointed at it, and the other followers start copying from it instead. Managed databases do this automatically, often within a minute or two. And when the old leader comes back, it must rejoin as a follower, never as a second leader. And the app has to find the new leader too, usually through one database address that the failover moves to the new machine.
But failover has sharp edges. With asynchronous copying, the newest writes may never have reached the follower, and they're lost. If the old leader was only cut off, not dead, you can end up with two leaders taking writes at once: split brain, which systems prevent by making sure the old one steps down. And a false alarm, a leader that was only slow, can trigger a failover nobody needed.
In the interview
Replication shows up in nearly every design, usually in a single sentence. Make it a good one.
Say it like this: a leader and followers, replicating asynchronously, with reads spread across replicas, and a user's own recent data read from the leader. Then say what happens on failover. That's one breath, and it answers the next three questions before they're asked.
Then expect these. What's lost when the leader dies? With asynchronous copying, possibly the last few writes, so say so. How do you handle replica lag? Read your own writes. And why not several leaders, each taking writes? Each region can, which helps global apps, but two leaders can change the same row at once, so conflicts need resolving. That's lesson thirty-eight.
The common mistake: adding read replicas and never mentioning lag. Interviewers wait for it, and the candidate who brings it up first stands out.
Quick check
Writes are fine, but reads are overloading the database. What do you add?
Show the answer
Read replicas: followers that copy the leader and take the reads.
A user changes their address, refreshes, and sees the old one. Why, and the fix?
Show the answer
Replica lag. Read the user's own data from the leader for a few seconds after a write.
Remember: writes go to the leader, and reads to the followers. Asynchronous copying is fast, but followers lag, so read your own writes from the leader. And failover promotes a follower when the leader dies. Next, the problem replication can't solve: data too big for one machine.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.