The whole course ▾
System Design · Lesson 38 of 39 · Week 8
Multi-region systems
QuickBite launches in Dubai, and every request crosses the sea to Mumbai and back. Today: multi-region systems.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
QuickBite launches in a second country.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
One region, two countries
Everything we've built so far lives in one region: one group of data centres, in Mumbai. Today QuickBite launches in a second country, and one region stops being enough.
Indian customers are close to the Mumbai region. Dubai customers aren't: every request crosses the sea and back, adding tens of milliseconds to every round trip, and an app screen makes many round trips. It works, but the app feels slower in Dubai, and it all depends on one region staying up.
There are three reasons to run in more than one region. Speed: serving users from nearby. Survival: whole regions do fail, from power, networking, or a bad deploy, and a business that matters must keep running. And the law: many countries require some personal data to be stored inside the country. For QuickBite, all three apply at once.
Active-passive or active-active
Two basic ways to use a second region, and they differ in what each region does on a normal day.
Active-passive: one region serves everyone, and a second region holds copies of the data, waiting. If the active region fails, you fail over to the passive one, which takes minutes, and the passive region's machines sit mostly idle. Simple, but it doesn't make Dubai faster. Active-active: every region serves its own users, every day, so everyone is fast. But now two regions can be writing at the same time, and that's where the hard problems start.
QuickBite's answer: active-active, but every customer, restaurant and order has a home region. A Dubai customer's orders are written in the Dubai region; a Mumbai customer's in Mumbai. Each piece of data has exactly one region that writes it, so two regions never fight over the same row. It's sharding from lesson twelve, with the region as the partition key.
Sending users to the right region
First, each phone must reach the right region. That starts with lesson one's DNS.
When the customer's phone looks up QuickBite's address, a Geo DNS service answers based on where the phone is: the nearest healthy region. A phone in Dubai gets the Dubai region's address; one in Pune gets the Mumbai region's. DNS answers are cached for a short time, say a minute, so if a region goes down, phones are sent elsewhere quickly. And behind the scenes, the two regions copy data to each other.
What about a Mumbai customer on holiday in Dubai? Geo DNS sends them to the Dubai region, which serves menus and restaurants locally. Their own data, like their account and payment methods, belongs to the Mumbai region, so writes to it are forwarded there. A little slower for that one traveller, and correct for everyone.
Copying across regions
Regions are far apart, so how they copy data to each other is a real trade-off, measured in milliseconds.
Decide for each kind of data. Orders and payments are written in their home region only, with strong consistency there. Menus and restaurant details are copied to every region where people read them. Copies between regions are async, from lesson eleven, because waiting for a far-away region on every write would add its round trip to every order. And some personal data never leaves its country at all, to meet the law.
Sometimes two regions really must write the same thing, like a global count of a restaurant's ratings. Then you need a conflict rule, from mock interview three and lesson thirty-five: last write wins for things that can be overwritten, versions or CRDTs for things that must merge, like counters. The home-region design keeps those cases rare, which is the whole point.
When a region fails
A second region is only as good as the day you need it.
Mumbai goes down. First, detect it, using health checks from outside the region, because a region can't report its own death. Then promote the Dubai region's copies of Mumbai's data, so Dubai becomes home for Mumbai's customers too. Geo DNS sends everyone to Dubai. And accept the cost of async copying: the last few seconds of writes that hadn't been copied may be lost, and reconciled once Mumbai returns.
And the senior lesson: an untested failover is a hope, not a plan. Big companies deliberately fail whole regions in practice drills, on purpose, during working hours, to find the broken scripts and missing permissions before a real disaster does. Every region must also have enough spare capacity to carry the other's users.
In the interview
Multi-region is often the last question in a senior interview: now make it global.
Say it like this: active-active regions, with a home region for each user, so each record has one writer; geo DNS sends users to the nearest healthy region; data is copied asynchronously across regions, with in-country data kept in-country; and failover is detected from outside and practised regularly. Then say what's lost in a failover, honestly.
Then expect these. Why not synchronous copies across regions? Every write would wait for a round trip across the sea. A user writes in two regions at once? Forward writes to their home region, or use a conflict rule. And how do you fail over? Detect from outside, promote the copies, move traffic with geo DNS, and accept a few seconds of possible loss.
The common mistake: saying we'd just deploy to another region, as if it were adding servers. Without deciding which region writes which data, two regions end up overwriting each other. Ownership first, then copying.
Quick check
Why give every customer a home region?
Show the answer
Their data is written in one place, so regions never conflict over it.
Mumbai fails. Its data was copied to Dubai asynchronously. What might be lost?
Show the answer
Recent writes not yet copied: the last few seconds.
Remember: regions are for speed, survival, and the law. Give each user a home region, so every record has one writer. And use geo DNS, asynchronous copies, and a failover that's been practised. Tomorrow, the final lesson: thinking like a senior engineer.
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.