The whole course ▾
System Design · Lesson 22 of 39 · Week 5
Distributed transactions: two-phase commit and sagas
A QuickBite order touches three services: orders, payments and the restaurant. What if the kitchen says no after you've paid? Today: sagas.
New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.
Got a doubt? Ask in the comments under the video.Subscribe
Order placed, payment taken, restaurant accepts: or not.
QuickBite is the food delivery app we grow through the course, from one server to millions of orders.
Three services, one order
In one database, placing an order was one transaction: all of it happened, or none of it did. On Monday we split the database, one per service. Today we find out what that costs.
Placing an order now touches three services. The orders service creates the order in the orders DB. The payments service charges four hundred and fifty rupees, recorded in the payments DB. And the restaurant service asks the kitchen to accept the order, recorded in the restaurant DB. Three services, three databases, and three separate transactions, each of which commits on its own.
Now the restaurant says no: they've run out of biryani. But the payment is already taken, and the order already exists. In one database, we'd simply roll back the whole transaction. Across three databases, there's no single transaction to roll back. We need another way to make three services agree: either everything happens, or everything is undone. And the customer is watching the screen the whole time, so whatever we do must be quick, and honest.
Two-phase commit
The textbook answer is two-phase commit. A coordinator asks everyone to promise, and only then tells everyone to commit.
Phase one: the coordinator asks every participant, can you commit? Each one does its work, locks the rows involved, saves everything to disk, and answers: yes, I promise. Phase two: if every participant said yes, the coordinator tells them all to commit. If anyone said no, it tells them all to roll back. All or nothing, across machines. It's correct, and it's been around for decades.
The catch is the waiting. Between yes and commit, every participant holds its locks, blocking other orders that touch the same rows. If the coordinator dies after collecting the yeses, the participants are stuck: they've promised, so they can't commit or roll back on their own until it returns. And every participant must speak the protocol, which the outside payment provider certainly doesn't. Inside one database system, it's fine. Between services, it blocks too much.
Sagas: undo, don't lock
What most real systems use instead is called a saga: a series of small local steps, each one with an undo.
A saga breaks the order into steps, and each step is a normal local transaction inside one service: create order, charge card, then the restaurant accepts. Each step commits straight away, so no locks are held across services while anyone waits. And every step that changes something has a compensating step, an undo, written in advance: for creating the order, cancel it; for charging the card, refund it.
Now the restaurant refuses. The saga runs the compensations backwards: refund the four hundred and fifty rupees, then mark the order cancelled, and tell the customer, sorry, this restaurant can't take your order right now. The money comes back. Nothing was locked while we waited, and nothing is left half-done at the end.
Notice the difference. Two-phase commit prevents anyone ever seeing a half-done order. A saga allows it, briefly, and then repairs it. For a few seconds, the customer may see: payment taken, waiting for the restaurant. That's honest, and it's how almost every real app with several services works.
Who runs the saga?
Someone has to decide which step comes next, and when to start undoing. There are two styles.
Choreography: nobody's in charge. Orders publishes order created; payments hears it, charges, and publishes payment taken; the restaurant hears that, and so on, using Kafka from lesson sixteen. There's no central piece to fail, but the flow is scattered across services, and hard to follow. Orchestration: one orchestrator calls each step in turn, waits for the answer, and runs the undos when something fails. The whole story lives in one place.
For a flow with three or more steps, most teams choose orchestration, often using a workflow engine like Temporal or AWS Step Functions, which remembers exactly where each saga is, even if the orchestrator crashes halfway. Choreography still fits simple reactions, like the rewards service quietly adding points when an order completes.
Making sagas safe
Sagas are simple to draw and easy to get wrong. Four rules keep them safe.
Every step, and every undo, must be idempotent, from lesson seventeen, because the orchestrator will retry them. An undo can't just fail and give up: it's retried until it succeeds, or a person is alerted. Put the steps that are hardest to undo last: a card charge is easy to refund, but once the kitchen starts cooking, the biryani can't be uncooked. And show the customer honest states, like waiting for the restaurant.
That third rule is the one interviewers love, because it shows judgement. QuickBite charges the card before the kitchen starts, because a refund is cheap and automatic, and an uncooked order is impossible. Ask yourself, for each step: what does undoing it cost? Then order the steps so the expensive undos almost never happen.
In the interview
This comes up whenever one action spans several services: orders, bookings, transfers, sign-ups.
Say it like this: across services, I'd use a saga, a series of local transactions, each with a compensating step, run by an orchestrator, with every step idempotent. And I'd avoid two-phase commit between services, because it holds locks and blocks if the coordinator fails.
Then expect these. Why not two-phase commit? Locks held across services, and blocking when the coordinator dies. What if a compensation fails? Retry it, because it's idempotent, and alert a person if it keeps failing. And what does the user see meanwhile? An honest in-between state, like payment taken, waiting for the restaurant, updated live. And a sharp one: can other requests see a saga's half-done data? Yes, briefly, so design for it, for example by reserving a table instead of taking it until the payment clears.
The common mistake: drawing three services with three databases, then talking as if one transaction covers them all. It doesn't. Say saga, and say what the undo is for each step.
Quick check
The card is charged, then the restaurant rejects the order. In a saga, what happens?
Show the answer
Compensate: refund the card, cancel the order, and tell the customer.
Why is two-phase commit risky between services?
Show the answer
Locks are held while waiting, and everyone is stuck if the coordinator dies.
Remember: no transaction spans services. Sagas chain local steps, each with an undo, and repair a half-done state instead of preventing it. And orchestrate flows with several steps, so the whole story lives in one place. Tomorrow, a deceptively simple question: how do we number our orders?
Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.