The whole course ▾

System Design · Lesson 17 of 39 · Week 4

Idempotency and exactly-once in practice

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

A customer double-taps Pay on a slow network, and QuickBite charges them twice. Today: idempotency, so every retry is safe.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

A customer double-taps Pay.

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

Charged twice

Charged twice
Customer app
Payment
service
Payment
provider
Payments
DB
POST /pay
1
charge ₹450
2
record it
3
lost reply
4
₹900 charged
retry, or a double-tap
THE RULE
Youcan'tstop
duplicatesarriving.You
canmakethemharmless.

This week we learned that queues and Kafka deliver at least once. Today, the bill for that arrives: duplicates. And they don't only come from queues.

A customer taps Pay on a weak connection. The customer app sends the request. The payment service asks the payment provider to charge four hundred and fifty rupees, and records it in the payments DB. The charge succeeds. But the reply back to the phone is lost in a lift. The app sees a timeout, and does what apps do: it tries again.

The retry reaches the payment service as a brand-new request, so it charges again. The customer pays twice. And it isn't only retries: people double-tap, load balancers retry, and queues redeliver. A network can always lose the reply after the work is done, so the sender can never be sure, and must try again. Duplicates aren't a rare bug. They're guaranteed.

So here's the rule this whole lesson is built on: you can't stop duplicates from arriving. You can make them harmless. The word for harmless, when it comes to repeated requests, is idempotent.

Safe to repeat

Safe to repeat
Safe to repeat
Set the status to delivered
Delete order 42
PUT the new address
Not safe to repeat
Charge ₹450
Add 50 reward points
POST a new order

An operation is idempotent if doing it twice has exactly the same effect as doing it once. Some operations are like that already.

Setting an order's status to delivered: do it twice, and it's still delivered. Delete order forty-two: twice, and it's still deleted. Put a new address: the same address, either way. These are naturally idempotent, so retrying them is free. That's why lesson three said PUT and DELETE should be idempotent in a good API.

But some operations add something every time. Charging four hundred and fifty rupees. Adding fifty reward points. Creating an order with a POST. Do them twice, and you get two of them. These are the dangerous ones, and payments are the most dangerous of all. We can't make charging a card naturally idempotent, so we make the request idempotent instead.

The idempotency key

The idempotency key
Doing it right
1
Key and result saved together
2
A unique constraint on the key
3
Same key in flight? Wait, or 409
4
Keep keys for a day or more
Charged twice
Customer app
Payment
service
Payment
provider
Payments
DB
POST /pay
1
charge ₹450
2
record it
3
lost reply
4
₹900 charged
retry, or a double-tap
Keys table
key -> saved result
seen this
key?
1
key: 7f3a, on every retry
THE RULE
Youcan'tstop
duplicatesarriving.You
canmakethemharmless.

The trick: give every action a name, chosen once by the client, and repeated on every retry.

When the customer taps Pay, the app creates a unique key, say seven-f-three-a, and sends it with the request, and with every retry of that same request. The payment service first checks its keys table: have I seen seven-f-three-a? If not, it charges the card and saves the key together with the result. If the same key arrives again, it doesn't charge. It returns the saved result. Same request, same answer: charged once.

Four details make it work. Save the key and the result together, in one transaction, so a crash can't leave one without the other. Put a unique constraint on the key, so two copies racing in at once can't both get through. If a request with the same key is still in flight, the second one waits, or gets a four-oh-nine, conflict. And keep keys for at least a day, longer than any client will keep retrying.

Exactly-once, in practice

Exactly-once, in practice
THE HONEST ANSWER
Exactly-once=
at-least-oncedelivery+
idempotentprocessing.
An idempotent consumer
1
Every event carries a unique ID
2
Record processed IDs with the work
3
Seen it? Skip it
4
Or write "set", never "add"

Queues and Kafka deliver at least once. So how do systems that promise exactly-once actually do it?

Here's the honest answer. Delivery is at least once, always. But processing can be exactly once, if the consumer is idempotent. The message may arrive twice; its effect happens once. Whenever someone says exactly-once, that's what they should mean, and in an interview, saying it this precisely stands out. Kafka even has a feature called exactly-once, but it covers reading and writing inside Kafka. The moment you charge a card or send an email, you're back to idempotency.

So the rewards worker from Tuesday gets order four-two-one twice. Every event carries a unique ID. The worker records each processed ID in the same transaction as the work itself, adding the points. When an ID comes back, it's already there, so the worker skips it. Or even simpler, design the write to be safe: set the points for order four-two-one, rather than add fifty points.

Saving and sending, together

Saving and sending, together
Order service
Orders
table
Outbox table
events to send
order row
1
event row
1
One transaction
Relay
polls the outbox
Orders
topic
new events?
2
publish
3

One more trap, and it's subtle. The order service saves the order, then publishes order placed to Kafka. What if it crashes between the two? The order exists, but no event: no text, no points, no fraud check. Two systems, and no transaction covering both.

The fix is the outbox pattern. In one database transaction, the order service writes the order into the orders table, and the event into an outbox table, in the same database. Either both are saved, or neither is. There's no moment where the order exists without its event, because the database's own transaction guarantees it.

Then a separate relay reads new rows from the outbox, publishes each one to the orders topic, and marks it sent. If the relay crashes after publishing but before marking, it publishes that event again: a duplicate. And that's fine, because our consumers are idempotent. Every piece of this lesson fits together. Another way to build the relay is change data capture: reading the database's own change log, the same one replicas use.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
Clientssendanidempotency
key;consumersdedupeby
eventID;eventsleave
throughanoutbox.
Follow-ups to expect
->
Two retries arrive at once?
->
How long do you keep keys?
->
Is true exactly-once possible?
THE COMMON MISTAKE
Retryingapayment
withoutakey:the
retryisabrand-new
charge.

Idempotency comes up in every design that touches money, and in every design with a queue.

Say it in one breath: clients send an idempotency key with every payment, consumers deduplicate by event ID, and events leave the service through an outbox. Three mechanisms, and together they make at-least-once safe from end to end.

Then expect these. Two retries arrive at once? The unique constraint lets one through, and the other waits for its result. How long do you keep keys? Longer than any client retries, a day is common. And is true exactly-once possible? Not for delivery across a network. For processing, yes: at least once, plus idempotency.

The common mistake: adding retries to a payment call, without an idempotency key. Every retry becomes a brand-new charge, and the retries you added for safety are what charge people twice.

Quick check

Quick check
?
Which is idempotent: "set
points to 50" or "add 50
points"?
Set points to 50: doing it twice
gives the same result.
?
A service saves an order, then
crashes before publishing its
event. What's the fix?
The outbox: save the event in
the same transaction, and let a
relay publish it.
Remember
Duplicates are guaranteed:
make them harmless
Idempotency keys for client
retries
Outbox, plus idempotent
consumers
Quick check

Which is idempotent: "set points to 50" or "add 50 points"?

Show the answer

Set points to 50: doing it twice gives the same result.

Quick check

A service saves an order, then crashes before publishing its event. What's the fix?

Show the answer

The outbox: save the event in the same transaction, and let a relay publish it.

Remember: duplicates are guaranteed, so make them harmless. Idempotency keys make client retries safe. And an outbox, plus idempotent consumers, makes events safe from end to end. In one sentence: at least once, plus idempotent, equals exactly once. Tomorrow, QuickBite's map comes alive.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.