The whole course ▾

System Design · Mock interview 10 of 20

Design a distributed cache

Updated 22 Oct 2026 · By Akshay, software engineer, India · 9 min read

Design a distributed in-memory cache, like Redis Cluster.

Try it first

Before you read the interview, answer the question yourself. Give yourself 10 minutes: say your design out loud, or sketch it on paper. What would you ask? Which numbers matter? Which boxes, and what's the hard part? Then read on, and score yourself with the feedback at the end.

The video for this mock interview is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

Requirements

Requirements
Clarifying questions
->
Operations? -> get, set with TTL, delete
->
Scale? -> 10 TB, a million requests/s
->
Losing data? -> Fine, but rare
->
Stale reads? -> Briefly, yes
Functional requirements
get, set and delete by key
A time to live per key
Evict when memory is full
Add or remove nodes, no downtime
Non-functional requirements
->
Under a millisecond, inside the data centre
->
A million operations a second, and growing
->
Survives a node dying
->
Twice the nodes, twice the capacity
Out of scope
Saving to disk and backups
Transactions and scripts
Copies across regions
interviewer
Hi. Today's one is infrastructure. Ready?
candidate
Ready. I'll draw as I go.
interviewer
Design a distributed, in-memory cache. Something like Redis Cluster, or Memcached.
candidate
Okay. Before it gets abstract, I'd like to ground it: who uses it, for what, and how much they care about losing data. A few questions?
interviewer
Sure.
candidate
Which operations?
interviewer
Get, set with a time to live, and delete.
candidate
What scale?
interviewer
About ten terabytes of data, and a million requests a second.
candidate
What about losing data?
interviewer
It's a cache. The real data lives in a database, so losing some is fine, as long as it's rare.
candidate
And stale reads?
interviewer
Briefly, yes.
candidate
That frees me from a lot of hard problems.
candidate
So, what it does. Get, set and delete values by key. An optional time to live per key, after which it disappears. Evict the least useful keys when memory is full, instead of refusing writes. And operators can add or remove nodes while it's running.
interviewer
Which eviction rule?
candidate
Least recently used, as the default. I'll show how a node tracks it.
candidate
And how well. Answers in under a millisecond, inside the data centre, or there's no point. A million operations a second, and growing. It survives a node dying, without a stampede onto the database. And twice the nodes should mean twice the memory and twice the throughput.
interviewer
Sounds right.
candidate
I'll leave out saving to disk and backups, transactions and scripts inside the cache, and copies across regions. Does that match what you had in mind?
interviewer
Yes. Focus on how keys are spread, and what happens when nodes fail.

Estimation

Estimation
Back of the envelope
10 TB ÷ 64 GB usable per node ~ 160 nodes
× 2 for a replica each = 320 nodes
1M ops/s ÷ 160 ~ 6,000/s per node
One node handles 100,000+ ops/s
WHAT THE NUMBERS DECIDE
Memorysetsthenode
count,nottraffic.Sothe
hardpartsareplacement,
failure,andhotkeys.
candidate
Let me size it. I'm curious which resource runs out first.
interviewer
How many machines do we need?
candidate
I'll size by memory first, then check the traffic.
candidate
By memory: ten terabytes at about sixty-four gigabytes usable per machine is around a hundred and sixty nodes. With one replica each, three hundred and twenty. Traffic: a million operations a second across a hundred and sixty primaries is about six thousand per node, and one cache node handles well over a hundred thousand simple operations a second.
interviewer
Why one replica, and not two?
candidate
It's a cache. One copy is enough to avoid a stampede when a node dies. The database is the real backup.
candidate
So memory sets the node count, not traffic. That means the hard problems aren't raw speed. They're placement: which node holds which key. Failure: what happens when one of three hundred machines dies. And hot keys: averages hide the one key that gets a hundred thousand reads a second.
interviewer
Good. That's where caches really break.

API and data

API and data
Hash table
key -> value,
expiry
LRU list
most recent ->
least
Expiry
sweeper
touch on
read
remove
expired
Commands
1
GET key -> value, or a miss
2
SET key value [TTL]
3
DELETE key
4
A small binary protocol over TCP
candidate
Next, the API, which is tiny, and how one node stores its keys.
interviewer
What does the API look like, and how is a key stored inside one node?
candidate
Three commands, and a couple of data structures.
candidate
GET a key returns its value, or a miss. SET stores a value, with an optional time to live. DELETE removes it, which applications use to invalidate. And they travel over a compact binary protocol, on long-lived TCP connections, not HTTP.
interviewer
Why not HTTP?
candidate
At under a millisecond, every byte and every handshake counts.
candidate
Inside one node: a hash table, mapping each key to its value and its expiry time, so a lookup is one step. An LRU list, least recently used: every read moves a key to the front, so when memory is full, the key at the back is evicted. And expiry is lazy: an expired key is deleted when someone reads it, while an expiry sweeper samples keys in the background to clear the rest.
interviewer
Why not scan every key?
candidate
With billions of keys, a full scan never finishes. Sampling is cheap, and good enough.

High-level design

High-level design
App server
cache client library
Config service
the ring map
Cache node
A
Cache node
B
Cache node
C
ring map
GET
menu:42
1
Why a smart client
->
One hop: no proxy in the way
->
The ring map is tiny and changes
rarely
->
Clients refresh it when nodes
change
TWO RISKS
Keysmovewhennodes
joinorleave.Andnodes
die,andkeysgethot.
interviewer
Okay. Show me how a read finds its node.
interviewer
Walk me through a read, end to end.
candidate
The clients, the nodes, and the one small piece that keeps them agreeing.
candidate
Each app server uses a smart client library. It keeps a copy of the ring map, which says which node owns which part of the hash ring, fetched from a small config service, kept by something like etcd or ZooKeeper. To read menu forty-two, the client hashes the key, finds its node on the ring, here cache node B, and talks to it directly. One network hop.
interviewer
What if the client's map is out of date?
candidate
The node it asks replies, not mine anymore. The client refreshes the map and tries again.
interviewer
Why put the routing in the client, and not a proxy?
candidate
It saves a hop. A proxy would add its own delay to every request, and be one more thing to scale. The ring map is tiny, a few kilobytes, and changes only when nodes join or leave. Clients refresh it when the config service announces a change, or when a node replies: not mine anymore.
interviewer
Where does this go wrong?
candidate
Two places. When a node joins or leaves, keys move, and the database has to survive the misses. And when a node dies, or one key suddenly gets a hundred thousand reads a second.
interviewer
Placement first.

Deep dive: placing keys

Deep dive: placing keys
hash mod N
Simple and evenly spread
N changes: almost every key moves
A cold cache floods the database
Consistent hashing, virtual nodes
Keys and nodes on one ring
A new node takes ~1/N of keys
Virtual nodes keep load even
THE CHOICE
Consistenthashingwith
virtualnodes;theconfig
serviceownsthering,and
clientscacheit.
High-level design
App server
cache client library
Config service
the ring map
Cache node
A
Cache node
B
Cache node
C
ring map
GET
menu:42
1
New node
D, warming up
a slice of keys
~1/N of keys
Why a smart client
->
One hop: no proxy in the way
->
The ring map is tiny and changes
rarely
->
Clients refresh it when nodes
change
TWO RISKS
Keysmovewhennodes
joinorleave.Andnodes
die,andkeysgethot.
interviewer
Let's start with placement.
interviewer
How do you spread keys across nodes, so that adding a node moves as little as possible?
candidate
Two options. Let me compare them.
candidate
Hash mod N is simple and spreads keys evenly. But when N changes, from a hundred and sixty to a hundred and sixty-one, almost every key moves, and a suddenly cold cache floods the database. Consistent hashing puts keys and nodes on one ring, so a new node takes only about one N-th of the keys, from its neighbours. And virtual nodes, many points per machine, keep the load even. Redis Cluster uses a close cousin: sixteen thousand three hundred and eighty-four fixed slots that move between nodes.
candidate
I'd use consistent hashing with virtual nodes. And the config service, a small, strongly consistent store, owns the ring. Nodes register with it and keep a lease alive with heartbeats; clients cache the ring and refresh it when it changes.
interviewer
Is the config service in the read path?
candidate
Never. Clients only ask it when something changes.
candidate
Now a new node, D, joins. The config service gives it its points on the ring, and it takes over one slice of keys from its neighbours. Those keys miss once, then fill from the database. To be gentle, D can start with a few virtual nodes and gain more over minutes, so the extra database load arrives as a trickle, not a wave.
interviewer
How long until D is fully warm?
candidate
Usually minutes. The popular keys fill first, because they're asked for first.
interviewer
A node is just slow, after a long pause, and the config service decides it's dead. What then?
candidate
Its lease expires, it's removed from the ring, and clients send its keys elsewhere. When it recovers, it must rejoin as an empty node, and never serve its old keys, because they may have changed in the meantime. Stale data is the one thing a cache mustn't serve for long.
interviewer
Good.

Deep dive: failures and hot keys

Deep dive: failures and hot keys
Surviving a node
1
Each primary has a replica
2
Writes copied asynchronously
3
Primary dies: promote the replica
4
A few recent writes may be lost: fine
FINDING HOT KEYS
Samplerequeststospot
hotkeysautomatically;
don'twaitforanodeto
melt.
High-level design
App server
cache client library
Config service
the ring map
Cache node
A
Cache node
B
Cache node
C
ring map
GET
menu:42
1
New node
D, warming up
a slice of keys
~1/N of keys
menu:42: 100,000 reads/s
local copy, 2 seconds
Why a smart client
->
One hop: no proxy in the way
->
The ring map is tiny and changes
rarely
->
Clients refresh it when nodes
change
TWO RISKS
Keysmovewhennodes
joinorleave.Andnodes
die,andkeysgethot.
interviewer
Now, what breaks.
interviewer
A node dies. What happens to its keys, and to the database behind them?
candidate
Without a plan, a hundred and sixtieth of all reads suddenly miss at once. So I'd plan for it.
candidate
Every primary gets a replica on a different machine, ideally in a different rack. Writes are copied asynchronously, so they stay fast. When a primary dies, the config service notices the missing heartbeats and promotes its replica, usually within seconds, and clients refresh their map. A few of the newest writes may be lost, and that's fine for a cache, because the database still has the truth.
interviewer
Why not copy writes synchronously?
candidate
That would double the write latency, for data we can afford to lose.
candidate
Now the hot key. Say a celebrity chef's menu gets a hundred thousand reads a second, all on node B, which melts while the others idle. Two fixes. Each app server keeps a tiny local copy of hot keys, for a second or two, so most reads never leave the server. And for very hot keys, I'd store several copies under suffixed keys, spread across nodes, and read a random one.
interviewer
Doesn't the local copy serve stale data?
candidate
For a second or two, yes, which our requirements allow.
interviewer
How do you know which keys are hot?
candidate
I wouldn't wait for a node to melt. Each node samples its requests, say one in a hundred, and reports its top keys. When a key crosses a threshold, clients start caching it locally. It has to be automatic, because hot keys come and go with the news.
interviewer
A hot key expires, and ten thousand requests miss at the same moment. Then?
candidate
That's a stampede. I'd let only one request per key refill it, while the others wait briefly or get the slightly old value. Refresh popular keys a little before they expire. And add jitter to expiry times, so related keys don't all expire together.
interviewer
Nice.

Wrap-up

Wrap-up
Next steps
1
Snapshots to disk, for faster warm-up
2
Smarter memory: fewer wasted bytes
3
Watch hit rate, evictions, p99
4
Authentication and encryption
INTERVIEWER'S FEEDBACK
Strong:sizedbymemory,consistent
hashingwithaconfigservice,failover,
hotkeysandstampedes.Nexttime:say
howclientsretry.Verdict:stronghire.
interviewer
We're nearly at time.
interviewer
What would you improve with more time?
candidate
A few things.
candidate
First, optional snapshots to disk, so a restarted node warms up from a file instead of from the database. Second, smarter memory use: allocating in size classes wastes less, which across three hundred machines is real money. Third, I'd watch the hit rate, evictions per second, and the ninety-ninth percentile latency, per node. And fourth, authentication and encryption between clients and nodes.
interviewer
Which would you do first?
candidate
The watching. One slow node quietly slows every page that touches it.
interviewer
Okay, let's stop there. That was strong. You sized by memory and then went straight to the real problems: placement, failure and hot keys. Consistent hashing with a config service, fast failover that accepts small losses, and protection from hot keys and stampedes. One thing I'd add: how clients retry while a key is moving. I'd say a strong hire.
candidate
Thank you. Good point about the retries.
Interviewer's feedback

Strong: sized by memory, consistent hashing with a config service, failover, hot keys and stampedes. Next time: say how clients retry. Verdict: strong hire.

Score yourself

Interviewers score these seven things. Give your own attempt a tick for each one you did.

  • Asks before drawing. Agrees what the system does, and how well: scale, speed, consistency.
  • Turns users into requests a second and bytes a day, and says what the numbers decide.
  • A small, clear contract, and a data model that fits how the data is read.
  • A working path end to end before any detail. Every box has a reason.
  • Picks the hard parts, compares two options, and chooses one with a trade-off said out loud.
  • What breaks, what happens then, and how users notice. Retries, duplicates, limits.
  • Thinks out loud, checks in with the interviewer, and takes a hint without losing the thread.

0 of 7

Your progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.