The whole course ▾

System Design · Mock interview 19 of 20

Design a metrics and monitoring system

Updated 22 Oct 2026 · By Akshay, software engineer, India · 9 min read

Design a metrics and monitoring system.

Try it first

Before you read the interview, answer the question yourself. Give yourself 10 minutes: say your design out loud, or sketch it on paper. What would you ask? Which numbers matter? Which boxes, and what's the hard part? Then read on, and score yourself with the feedback at the end.

The video for this mock interview is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

Requirements

Requirements
Clarifying questions
->
What data? -> Metrics only, not logs
->
Scale? -> 10,000 servers, 1,000 metrics each
->
How often? -> Every 10 seconds
->
Kept how long? -> 2 weeks raw, 2 years summarised
Functional requirements
Collect metrics, with labels
Store them as time series
Query: dashboards and graphs
Alert rules that page on-call
Non-functional requirements
->
Take in a million points a second
->
Dashboards load in about a second
->
Alerts within a minute
->
Works when everything else is broken
Out of scope
Logs and traces
Fixing problems automatically
Machine-learned anomaly detection
interviewer
Hi. Ready?
candidate
Ready. I'll draw as I go.
interviewer
Design a metrics and monitoring system for a company with ten thousand servers. Collect numbers from everything, show dashboards, and alert engineers when something's wrong.
candidate
Okay. I'd like to clarify what kind of data this is and how long it lives, because time-series data behaves very differently. A few questions?
interviewer
Sure.
candidate
What data?
interviewer
Metrics only: numbers over time, like requests a second or memory used. Logs and traces are separate systems.
candidate
What scale?
interviewer
Ten thousand servers, each reporting about a thousand metrics.
candidate
How often?
interviewer
Every ten seconds.
candidate
And kept how long?
interviewer
Full detail for two weeks, and summaries for two years.
candidate
That last one will shape the storage.
candidate
So, what it does. Collect metrics from every server, each with labels, like service equals checkout, region equals Mumbai. Store them as time series: a value at each moment. Answer queries for dashboards, like the error rate per service for the last six hours. And evaluate alert rules continuously, paging the on-call engineer when one fires.
candidate
And how well. Take in about a million data points a second. Dashboards load in about a second. Alerts fire within a minute of a problem. And it must keep working when everything else is broken.
interviewer
Why does that one matter so much?
candidate
The worst outage is exactly when engineers need monitoring most.
candidate
I'll leave out logs and traces, fixing problems automatically, and learned anomaly detection. Does that match what you had in mind?
interviewer
Yes. Storage and alerting are what I want to hear about.

Estimation

Estimation
Back of the envelope
10,000 servers × 1,000 = 10M time series
÷ 10 s = 1M data points a second
× 86,400 s ~ 86B points a day
Compressed to ~1.4 bytes: ~120 GB a day
WHAT THE NUMBERS DECIDE
Writesareappend-only,intime
order,andenormous:usea
time-seriesstore,with
compressionanddownsampling.
candidate
Let me size it. The raw numbers will look big.
interviewer
How much data is this, coming in, and stored?
candidate
Series first, then points, then bytes.
candidate
Ten thousand servers with a thousand metrics each is ten million time series. Each reports every ten seconds: a million data points a second. Over a day, about eighty-six billion points. Stored naively, sixteen bytes each, that's well over a terabyte a day. But time series compress very well, to about one point four bytes a point, so around a hundred and twenty gigabytes a day.
interviewer
Is a terabyte a day really a problem?
candidate
Over two years it's hundreds of terabytes, on fast disks. Compression is what makes it affordable.
candidate
So writes are append-only, in time order, and enormous. No general database is built for a million inserts a second that are never updated. Reads are mostly recent data, summarised over time ranges. So I want a store built for time series, with heavy compression, and older data downsampled into summaries.
interviewer
Agreed.

API and data

API and data
Series index
labels -> series IDs
Chunks
one series, 2 hours,
compressed
Rollups
5-minute summaries
finds
downsampled
Getting data in and out
1
Each server exposes its metrics
2
Collectors scrape them every 10 s
3
Query: a function over a time
range
4
e.g. error rate, per service, 6
hours
candidate
Next, how data gets in and out, and the data model.
interviewer
How does data get in, and how do dashboards ask for it?
candidate
Scraping in, queries out.
candidate
Each server exposes its current metrics on a small HTTP page. Collectors scrape them every ten seconds, which is how Prometheus works. A query is a function over a time range: for example, the rate of errors, per service, over the last six hours, in one-minute steps.
interviewer
Why scrape, and not have servers push?
candidate
Either works. Scraping has a bonus: a server that stops answering is itself a signal.
candidate
Three structures. A series index: an inverted index, mapping each label, like service equals checkout, to the IDs of matching series. Chunks: for each series, two hours of points, packed and compressed together, in time order. And rollups: older data summarised into five-minute blocks, keeping the minimum, maximum, sum and count.
interviewer
What do labels buy you?
candidate
One metric, sliced by service, region or status code, without defining new metrics.

High-level design

High-level design
10,000
servers
Collectors
scrape every
10 s
Ingesters
recent data in
memory
Object
storage
compressed
blocks
scrape
1
batches, by series
2
every 2
hours
3
Query
service
Dashboards
Alert
evaluator
rules every 30 s
Notifier
group, route,
page
error rate,
6 h
4
recent
5
older
5
check rules
firing
TWO QUESTIONS
Howdoyoustore86billion
pointsadaycheaply?And
canalertsbetrustedwhen
everythingelseisdown?
interviewer
Okay. Show me the flow.
interviewer
Walk me through a data point, from a server to a dashboard, and to an alert.
candidate
Data in first, then queries and alerts out.
candidate
Collectors scrape the ten thousand servers every ten seconds, and send batches of points to ingesters, sharded by a hash of each series, so one series always lands on the same ingester. Each ingester keeps the last two hours of its series in memory, appending points and writing a log, so a crash loses nothing. Every two hours, it compresses those chunks into a block, and uploads it to cheap object storage.
interviewer
Why shard by series?
candidate
All the points of one series end up together, so they compress well, and a query for it goes to one place.
candidate
Reading: dashboards ask the query service, which gets recent data from the ingesters' memory and older data from blocks in object storage, then merges and computes the answer. Alerting: an alert evaluator runs every rule, like checkout errors above two percent for five minutes, every thirty seconds, using the same queries. When a rule fires, the notifier groups related alerts, routes them to the right team, and pages someone.
interviewer
What would you want to go deeper on?
candidate
Two things. How to store eighty-six billion points a day cheaply, and keep two years. And whether alerts can be trusted during the outage itself, when the network or the whole data centre is struggling.
interviewer
Storage first.

Deep dive: storing time series

Deep dive: storing time series
A row per point
16+ bytes, plus index overhead
A million inserts a second
Range reads jump all over the disk
Compressed chunks per series
About 1.4 bytes a point
Appends in memory, flushed in blocks
A range read is one sequential read
THE CHOICE
Per-seriescompressedchunks,
recentonesinmemory,older
onesinobjectstorage,plus
rollupsforlongranges.
High-level design
10,000
servers
Collectors
scrape every
10 s
Ingesters
recent data in
memory
Object
storage
compressed
blocks
scrape
1
batches, by series
2
every 2
hours
3
Query
service
Dashboards
Alert
evaluator
rules every 30 s
Notifier
group, route,
page
error rate,
6 h
4
recent
5
older
5
check rules
firing
10M series: cardinality
limit
~120 GB a day
TWO QUESTIONS
Howdoyoustore86billion
pointsadaycheaply?And
canalertsbetrustedwhen
everythingelseisdown?
interviewer
Let's start with storage.
interviewer
Why not store each point as a row in an ordinary database?
candidate
Let me compare that with compressed chunks.
candidate
A row per point costs sixteen bytes or more, plus index overhead, a million random inserts a second, and reading six hours of one series jumps all over the disk. Compressed chunks store each series' points together, in time order. Since timestamps arrive every ten seconds, I store only the tiny differences between differences, and since values barely change, only the bits that changed. That's how Facebook's Gorilla design reached about one point four bytes a point.
candidate
So, per-series compressed chunks: two hours in memory, then blocks in object storage, which is cheap and practically unlimited. And after two weeks, keep only the five-minute rollups, for the two-year history.
interviewer
Why roll up?
candidate
A graph of last year doesn't need ten-second detail, and a query over a year reads rollups instead of billions of points.
candidate
So the blocks in object storage are cheap: about a hundred and twenty gigabytes a day. The real danger is in memory: the number of series, called cardinality. Every unique combination of labels is a separate series. Ten million is fine. Someone adding a user ID as a label turns it into a billion, and the ingesters run out of memory.
interviewer
An engineer adds the user ID as a label on a request metric. What happens, and how do you prevent it?
candidate
Each user becomes a new series, millions appear in minutes, and memory explodes. So the system enforces limits: a maximum number of series per metric and per team, rejecting new series above it with a clear error. Unbounded values, like user IDs, belong in logs and traces, not in labels.
interviewer
Good.
interviewer
A dashboard shows the month's p99 latency by averaging thirty daily p99 values. Is that right?
candidate
No. Percentiles can't be averaged, because the slow requests of one bad day don't spread evenly across the others. I'd store latency as a histogram, counts of requests in latency buckets, which can be added across days and servers. The month's p99 comes from the summed histogram.
interviewer
Nice. Many people miss that.

Deep dive: alerts you can trust

Deep dive: alerts you can trust
Alerts that work
1
Fire only if true for a few minutes
2
Group: one page for 300 servers
3
Route to the owning team
4
Escalate if nobody acknowledges
WHO WATCHES THE WATCHER?
Runmonitoringseparately,
inanotherregion,witha
heartbeatalertthat
pageswhenitstops.
interviewer
Now, alerting.
interviewer
How do you make alerts reliable, so real problems always page someone, and quiet, so people don't learn to ignore them?
candidate
A few rules.
candidate
Fire only if a condition stays true for a few minutes, so a single blip doesn't wake anyone. Group related alerts: three hundred servers with high errors during one outage become one page, not three hundred. Route each alert to the team that owns the service, using its labels. And escalate: if nobody acknowledges in ten minutes, page the next person.
interviewer
What should you alert on?
candidate
Symptoms users feel, not every internal number.
candidate
And the requirement from the start. Monitoring runs on separate machines, ideally in a separate region, never depending on the systems it watches. Plus a heartbeat: an alert that's always firing, sent to an outside service every minute. If the heartbeat ever stops, the outside service pages someone.
interviewer
Because silence might mean monitoring died.
candidate
Exactly.
interviewer
The main data centre's network fails completely. Can monitoring still page anyone?
candidate
Yes, by design. The alert evaluator and notifier run in pairs, in two regions, and the outside heartbeat notices the silence within minutes. The page might simply say: we've lost contact with the Mumbai data centre. That's exactly what engineers need to hear first.
interviewer
During an outage, a hundred engineers open heavy dashboards at once. Does monitoring slow down?
candidate
I'd protect it. Cache dashboard queries for a few seconds, limit how much data one query may scan, and keep the alert evaluator's queries on their own capacity, so a crowd looking at graphs can never delay an alert.
interviewer
Good.

Wrap-up

Wrap-up
Next steps
1
Link metrics to logs and traces
2
SLO dashboards and error budgets
3
Anomaly detection for unknown problems
4
Cost per team, from series counts
INTERVIEWER'S FEEDBACK
Strong:compressedchunks,rollups,
cardinalitylimits,alertingthatsurvives
theoutage.Nexttime:howingesters
arereplicated.Verdict:stronghire.
interviewer
We're nearly at time.
interviewer
What would you improve with more time?
candidate
A few things.
candidate
First, link metrics to logs and traces, so a spike on a graph opens the slow requests behind it. Second, SLO dashboards and error budgets, built from the same metrics. Third, anomaly detection, for problems nobody wrote a rule for. And fourth, cost per team, from how many series each one stores, because monitoring bills grow quietly.
interviewer
Which would you do first?
candidate
Linking to traces. It turns a scary graph into a reason, in one click.
interviewer
Okay, let's stop there. Very good. You built a real time-series design: compressed chunks, in memory then object storage, with rollups for long ranges. You knew the real dangers, cardinality and averaging percentiles. And your alerting is quiet, grouped, and survives the outage it's reporting. One thing: say how the ingesters are replicated, so losing one doesn't lose recent data. I'd say a strong hire.
candidate
Thanks. I'd send each series to two ingesters.
Interviewer's feedback

Strong: compressed chunks, rollups, cardinality limits, alerting that survives the outage. Next time: how ingesters are replicated. Verdict: strong hire.

Score yourself

Interviewers score these seven things. Give your own attempt a tick for each one you did.

  • Asks before drawing. Agrees what the system does, and how well: scale, speed, consistency.
  • Turns users into requests a second and bytes a day, and says what the numbers decide.
  • A small, clear contract, and a data model that fits how the data is read.
  • A working path end to end before any detail. Every box has a reason.
  • Picks the hard parts, compares two options, and chooses one with a trade-off said out loud.
  • What breaks, what happens then, and how users notice. Retries, duplicates, limits.
  • Thinks out loud, checks in with the interviewer, and takes a hint without losing the thread.

0 of 7

Your progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.