The whole course ▾

System Design · Lesson 24 of 39 · Week 5

Observability: logs, metrics and traces

Updated 22 Oct 2026 · By Akshay, software engineer, India · 5 min read

A customer's order took nine seconds, and QuickBite now has eight services. Which one was slow? Today: logs, metrics and traces.

The video for this lesson is coming soon.

New System Design videos go up at 6 pm. Subscribe on YouTube so you don't miss it.

Got a doubt? Ask in the comments under the video.Subscribe

QuickBite in this lesson

Why did that order take 9 seconds?

QuickBite is the food delivery app we grow through the course, from one server to millions of orders.

Nine seconds, eight services

Nine seconds, eight services
Mobile app
API gateway
Orders
service
Payments
service
Restaurant
service
Rider
matching
POST
/orders
9 s
which one was slow?

This week QuickBite became many services. Today a customer complained, and we discovered that we can't see inside any of them.

A customer writes in: my order took nine seconds to go through. The request went from the mobile app, through the API gateway, to the orders service, which called the payments service, the restaurant service, and rider matching, and each of those called others. Somewhere in there, nine seconds disappeared. The question is simple: where? Every on-call engineer faces this question, many times a year.

Each service writes its own logs, on its own machines, in its own format. To follow this one nine-second request, an engineer opens eight sets of log files, and tries to match timestamps by hand. It takes an hour, and they're still not sure. In a monolith, one stack trace told you everything. With many services, the ability to see inside has to be built on purpose. That's called observability.

Logs, metrics and traces

Logs, metrics and traces
The three pillars
1
Logs: what happened, event by event
2
Metrics: numbers over time
3
Traces: one request's whole path
WHICH ANSWERS WHAT
Metricssaysomethingis
wrong;tracessaywhere;
logssaywhy.

Observability rests on three kinds of data. Each one answers a different question.

Logs: a record of each event, like order four-two-one created, with its details. Best written as structured JSON, with the same field names in every service, and shipped to one searchable place. Metrics: numbers measured over time, like requests per second, errors, and latency. They're cheap to keep, and perfect for dashboards and alerts. And traces: the path of one single request through every service it touched, with the time spent at each step.

Here's how they work together. Metrics tell you something is wrong: latency jumped at eight o'clock. Traces tell you where: inside the payments service. And logs tell you why: the payment provider returned a timeout, three hundred times in a minute. You need all three, because each one is blind to what the others see. Many teams start with metrics and logs, and add tracing once they run more than a handful of services.

What to measure

What to measure
RED, for every service
Rate: requests per second
Errors: failed requests
Duration: latency, p50 and p99
USE, for every resource
->
Utilisation: how busy it is
->
Saturation: how much is waiting
->
Errors: hardware and system errors

You could measure thousands of things. Two short checklists cover what matters most.

For every service, measure RED. Rate: how many requests a second it's handling. Errors: how many of them fail. And duration: how long they take, as percentiles, the p fifty and p ninety-nine from lesson two, never just the average, because the average hides the slow requests that customers actually complain about. Put these three graphs at the top of every service's dashboard, and most problems show up within seconds.

And for every resource, the machines, disks and connection pools, measure USE. Utilisation: how busy it is. Saturation: how much work is queued, waiting for it. And errors, from the hardware and the system. Saturation is the early warning: a full thread pool, like Tuesday's payments service, shows up there before anything crashes.

Following one request

Following one request
MAKE IT PAY OFF
PutthetraceIDineverylogline;
samplemosttraces,butkeepevery
slowone.
Nine seconds, eight services
Mobile app
API gateway
Orders
service
Payments
service
Restaurant
service
Rider
matching
POST
/orders
9 s
which one was slow?
trace ID a1b2
120 ms
300 ms
8.2 s

Metrics told us payments got slow at eight. But which call, for this customer's order? That's what tracing is for.

When a request enters the gateway, it gets a trace ID, say a-one-b-two. Every service passes that ID along, in a header on every call it makes, and records a span: what it did, when it started, and how long it took. Collect the spans, and you can draw the whole request as a timeline. For this customer: the restaurant, a hundred and twenty milliseconds. Rider matching, three hundred. And payments, eight point two seconds. Found it.

Two habits make tracing pay off. Put the trace ID in every log line, so one search finds every log from that request, across every service. And don't keep every trace: store a sample, say one in a hundred, but keep every trace that was slow, or failed. Open standards like OpenTelemetry pass the IDs along for you, in almost every language. Then the hour-long search becomes a one-minute one.

Promises and alarms

Promises and alarms
SLI, SLO, error budget
1
SLI: what you measure
2
SLO: the target, e.g. 99.9% a month
3
Error budget: the 0.1% you may miss
ALERT ON SYMPTOMS
Wakesomeoneforwhat
usersfeel:errorsand
slowness,notCPUat
80%.

Measuring is half of it. The other half is deciding what good looks like, and when to wake someone up.

An SLI, a service level indicator, is what you measure: say, the share of order requests that succeed in under a second. An SLO, the objective, is the target: ninety-nine point nine percent, every month. And the gap is your error budget: the point one percent you're allowed to miss, about forty-three minutes of bad time a month. Budget left? Ship new features. Budget spent? Slow down, and fix reliability first.

And alert wisely. Wake someone at three in the morning only for symptoms that users feel: errors rising, latency past the SLO, the error budget burning fast. Not for causes, like a processor at eighty percent, which may be perfectly fine. Too many alerts, and people learn to ignore them, including the one that really matters. A good rule: every alert must need a person to act, right now. If it doesn't, it's a ticket for tomorrow, not a page tonight.

In the interview

In the interview
SAY THIS IN THE INTERVIEW
REDmetricsperservice,
tracescarryinganIDthrough
everycall,structuredlogs,
andalertsonSLOs.
Follow-ups to expect
->
How do you find one slow
request?
->
What would you alert on?
->
What's on the main
dashboard?
THE COMMON MISTAKE
Saying"add
monitoring"withno
numbers:namethe
metricsandtheSLO.

Most candidates forget observability entirely, which makes it an easy way to stand out in the last five minutes.

Say it like this: RED metrics for every service, distributed traces with one ID passed through every call, structured logs that include that ID, and alerts tied to SLOs, not to raw resource numbers. Then name the two or three numbers you'd watch for this particular system.

Then expect these. How do you find one slow request? Search by its trace ID, and read the timeline. What would you alert on? Symptoms and SLOs, like error rate and p ninety-nine latency. And what's on the main dashboard? Rate, errors and duration for each service, plus one business number, like orders per minute, which falls first when anything breaks.

The common mistake: ending with we'd add monitoring, and nothing more. Name the metrics, the SLO, and what pages someone at night. That's the difference between a phrase and a plan.

Quick check

Quick check
?
An SLO of 99.9% a month.
How much bad time does the
error budget allow?
About 43 minutes a month.
?
A trace: 9 s in total; restaurant 0.1
s, rider matching 0.3 s, payments
8.2 s. Where do you look first?
Payments: it holds 8.2 of the 9
seconds.
Remember
Metrics: what; traces: where;
logs: why
One trace ID, through every
call and log
SLOs and budgets; alert on
symptoms
Quick check

An SLO of 99.9% a month. How much bad time does the error budget allow?

Show the answer

About 43 minutes a month.

Quick check

A trace: 9 s in total; restaurant 0.1 s, rider matching 0.3 s, payments 8.2 s. Where do you look first?

Show the answer

Payments: it holds 8.2 of the 9 seconds.

Remember: metrics say something's wrong, traces say where, and logs say why. Pass one trace ID through every call and every log line. And set SLOs with error budgets, and alert on symptoms users feel. That's week five: services, failures, sagas, IDs, and seeing inside them all.

Practise with flashcardsYour progress stays on this device.
New System Design videos at 6 pm

Lessons on weekdays, the week's revision on Saturday, a mock interview on Sunday.