Observability: Logs, Metrics & Traces

Question 1

Which observability pillar is best for answering "What exactly happened at 12:00:01.310 inside the payments service?"

Accepted Answer

Logs — they are discrete, timestamped, human-readable event records — Logs are the right tool for pinpointing a specific moment in time. Each log line carries a precise timestamp, a severity level, the emitting service, and a human-readable message. Metrics aggregate many events into a number and lose per-event detail; traces describe the path of a request but do not reproduce the raw event text.

Question 2

What is the primary weakness of logs compared to metrics?

Accepted Answer

Querying logs for trends across millions of events requires scanning each individual line, whereas metrics pre-aggregate that data — Logs shine for investigating one specific moment but they do not aggregate. To find "how many checkout requests failed in the last hour?" you would need to scan and count potentially millions of individual log lines. Metrics pre-aggregate that count into a single number, making trend detection and alerting far more efficient.

Question 3

A metrics dashboard shows your error rate has spiked to 15%. Which observability pillar would you turn to next to find out which service is causing the errors?

Accepted Answer

A distributed trace for a representative failing request, which shows which service span is errored or unexpectedly wide — A metric tells you *that* something is wrong. The next step is to look at a trace for a representative failing request. The trace's waterfall will show you exactly which service's span is red (failed) or unexpectedly wide (slow), giving you a target for deeper log investigation.

Question 4

What is a "span" in the context of distributed tracing?

Accepted Answer

A named operation within a trace, recording the service, start time, and duration of one unit of work — A span represents one named operation (e.g., "payments-svc: charge card") within a trace. Spans have a start time and a duration, and they nest: a parent span (e.g., the API gateway handling the full request) contains child spans for each downstream service it calls, forming a tree that maps the complete call chain.

Question 5

Why are distributed traces particularly valuable in microservices architectures rather than in a monolith?

Accepted Answer

In a microservices system a request crosses multiple independent processes; a trace stitches those hops into one view — In a monolith, the entire request executes in one process, so a single log file contains the full picture. In a microservices system, a checkout request might touch an API gateway, an orders service, a payments service, and a notification service — each running independently with its own logs. A trace propagates a shared trace-ID across all those services so you can reconstruct the complete end-to-end journey in one view.

Question 6

p99 latency for your checkout endpoint is 850 ms but the average latency is 120 ms. What does this tell you?

Accepted Answer

Most users see fast responses, but the slowest 1% of requests wait up to 850 ms — a tail the average completely obscures — p99 latency means 99% of requests complete faster than that number, but the slowest 1% can take up to 850 ms. Latency distributions are typically right-skewed (long tail), so the average can look healthy while a meaningful fraction of real users experience painful slowness. That tail is what causes user complaints and SLA breaches — which is why p99 (and p95) are standard SLO metrics.

Question 7

What is a metric alert threshold used for?

Accepted Answer

Triggering a notification when a metric crosses a defined boundary — e.g., error rate > 5% or p99 > 500 ms — Alert thresholds turn passive metric collection into proactive notifications. You define a rule like "fire a PagerDuty alert if p99 latency > 500 ms for 5 consecutive minutes." Without alerts, an engineer would need to stare at dashboards 24/7 to catch problems. Alerting on metrics is the first layer of incident detection.

Question 8

What is the "cardinality" problem in metrics, and why can it prevent you from adding user IDs as metric labels?

Accepted Answer

Each unique label-value combination creates a separate time series; a userId label with millions of users exhausts memory and query capacity — Metric systems like Prometheus store one time series per unique label set. A label like `user_id=12345678` would create a distinct time series for every unique user — potentially millions. This explodes storage and query cost. The fix is to keep metric labels low-cardinality (status codes, service names, regions) and move per-entity detail to logs or traces instead.

Question 9

How do logs, metrics, and traces complement each other during an incident response?

Accepted Answer

Metrics alert you that something is wrong; a trace localises the failing service or span; logs expose the exact root cause — The three pillars form a funnel: broad (metrics alert) → narrowing (trace isolates the service) → specific (logs expose the root cause). Skipping any layer forces you to brute-force search across the others. A mature observability practice correlates all three, for example by embedding a trace ID in each log line so you can jump from a log entry directly to its parent trace.

Question 10

Which of the following is a log-level convention and what does it signal?

Accepted Answer

ERROR — something failed that likely requires attention; the operation did not complete successfully — Log levels (DEBUG, INFO, WARN, ERROR, FATAL) communicate severity. ERROR means an operation failed — for example, a card-decline exception in the payments service. WARN signals something unexpected but non-fatal. INFO is routine operational noise. Setting alert rules on the *count* of ERROR-level log lines is a common bridge between logging and metrics-based alerting.

Logs, metrics or traces?

Test your knowledge

Which observability pillar is best for answering "What exactly happened at 12:00:01.310 inside the payments service?"

The short version

Interview Questions

References & Further Reading

Related topics

Love this visualization?