BE9: Testing and running in production
Logs, metrics and traces
🎯You'll be able to
Add structured logs, a few key metrics and request IDs, so you can tell what a live service is doing and why a request was slow.
🏠The everyday picture
A hospital has three ways to watch a patient: the notes nurses write (logs), the monitor showing heart rate and blood pressure over time (metrics), and the record of every department the patient visited and how long each took (a trace). Each answers a different question.
📘What it really is
- Logs record events. Write them as structured JSON, one per line, with a level, a message and fields:
{"level":"info","msg":"order placed","request_id":"7f3a","order_id":120043,"ms":42}. - include a request ID on every line for a request, passed between services in a header;
-
never log tokens, passwords or full personal details (BE6 Lesson 4).
-
Metrics are numbers over time, cheap to store and graph. For each route, the RED set:
- Rate: requests per second;
- Errors: failures per second (5xx, timeouts);
- Duration: latency percentiles (p50, p95, p99).
-
For queues: lag and pending count; for the outbox: unpublished rows.
-
Traces follow one request across services as spans (API → database → orders service → Redis). OpenTelemetry is the common standard; the
traceparentheader carries the trace ID between services. -
Together they're called observability: metrics tell you something's wrong, traces tell you where, logs tell you what exactly.
📝A worked example
Input: a customer reports that ordering took 4 seconds at 19:42.
Steps (an illustrative investigation):
- Metrics: the
POST /v1/ordersp95 graph shows a spike from 19:40 to 19:45; error rate is normal. - Traces: a slow trace from 19:42 shows 3.8 s in the span
SELECT ... FROM menu_items WHERE id = ANY(...). - Logs: filtered by that trace's request ID, a warning shows the database pool was waiting for a free connection.
Output: the cause is pool exhaustion during a peak (BE3 Lesson 4), not the order code itself. In the kit, /health on the orders service
already reports outbox_pending, a simple metric.
🌍Where it's used
- Prometheus and Grafana for metrics and dashboards; Loki, Elasticsearch or cloud logging for logs.
- OpenTelemetry with Jaeger, Tempo or a commercial tool for traces.
- On-call teams everywhere, who start from a dashboard and drill into traces and logs.
📝Case scenario
A school results site was slow on results day, and the team spent three hours reading plain-text logs from 12 servers with no request IDs. After adding request IDs, RED metrics per route and tracing, a similar slowdown the next term was traced to one uncached query in under ten minutes.
🖱️Try it
Add a small middleware to the finished API that gives each request an ID (from an X-Request-Id header, or a new random one), returns it
in the response, and logs one JSON line per request with method, path, status and milliseconds. Success: every log line has a request ID,
and no line contains a token.
⚠️Common mistakes
- Plain-text logs. Sign: searching logs with guesswork. Fix: structured JSON with consistent field names.
- Only averages. Sign: "average 80 ms" while some users wait 5 seconds. Fix: track p95 and p99.
- Logging secrets. Sign:
Authorizationheaders in logs. Fix: log a short list of safe fields, never whole headers or bodies.
🛠️For your project
Your capstone adds request IDs and one JSON log line per request to the API, and shows a sample in the demo.
✅Quick check
What are the RED metrics?
Show the answer
Rate, errors and duration, measured per route.
What does a trace show that logs don't?
Show the answer
How long each step of one request took across services, in one picture.