Metrics & Dashboards

A metric is a number, sampled repeatedly over time - CPU percent, requests per second, queue depth. Graph enough of the right ones, and a system's health becomes visible at a glance instead of something you have to go dig for.

Three shapes of metric

• Counter - only ever goes up (total requests served since start). Useful as a rate - "requests per second" is really "how fast is this counter climbing."
• Gauge - goes up and down freely (current memory usage, current queue length).
• Histogram - buckets of values, usually for latency ("how many requests finished in under 100ms? Under 500ms?").

Pull vs. push

Two philosophies for getting a metric from an application into a monitoring system:

• Push - the application sends its metrics to a central collector on its own schedule.
• Pull (the model Prometheus popularized) - the monitoring system reaches out and scrapes a known HTTP endpoint on each application, on its own schedule.

Pull has a quietly useful property: if a scrape ever fails, the monitoring system knows immediately that something is wrong with that target - with push, a server going silent from a crash looks identical to a server with simply nothing new to report.

What to actually graph

For almost any service, four numbers go a very long way - sometimes called the RED method:

• Rate - requests per second.
• Errors - how many of those are failing.
• Duration - how long they take.

For the underlying machine itself, the USE method (from the earlier Linux performance lesson) covers the rest: Utilization, Saturation, Errors, for CPU/memory/disk/network.

Dashboards

Grafana is the most common layer sitting on top of a metrics store like Prometheus - turning raw numbers into the graphs a human actually scans during an incident.

Try it

free -h

When you're ready, hit Mark complete and move to the next lesson.