Alerting That Doesn't Cry Wolf

Metrics and logs are only useful if someone - or something - actually looks at them. Alerting is what turns "the data shows a problem" into "a human was told about the problem."

Symptom-based, not cause-based

The most durable alerting rule: page on symptoms users would notice, not every possible internal cause.

• Cause-based: "disk is at 85%." Maybe fine, maybe not - depends entirely on context.
• Symptom-based: "error rate is above 5% for 5 minutes" or "p99 latency exceeds 2 seconds." This is something a real user is feeling, right now.

A disk slowly filling isn't automatically an emergency. A spike in failed checkouts always is.

Alert fatigue

If an on-call engineer gets paged every night for something that turns out to be fine, they learn - correctly, if irrationally - to tune out pages. The night it's real, that instinct costs you. Every alert that fires and isn't actionable is a small withdrawal from a trust account that's expensive to refill.

The fix isn't fewer metrics - it's fewer pages. Keep collecting everything; only page on what's both severe and something a human can actually do something about right now.

Severity tiers

Not everything deserves the same response:

Runbooks

A good alert links straight to a runbook - a short, specific document: "when this fires, check X, then Y; if that doesn't fix it, escalate to Z." The goal is for the first responder, possibly half-asleep at 3 AM, to have a clear next step instead of starting from zero.

Try it

ps aux | grep java

When you're ready, hit Mark complete and move to the next lesson.