Incident Response
Severity levels, the incident commander role, and the discipline of mitigating before explaining.
On this page4 sections
An incident is any unplanned event degrading or interrupting a service. What separates a team that handles incidents well from one that doesn't usually isn't technical skill - it's having an agreed process before the pressure of a live outage ever starts.
Severity levels
Not every incident deserves the same response. Most teams land on something like:
| Severity | Example | Response |
|---|---|---|
| SEV1 | Full outage, every user affected | Immediate, all-hands |
| SEV2 | Degraded for a subset of users | Urgent, dedicated responder |
| SEV3 | Minor, workaround exists | Normal business-hours priority |
Agreeing on these tiers before an incident means nobody has to argue about how urgent something is while it's actively happening.
The incident commander
For anything serious, one person takes the incident commander role: coordinating the response, making the call on what to try next, and handling communication - explicitly not necessarily the person actually typing commands to fix it. Separating "coordinating" from "fixing" keeps the person doing hands-on work from also having to context-switch into managing status updates.
Mitigate first, root-cause second
The single most important habit: stop the bleeding before you fully understand it. Roll back the last deploy, restart the stuck service, fail over to a backup - any of these can restore service in minutes, long before a full root-cause investigation would finish. The investigation absolutely still happens - just afterward, calmly, as a postmortem, not under the pressure of an ongoing outage.
Try it
ps aux | grep nginx
When you're ready, hit Mark complete and move to the next lesson.
Challenge
An incident report says nginx might be down - list every process and filter it down to the ones mentioning nginx to confirm.
Want to experiment first? Try things in the practice terminal (opens in a new tab), then enter the command that solves the task here.
Quiz
Lock in the concepts. Score at least 80% to mark the lesson complete.
Done with this one?
Skipped the challenge? You can mark the lesson complete by hand.