Toil Reduction & Automation
The SRE habit of never doing the same manual fix a third time.
On this page4 sections
This lesson closes the loop on the whole track: the point of measuring reliability and responding well to incidents is to free up time for the thing that actually prevents the next one - eliminating toil.
What counts as toil
Not all operational work is toil. Work is toil specifically when it's:
- Manual - a human doing it by hand, not a system.
- Repetitive - the same task, again and again.
- Automatable - a script genuinely could do it.
- Of no enduring value - once it's done, nothing is actually better than before; you're just back to baseline until the next time it recurs.
Clearing disk space on a server by hand, every week, because logs fill it up, is toil in exactly this sense: real work, that accomplishes nothing new, forever.
Why SRE caps it
Google's own guidance suggests capping toil at around 50% of an SRE's time. The reasoning is direct: time spent on toil is time not spent on the engineering work - automation, better tooling, fixing root causes - that would actually reduce toil, and incidents, going forward. Left unchecked, toil grows to fill all available time, and the team never gets ahead of it.
The loop
A simple, repeatable habit for not accumulating toil in the first place:
- Do it manually once - understand the problem for real.
- Script it the second time - stop relying on memory and manual steps.
- Schedule or automate it entirely the third time - a human shouldn't need to remember to run it at all.
Try it
That log-cleanup example from earlier, automated the way this whole curriculum has been building toward - cron, the same tool from the very first Job Scheduling lesson:
echo '0 0 * * * /usr/local/bin/cleanup-tmp.sh' > cleanup-job
When you're ready, hit Mark complete. You've completed the Site Reliability Engineering track!
Challenge
Write a crontab line into a file called cleanup-job that runs /usr/local/bin/cleanup-tmp.sh every day at midnight, automating away a task you'd otherwise do by hand.
Want to experiment first? Try things in the practice terminal (opens in a new tab), then enter the command that solves the task here.
Quiz
Lock in the concepts. Score at least 80% to mark the lesson complete.
Done with this one?
Skipped the challenge? You can mark the lesson complete by hand.