Text Processing with grep, sed, and awk

Three small tools handle almost every text-processing job you will run into on the command line: grep finds lines, sed transforms them, and awk extracts fields out of them. Learn these three and you can process logs, configs, and CSVs without ever opening an editor.

grep: finding lines that match

grep ERROR app.log          # lines containing "ERROR"
grep -i error app.log       # case-insensitive
grep -v INFO app.log        # invert: lines that do NOT match
grep -n ERROR app.log       # show line numbers

grep reads from a file or from a pipe, which is how it is most often used in practice:

ps aux | grep nginx

sed: transforming text

sed is a stream editor - it reads text, applies an edit, and prints the result. The classic edit is substitution, s/find/replace/:

sed 's/TODO/DONE/' report.txt      # replaces the FIRST match on each line
sed 's/TODO/DONE/g' report.txt     # the g flag replaces EVERY match on each line

By itself, sed only prints the changed text - it does not touch the file. To keep the result, pipe it into tee and write back to the same filename:

sed 's/TODO/DONE/g' report.txt | tee report.txt

awk: extracting fields

Text is often organized into whitespace-separated fields - a process list, a log line, a CSV row. awk refers to them as $1, $2, and so on ($0 is the whole line):

echo "alice 27 engineer" | awk '{print $1}'   # -> alice
echo "alice 27 engineer" | awk '{print $2}'   # -> 27

cut does a simpler version of the same job when your data has an explicit delimiter, like a colon or comma:

echo "alice:27:engineer" | cut -d ":" -f 2    # -> 27

Try it

echo "TODO fix the login bug" > report.txt
echo "TODO write the changelog" >> report.txt
cat report.txt
sed 's/TODO/DONE/g' report.txt
grep -c TODO report.txt

Further reading

• "sed & awk" by Dale Dougherty and Arnold Robbins (O'Reilly) - the definitive, classic reference for both tools, still relevant decades after publication.
• "The Linux Command Line" by William Shotts (No Starch Press) - its text-processing chapters cover grep, sed, and awk together with plenty of realistic examples, plus an introduction to regular expressions.