Skip to content

Pipelines that answer real questions: awk, tee and pipefail

NinjaLesson 50 of 669 min

Answer real questions from a web server log in one line: awk to pick fields, sort and uniq to rank, tee to save, and pipefail so a failure isn’t hidden.

Someone asks "who is hammering the site with bad URLs?" and the answer is in the web server's log. You don't need a dashboard. You need one pipeline, built a piece at a time. The log here is short; the same pipeline works on a million lines.

Pick the fields with awk

Every line of a web server's access log has the same shape:

198.51.100.23 - - [12/Jan/2026:08:01:37 +0000] "GET /.env HTTP/1.1" 404 162 "-" "python-requests/2.31"

awk splits each line on spaces and numbers the pieces: $1 is the address, $7 the page, $9 the status and $10 the size in bytes. A pattern in front of the braces picks the lines:

awk '$9 == 404 {print $1}' logs/access.log | head -n 3

Tip:Why not grep 404? It matches 404 anywhere on the line, including a size like 14044. awk compares the one field you mean.

Rank them, and keep a copy with tee

You know sort | uniq -c | sort -nr from the Beginner lessons. Put tee at the end and the result goes to the screen and to a file, so you can check it and send it on:

awk '$9 == 404 {print $1}' logs/access.log | sort | uniq -c | sort -nr | tee 404s.txt

awk can also add things up. An array keyed by page, then a loop at the END:

awk '{bytes[$7] += $10} END {for (p in bytes) print bytes[p], p}' logs/access.log | sort -nr | head -n 3

When a pipe hides a failure

A pipeline's exit status is the status of its last command. So this says everything went fine, even though the file doesn't exist:

cat missing.log | sort | head -n 1; echo "status: $?"

set -o pipefail makes the pipeline fail when any part fails. Scripts that matter should have it (the next lessons put it at the top of every script):

set -o pipefail; cat missing.log | sort; echo "status: $?"

And errors don't go down a pipe unless you send them: 2>&1 joins error output to normal output, so some-command 2>&1 | grep -i error searches both.

Your turn

Find the three addresses that got the most 404s, most first, and save that list to ~/report.txt while it shows on screen. Then print the three pages that sent the most bytes in total, biggest first.

Next