Pipelines that answer real questions: awk, tee and pipefail
Answer real questions from a web server log in one line: awk to pick fields, sort and uniq to rank, tee to save, and pipefail so a failure isn’t hidden.
Someone asks "who is hammering the site with bad URLs?" and the answer is in the web server's log. You don't need a dashboard. You need one pipeline, built a piece at a time. The log here is short; the same pipeline works on a million lines.
Pick the fields with awk
Every line of a web server's access log has the same shape:
198.51.100.23 - - [12/Jan/2026:08:01:37 +0000] "GET /.env HTTP/1.1" 404 162 "-" "python-requests/2.31"
awk splits each line on spaces and numbers the pieces: $1 is the address, $7 the page, $9 the status and $10 the size in bytes. A pattern in front of the braces picks the lines:
awk '$9 == 404 {print $1}' logs/access.log | head -n 3Tip:Why not grep 404? It matches 404 anywhere on the line, including a size like 14044. awk compares the one field you mean.
Rank them, and keep a copy with tee
You know sort | uniq -c | sort -nr from the Beginner lessons. Put tee at the end and the result goes to the screen and to a file, so you can check it and send it on:
awk '$9 == 404 {print $1}' logs/access.log | sort | uniq -c | sort -nr | tee 404s.txtawk can also add things up. An array keyed by page, then a loop at the END:
awk '{bytes[$7] += $10} END {for (p in bytes) print bytes[p], p}' logs/access.log | sort -nr | head -n 3When a pipe hides a failure
A pipeline's exit status is the status of its last command. So this says everything went fine, even though the file doesn't exist:
cat missing.log | sort | head -n 1; echo "status: $?"set -o pipefail makes the pipeline fail when any part fails. Scripts that matter should have it (the next lessons put it at the top of every script):
set -o pipefail; cat missing.log | sort; echo "status: $?"And errors don't go down a pipe unless you send them: 2>&1 joins error output to normal output, so some-command 2>&1 | grep -i error searches both.
Your turn
Find the three addresses that got the most 404s, most first, and save that list to ~/report.txt while it shows on screen. Then print the three pages that sent the most bytes in total, biggest first.
Your task
Find the three addresses that got the most 404s, most first, and save that list to ~/report.txt while it shows on screen. Then print the three pages that sent the most bytes in total, biggest first.
- The top three 404 addresses, in ~/report.txt and on screen (not done yet)
- The three pages that sent the most bytes, biggest first (not done yet)
The checklist ticks itself off as you work in the terminal: any way that gets the result counts.
Hints come one at a time, then one way to do it. Try each before the next.
What this lesson uses, in one place.
awk- Scan text for patterns and work with its columns More on awk
sort -nr- Sort by the number at the start of each line, so 9 comes before 10; reverse the order More on sort
tee- Save what passes through a pipe to a file, and pass it on More on tee
set -o- A pipeline fails if any command in it fails
2>&1- Sends error output to the same place as normal output, so errors go down the pipe too
Every everyday command on one page: the Linux commands cheat sheet.