Skip to content

awk: columns, filters and totals

AdvancedLesson 36 of 669 min

Use awk to pull columns out of a web server log, keep the lines you want and add up a CSV column, total and per group. Practise awk in your browser.

access.log is a web server's log: one line per request, with the visitor's address, the time, the page and the answer. Which pages were missing? Who visits most? grep finds lines, but these questions are about columns, and that's what awk is for.

head -n 3 access.log

Fields

awk reads its input one line at a time and splits each line at spaces into fields: $1 is the first, $2 the second, and $NF the last (NF is how many there are). $0 is the whole line. In this log, $1 is the address, $7 the page and $9 the status code.

awk '{print $1}' access.log
awk '{print $9, $7}' access.log

Always put the program in single quotes. In double quotes the shell would replace $1 with its own (empty) variable before awk ever saw it.

Pattern { action }

An awk program is a list of pattern { action }. The action runs on the lines where the pattern is true. Leave out the action and awk prints the line; leave out the pattern and the action runs on every line:

awk '$9 == 500' access.log
awk '$9 >= 400 {print $9, $7}' access.log

Other separators: -F

A CSV file uses commas, so tell awk with -F,. NR is the line number, so NR > 1 skips the header:

awk -F, 'NR > 1 {print $2, $3}' sales.csv

Adding up: variables and END

Variables need no setting up: they start at 0. An END block runs after the last line, which is the place to print a total:

awk -F, 'NR > 1 {total += $3} END {print total}' sales.csv

An array keeps one total per key. for (k in sum) goes through the keys (in no fixed order):

awk '{hits[$1]++} END {for (ip in hits) print hits[ip], ip}' access.log

printf formats numbers: printf "%s %.2f\n", $2, $3 prints two decimals.

Your turn

Three questions, each answered with awk. (1) Print the page (the 7th field) of every request in access.log that got a 404, one per line. (2) Print the total of the amount column in sales.csv. (3) Print each region in sales.csv with its own total, one region per line.

Next