awk: columns, filters and totals
Use awk to pull columns out of a web server log, keep the lines you want and add up a CSV column, total and per group. Practise awk in your browser.
access.log is a web server's log: one line per request, with the visitor's address, the time, the page and the answer. Which pages were missing? Who visits most? grep finds lines, but these questions are about columns, and that's what awk is for.
head -n 3 access.logFields
awk reads its input one line at a time and splits each line at spaces into fields: $1 is the first, $2 the second, and $NF the last (NF is how many there are). $0 is the whole line. In this log, $1 is the address, $7 the page and $9 the status code.
awk '{print $1}' access.logawk '{print $9, $7}' access.logAlways put the program in single quotes. In double quotes the shell would replace $1 with its own (empty) variable before awk ever saw it.
Pattern { action }
An awk program is a list of pattern { action }. The action runs on the lines where the pattern is true. Leave out the action and awk prints the line; leave out the pattern and the action runs on every line:
awk '$9 == 500' access.logawk '$9 >= 400 {print $9, $7}' access.logOther separators: -F
A CSV file uses commas, so tell awk with -F,. NR is the line number, so NR > 1 skips the header:
awk -F, 'NR > 1 {print $2, $3}' sales.csvAdding up: variables and END
Variables need no setting up: they start at 0. An END block runs after the last line, which is the place to print a total:
awk -F, 'NR > 1 {total += $3} END {print total}' sales.csvAn array keeps one total per key. for (k in sum) goes through the keys (in no fixed order):
awk '{hits[$1]++} END {for (ip in hits) print hits[ip], ip}' access.logprintf formats numbers: printf "%s %.2f\n", $2, $3 prints two decimals.
Your turn
Three questions, each answered with awk. (1) Print the page (the 7th field) of every request in access.log that got a 404, one per line. (2) Print the total of the amount column in sales.csv. (3) Print each region in sales.csv with its own total, one region per line.
Your task
Three questions, each answered with awk. (1) Print the page (the 7th field) of every request in access.log that got a 404, one per line. (2) Print the total of the amount column in sales.csv. (3) Print each region in sales.csv with its own total, one region per line.
- The pages that got a 404 (not done yet)
- The total of all sales (650.5) (not done yet)
- Each region with its total (not done yet)
The checklist ticks itself off as you work in the terminal: any way that gets the result counts.
Hints come one at a time, then one way to do it. Try each before the next.
What this lesson uses, in one place.
awk- Scan text for patterns and work with its columns More on awk
awk -F- Fields are separated by SEP, such as -F, for CSV or -F: for /etc/passwd More on awk
awk NR- The line number and the number of fields on the line More on awk
awk END- Run before the first line and after the last More on awk
Every everyday command on one page: the Linux commands cheat sheet.