Skip to content

Process substitution: compare commands with <( )

NinjaLesson 51 of 669 min

Use <( ) to diff the output of two commands without temporary files, and fix the classic bug where a loop at the end of a pipe forgets its variables.

Some commands only take file names. diff compares two files. But what you want to compare is often the output of two commands: two sorted lists, a config on two servers, yesterday's process list and today's.

The slow way is temporary files: sort one into a.tmp, the other into b.tmp, diff them, delete them. The quick way is process substitution:

diff <(sort lists/march.txt) <(sort lists/april.txt)

<(command) runs the command and gives the outer command a file name to read its output from. You can see the name:

echo <(true)

That /dev/fd/… path is a pipe the shell made, so nothing is written to disk. In diff's answer, < lines are only in the first list and > lines only in the second.

Tip:It works with anything that reads files: wc -l <(ls), or diff <(ssh web cat /etc/hosts) /etc/hosts to compare a file on two machines.

The loop that forgets

Here is a bug that has cost people whole afternoons. This loop counts lines, and still prints 0:

n=0; cat lists/march.txt | while read -r line; do n=$((n + 1)); done; echo "$n"

Every part of a pipeline runs in its own copy of the shell (a subshell). The while loop counted happily in its copy, and the copy was thrown away when the pipe ended. The outer n never changed.

The fix is to keep the loop in your shell and feed it with process substitution instead of a pipe:

n=0; while read -r line; do n=$((n + 1)); done < <(cat lists/march.txt); echo "$n"

The first < is ordinary input redirection; the <( ) after it is the command's output. The space between them matters.

Your turn

Show who left and who joined between the March and April lists with one diff of the two sorted lists, without temporary files. Then fix ~/bin/big-orders.sh, which always says 0, and run it.

Next