Pipes: chaining commands

The pipe: stdout of one, stdin of the next

Redirection connects a command to a file. The pipe | connects a command directly to another command: the stdout of the left side becomes the stdin of the right side.

grep error app.log | head -n 5

grep finds every error line, and head keeps only the first five. No temporary file needed.

This is the famous Unix philosophy: many small programs, each doing one thing well, snapped together like plumbing. You already know grep, head, tail, and sort. One more joins here: wc -l (word count, -l for lines) counts the lines it receives.

Two pipelines, one with no file at all

The first pipeline finds lines containing 7 and keeps the first three of them. The second counts 200 lines without any file being involved, and tr -d ' ' strips the padding spaces some systems print around wc's number.

seq 1 50 > nums.txt
grep 7 nums.txt | head -n 3
seq 1 200 | wc -l | tr -d ' '

Output

7
17
27
200

In the first pipeline grep 7 matches five numbers, 7, 17, 27, 37, and 47, and head -n 3 keeps only the first three, which is why 37 and 47 never appear. The second pipeline is the more interesting one: seq's output never touches the disk, it flows straight into wc through the pipe.

tee: save a copy mid-pipeline

A pipe passes output onward and keeps nothing behind. Sometimes both are wanted, continued processing and a saved copy, without paying to run an expensive left-hand side twice. tee file does exactly that. Placed in the middle of a pipeline, it writes everything passing through into the file and also hands it along unchanged to the next command. The name comes from the T-shaped junction in plumbing, which splits one flow into two.

grep error app.log | tee errors.txt | wc -l

One grep run produces two results here. errors.txt holds the matching lines for later reading, and wc still receives the same lines and counts them now.

Counting and saving in a single pass

This pipeline counts how many of the numbers 1 through 100 contain a 7, while tee snapshots grep's full output into sevens.txt on the way past. The head afterwards proves the file really was written.

seq 1 100 | grep 7 | tee sevens.txt | wc -l | tr -d ' '
head -n 2 sevens.txt

Output

19
7
17

Three output lines come from two commands. The 19 is wc's count, reported at the end of the pipeline, and the 7 and 17 are the first two lines that tee saved into the file. Both results came from a single pass over the data.

In grep error app.log | wc -l, the input wc -l receives is only the lines of app.log that contain "error". The pipe hands it exactly what grep printed and nothing else, so the number wc reports is the count of error lines rather than the size of the file.

Chaining filters this way is the everyday superpower of the shell. Each stage narrows or transforms the stream, and the final stage sees only what survived every filter before it.

Three filters chain together to answer a counting question in one line. seq 1 30 generates the numbers, grep 2 keeps only those containing the digit 2, and wc -l counts what is left, with tr -d ' ' tidying the number.

seq 1 30 | grep 2 | wc -l | tr -d ' '

Output

12

The twelve matches are worth enumerating to see why the count is right: 2 and 12 on their own, plus the ten numbers from 20 through 29. No intermediate file is created at any point, since each stage streams its output directly into the next.