sort, uniq, and the Counting Pipeline

Two more tools, then a pattern worth memorizing.

sort orders lines alphabetically, or numerically with -n, or backwards with -r.

uniq collapses adjacent duplicate lines, and with -c prefixes each with how many there were.

✦ uniq only sees neighbors

This catches everyone once. uniq compares each line to the one immediately before it, so duplicates scattered through a file are invisible to it. That is why sort almost always comes first: sorting brings identical lines together so uniq can see them. sort | uniq is nearly a single idea, and uniq on its own is usually a bug.

The counting pipeline. This shape answers so many questions that it is worth learning as one unit:

`

sort | uniq -c | sort -rn

`

Sort to group, count the groups, then sort those counts highest first. Feed it a log and it tells you the most common errors. Feed it a list of file extensions and it tells you what a codebase is made of.