uniq Requires Sorted Input: Why and How
uniq compares each line only to the one before it, so duplicates must be adjacent to be collapsed.
The single most common uniq mistake is running it on unsorted data. uniq is a streaming, adjacent-only filter by design.
What it does
uniq reads line by line and removes a line only when it equals the immediately preceding line. It deliberately does not buffer or sort, which keeps it fast and constant-memory but means non-adjacent duplicates survive. Sorting first puts equal lines next to each other so uniq can collapse them.
Common usage
sort file.txt | uniq # correct: sorted, then deduped
sort -u file.txt # same result in one process
# wrong: leaves non-adjacent duplicates
uniq file.txtOptions
| Flag | What it does |
|---|---|
| (default) | Collapse adjacent duplicate lines |
| -c | Count occurrences (needs sorted input to be meaningful) |
| -i | Ignore case in the comparison |
| -f N / -s N | Skip leading fields or characters when comparing |
In CI
If you only need a deduplicated set, prefer sort -u: it is one process and cannot be misused on unsorted input. Use sort | uniq -c when you also want counts.
Common errors in CI
Duplicates remain after uniq because the input was not sorted; this is the expected adjacent-only behavior, not a bug. uniq -c counts are wrong for the same reason. Note that uniq with -f/-s compares a substring, so two visually different lines can be treated as equal.