uniq: Usage, Options & Common CI Errors

uniq collapses or counts consecutive identical lines.

uniq is deceptively simple: it only looks at adjacent lines, so it must follow sort to deduplicate a whole file. Forgetting that silently lets duplicates through.

What it does

uniq reads input and removes (or counts, or isolates) runs of consecutive identical lines. Non-adjacent duplicates are not detected unless the input is sorted first.

Common usage

sort file.txt | uniq            # dedup a whole file
sort file.txt | uniq -c         # count occurrences
sort file.txt | uniq -d         # only duplicated lines
sort file.txt | uniq -u         # only lines that appear once
uniq -c log.txt | sort -rn      # frequency, most common first

Options

Common errors in CI

The number-one mistake: running uniq on unsorted input, so duplicates spread across the file survive - always sort first (or use sort -u). uniq itself rarely errors, but a pipeline that assumed dedup-then-count will report wrong totals if the sort was skipped. For counting unique values regardless of order, sort | uniq -c is the canonical idiom.

Related guides