uniq Command Reference for CI Scripts

uniq collapses or counts consecutive identical lines.

uniq is deceptively simple: it only looks at adjacent lines, so it must follow sort to deduplicate a whole file. Forgetting that silently lets duplicates through.

Common flags/usage

• -c / --count: prefix each line with its count
• -d / --repeated: print only duplicated lines
• -u / --unique: print only non-duplicated lines
• -i / --ignore-case: case-insensitive comparison
• -f N / -s N: skip first N fields / chars when comparing

Example

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
sort emails.txt | uniq -d        # show only duplicates

In CI

The number-one mistake is running uniq on unsorted input, so duplicates spread across the file survive; always sort first or use sort -u. For counting unique values regardless of order, sort | uniq -c is the canonical idiom for log frequency reports.

Key takeaways

• uniq only collapses adjacent lines, so sort the input first.
• sort | uniq -c is the canonical idiom for frequency counts.
• -d shows only duplicates; -u shows only lines that appear once.

Related guides