awk Count Unique Values in a Column
awk tallies distinct values by incrementing an array keyed on the column, then prints each key and its count in END.
Counting how often each status, file, or error appears is a one-pass array tally. It replaces sort | uniq -c and lets you key on any field.
What it does
For each line, count[$N]++ increments the tally for that field value. After all input, an END loop over for (k in count) prints each distinct value with its count. Unlike sort | uniq -c, no sorting is required and you choose the key column.
Common usage
# count occurrences of each value in column 1
awk '{c[$1]++} END {for (k in c) print c[k], k}' data.txt
# count HTTP status codes (field 9 of access log)
awk '{c[$9]++} END {for (k in c) print k, c[k]}' access.log
# tally then sort by count, descending
awk '{c[$1]++} END {for (k in c) print c[k], k}' data.txt | sort -rnIdioms
| Snippet | What it does |
|---|---|
| c[$1]++ | Increment the count for this key |
| for (k in c) print k, c[k] | Emit each distinct value and its count |
| END{print length(c)} | gawk: number of distinct values |
| | sort -rn | Sort the tally by count, highest first |
| c[$1 FS $2]++ | Count on a composite key (two fields) |
In CI
Tally test results or error types in one pass, then write the count of the worst bucket to $GITHUB_OUTPUT. To rank the output, pipe the END loop through sort -rn, since for (k in c) does not emit in any guaranteed order.
Common errors in CI
The iteration order of for (k in c) is unspecified, so always sort afterward if order matters. Counting on the wrong field (a separator mismatch) tallies the wrong column; verify -F first. A composite key needs an explicit separator (c[$1 SUBSEP $2] or c[$1 FS $2]) so that "a b" and "ab" do not collide.