awk Associative Arrays: Dedup and Group
awk arrays are associative, indexed by strings, which makes !seen[$0]++ a one-line uniq that keeps the first occurrence.
The seen idiom is the most cited awk one-liner: it dedups a stream while preserving order, something sort -u cannot do.
What it does
awk arrays map string keys to values. seen[$0]++ stores and post-increments a counter per line; the first time a line appears the count is 0 (falsy), so !seen[$0]++ is true exactly once per unique line. The in operator and delete manage keys.
Common usage
# dedup keeping first occurrence and original order
awk '!seen[$0]++' input.txt
# dedup by a key column instead of the whole line
awk '!seen[$1]++' data.txt
# group: count rows per key
awk '{count[$1]++} END {for (k in count) print k, count[k]}' rows.txt
# membership test
awk 'NR==FNR{ok[$1]; next} $1 in ok' allow.txt data.txtIdioms
| Snippet | What it does |
|---|---|
| !seen[$0]++ | Print each unique line once, preserving order |
| !seen[$1]++ | Dedup by the first field |
| count[$1]++ | Tally rows per key |
| k in arr | True if key k exists in arr |
| for (k in arr) | Iterate keys (order is unspecified) |
| delete arr[k] | Remove a key |
In CI
!seen[$0]++ dedups a list of changed files or test names while keeping the order they appeared, which sort -u would scramble. For grouping (errors per file, counts per status), an array plus an END loop produces the tally in a single pass.
Common errors in CI
for (k in arr) iterates in an unspecified order; do not rely on it being sorted or insertion order (gawk can sort with PROCINFO, mawk cannot). Referencing arr[k] in a condition like (k in arr) is correct, but writing if (arr[k]) auto-creates the key as a side effect, inflating later counts; use the in operator to test without creating. Large keysets grow memory, so a huge unbounded log can blow up the dedup.