jq unique and unique_by : Drop Duplicates

jq unique sorts an array and removes duplicate elements; unique_by(f) dedupes by a derived key.

Deduping is common when an API paginates overlapping results or when two sources are merged before building a matrix.

What it does

unique sorts the array and removes equal elements, returning the distinct values. unique_by(f) keeps the first element for each distinct value of f, which is useful for objects where you dedupe by one field.

Common usage

Terminal

# distinct scalar values
echo '[3,1,2,1,3]' | jq 'unique'
# distinct objects by a field
jq 'unique_by(.author)' commits.json
# count distinct values
jq '[.[].label] | unique | length' issues.json

Functions

Function	What it does
unique	Sort and remove duplicate elements
unique_by(f)	Keep one element per distinct f
group_by(f)	Group instead of dedupe
length	Count after deduping

Common errors in CI

"jq: error: Cannot iterate over null (null)" means unique got null; wrap the values in [ ... ] to form an array first. "jq: error: null and string cannot be compared" from unique_by means f is missing on some objects; default it with (.field // "") before deduping.

Related guides

jq sort_by, min_by and max_by : Order and Pickjq sort_by(.field) orders an array and min_by/max_by pick the extreme element. Reference for ordering gh api… jq group_by : Group Array Elements by a Keyjq group_by(.field) partitions an array into sub-arrays sharing a key value. Reference for bucketing gh api r… jq add : Sum or Concatenate an Arrayjq add reduces an array by summing numbers, concatenating strings/arrays, or merging objects. Reference for a… jq length : Count Elements or String Sizejq length returns array size, object key count, or string length. Reference for counting gh api results in CI…