xsv index and join: Fast Lookups Across CSVs
xsv index builds a sidecar index that makes count, slice and stats near-instant, and xsv join merges two CSVs on a shared key column.
Indexing a CSV once turns repeated row-count and slice operations from full scans into O(1) seeks, and join brings relational merges to flat files.
What it does
xsv index writes a .idx file next to the CSV recording row offsets; subsequent count, slice and stats use it for fast random access and parallelism. xsv join performs an inner (or left/right/full) join of two CSVs on named columns.
Common usage
# build an index, then count is instant
xsv index big.csv
xsv count big.csv
# slice rows 100-110 without scanning the whole file
xsv slice -s 100 -e 110 big.csv
# inner join two files on a key column each
xsv join id orders.csv cust_id customers.csvOptions
| Subcommand / flag | What it does |
|---|---|
| index <file> | Create the .idx sidecar for fast access |
| count | Count rows (instant with an index) |
| slice -s <n> -e <m> | Extract a row range by index |
| join <col1> f1 <col2> f2 | Join two CSVs on the named columns |
| --left / --right / --full | Outer join variants |
| --no-case | Case-insensitive key comparison |
In CI
Index a large fixture once at the start of a job so repeated slice/count checks across test cases stay fast. Use xsv join to reconcile two exported datasets (for example expected vs actual) and pipe the result into a diff or a row-count assertion.
Common errors in CI
A stale .idx after the CSV changes gives wrong counts or a CSV index ... is out of date style error; delete and rebuild the index. join silently drops non-matching rows in the default inner mode, which can look like data loss; use --left to keep all left rows. Joining on a mistyped column name errors with a selection failure naming the bad column.