pbzip2: Parallel bzip2 for Faster CI
pbzip2 compresses to the standard .bz2 format using multiple threads, cutting wall-clock time for large files on multi-core runners.
pbzip2 is to bzip2 what pigz is to gzip. Files it writes are read by any bzip2, so consumers need nothing special.
What it does
pbzip2 splits input into blocks compressed in parallel with the bzip2 algorithm, producing a standard .bz2 stream. Files created by pbzip2 decompress with bzip2, bunzip2, or pbzip2. Note that pbzip2 can only parallelize decompression of files it (or another block-aware writer) created.
Common usage
tar cf - data/ | pbzip2 -9 -p8 > data.tar.bz2
pbzip2 -p$(nproc) big.sql
pbzip2 -d data.tar.bz2 # decompress
pbzip2 -k -9 report.csv # keep sourceOptions
| Flag | What it does |
|---|---|
| -p<n> | Number of processors to use (e.g. -p8) |
| -9 | Best compression (default) |
| -k / --keep | Keep the input file |
| -c / --stdout | Write to stdout |
| -d / --decompress | Decompress |
| -r / --read | Read entire input before splitting (for pipes) |
In CI
Install with apt-get install -y pbzip2. Note the flag form is -p8 (no space), unlike some tools. Use it when you already standardized on .bz2; otherwise zstd -T0 is usually faster and smaller.
Common errors in CI
"pbzip2: command not found" means the package is missing; apt-get install pbzip2. "-p<n>: with a space" can be misread; write -p8 with no space. Decompressing a bzip2 stream written by plain bzip2 falls back to single-threaded, which is expected, not an error.