pigz: Parallel gzip for Fast CI Artifacts
pigz compresses to the standard gzip format using multiple threads, so large artifacts compress much faster on multi-core runners.
pigz is a drop-in speed upgrade for gzip. The output is a normal .gz any tool can read; the win is that it saturates every core with -p.
What it does
pigz (parallel implementation of gzip) splits the input into blocks and compresses them across threads, emitting a standard gzip stream. Decompression is largely single-threaded but still fast. gunzip and gzip -d read pigz output normally.
Common usage
tar cf - build/ | pigz -9 -p 8 > build.tar.gz
pigz -p $(nproc) large.log # use every core
pigz -d cache.tar.gz # decompress (like gunzip)
tar -I pigz -cf out.tar.gz dir/ # let tar drive pigzOptions
| Flag | What it does |
|---|---|
| -p <n> | Number of threads (default: all online CPUs) |
| -9 / --best | Maximum compression |
| -1 / --fast | Fastest, least compression |
| -k / --keep | Keep the input file |
| -c / --stdout | Write to stdout |
| -d / --decompress | Decompress a .gz file |
In CI
Pipe tar into pigz (tar cf - dir | pigz) or use tar -I pigz so the compression step uses all cores. Install with apt-get install -y pigz or apk add pigz on Alpine. The .gz it writes is portable, so consumers do not need pigz.
Common errors in CI
"pigz: command not found" means the package is missing; install pigz (apt/apk/brew). "pigz: skipping: <file> does not exist" is a wrong path. Setting -p higher than the core count does not error but gives no extra speed; use $(nproc) to match the runner.