lz4: Ultra-Fast Compression for Hot Caches

lz4 compresses files to .lz4 at very high speed with a modest ratio, which suits caches that are written and read constantly.

lz4 wins when compress and decompress speed dominate and size is secondary: scratch caches, intermediate build outputs, hot layers. It keeps the source by default.

What it does

lz4 compresses input with the LZ4 algorithm, appending .lz4 and keeping the source by default. Levels run 1 (fastest) to 12 (best, via -9 for high compression mode). unlz4 or lz4 -d reverses it. It is the fastest of the common compressors, but ratios are lower than gzip.

Common usage

Terminal

lz4 scratch.tar               # fast, keeps scratch.tar
lz4 -9 build.tar build.tar.lz4  # high-compression mode
lz4 -d cache.tar.lz4          # decompress
tar cf - obj/ | lz4 > obj.tar.lz4
lz4 --rm big.bin              # remove source after compress

Options

Flag	What it does
-1 .. -12	Level; -9 selects high-compression mode
-d / --decompress	Decompress a .lz4 file
-c / --stdout	Write to stdout
-k / --keep	Keep the input (default)
--rm	Remove the source after a successful operation
-f / --force	Overwrite existing output

In CI

Use lz4 for caches that are rebuilt and restored frequently within a job, where CPU time saved outweighs the larger size. For caches stored across runs, zstd usually gives a better size-to-restore-time tradeoff. Install with apt-get install -y lz4 (some distros ship the binary as lz4c).

Common errors in CI

"Unrecognized header : file cannot be decoded" means the input is not an lz4 frame, or it is the older legacy format; a mismatched extension is the usual cause. "command not found" means the liblz4-tool / lz4 package is missing. Note lz4 does not overwrite by default and errors if the output exists; add -f.