ccache + distcc not caching or distributing in CI
ccache and distcc must be layered so ccache checks the cache first and only distributes real misses. The correct wiring is ccache in front with CCACHE_PREFIX=distcc; get it wrong and you lose caching, distribution, or both.
What this error means
Builds are slow because distcc runs every compile (no cache hits), or distcc never contacts hosts and everything compiles locally. ccache -s and distcc --show-hosts disagree with expectations.
# wrong: distcc wraps ccache, so distcc runs before the cache is consulted
export CXX="distcc ccache g++" # caches nothing useful
# right: ccache in front, distcc as the prefix for misses
export CXX="ccache g++"
export CCACHE_PREFIX="distcc"Common causes
The wrapper order is inverted
Putting distcc in front of ccache sends every compile to distcc before the cache is checked, so cacheable hits are lost.
CCACHE_PREFIX is not set to distcc
Without CCACHE_PREFIX=distcc, ccache runs the compiler locally on a miss, so distribution never happens.
How to fix it
Layer ccache in front with CCACHE_PREFIX
- Use ccache as the compiler launcher so the cache is consulted first.
- Set CCACHE_PREFIX=distcc so only real misses are distributed.
- Confirm hosts with
distcc --show-hostsand hits withccache -s.
export CC="ccache gcc" CXX="ccache g++"
export CCACHE_PREFIX="distcc"
export DISTCC_HOSTS="localhost/4 10.0.0.5/8"Verify both layers report activity
Check that ccache shows hits and distcc shows distributed jobs, so you know both layers engage.
ccache -s | grep 'Cache hit rate'
distccmon-text 1How to prevent it
- Always put ccache in front and distribute misses via CCACHE_PREFIX=distcc.
- Verify hosts and hit rate after wiring the two tools.
- Keep DISTCC_HOSTS in sync with reachable build hosts.