Self-Healing CI: Auto-Retrying a Submodule Fetch Timeout
A submodule that fails to fetch is a network failure on one extra clone -- the same submodule update usually completes on a clean retry.
The problem
A git submodule update --init fails because one submodule’s fetch timed out or dropped its connection. The submodule, its ref, and credentials are all valid; the extra clone hit a transient transport problem. A human re-runs the job and the submodules initialize with no change.
fatal: clone of 'https://...' into submodule path '...' failed
fatal: unable to access '...': Failed to connect ... Operation timed outWhy it happens
Initializing submodules triggers a separate clone per submodule, each over its own connection, so the more submodules a repo has, the more independent transfers can hit a brief network blip and fail.
It is transport flakiness, not a repository problem: nothing about the submodule changed, and the same fetch completes once the connection is retried.
The manual fix
Manual mitigations for submodule fetch flakiness:
- Re-run the job to retry the submodule fetch.
- Use shallow submodules (
--depth) or parallel-but-bounded fetching to shrink the transfer window. - Wrap
git submodule updatein a bounded retry loop.
git submodule update --init --recursive --depth=1 || (sleep 5 && git submodule update --init --recursive --depth=1)How this gets automated
A failed submodule fetch has the same recognizable transient signature as any dropped clone, and the safe response is to retry the transfer. A self-healing CI pipeline detects the submodule transport failure, retries with backoff, and only surfaces the step if the submodule is genuinely unreachable or its credentials are wrong, so a flaky fetch never fails the build.