Retries and Idempotency in CI: Retry Safely Without Side Effects
Retrying a flaky step is only safe if running it twice does no harm. That property is idempotency - and knowing which steps have it is the difference between a clean recovery and a duplicated deploy.
Retries are the simplest cure for transient CI failures, but a blind retry can do real damage if the step had side effects. Idempotency is the property that makes retries safe.
What idempotency means
An operation is idempotent if running it once or many times produces the same end state. npm ci is idempotent - it converges to the same node_modules whether run once or three times. A step that posts a payment or appends a row is *not* - each run changes the world again.
Safe vs unsafe to retry
- Safe: dependency installs, checkouts, read-only tests, builds.
- Safe: deploys designed to be idempotent (declarative apply, upsert).
- Unsafe: appending records, sending notifications, incrementing counters.
- Unsafe: any step that creates a new external resource each run.
Making steps retry-safe
Design side-effecting steps to be idempotent: use upserts keyed on a stable id, declarative apply instead of imperative create, and idempotency keys for API calls so a retried request is deduplicated server-side. Then a retry re-converges instead of double-acting.
Retry the noise, not the signal
Limit automatic retries to transient, idempotent steps (network fetches, dependency installs, flaky-but-known tests) with a small bounded count. Never blanket-retry a whole pipeline that contains non-idempotent steps, and never retry a deterministic logic failure - that just hides a real bug.
Key takeaways
- Idempotent = running it multiple times yields the same end state.
- Installs, checkouts, and declarative deploys are safe to retry; appends and sends are not.
- Use upserts and idempotency keys to make side-effecting steps retry-safe.
- Bound retries to transient, idempotent steps - never mask a real failure.