Self-Healing CI: Recovering a DNS Rebind During Deploy
A deploy that fails because a hostname briefly resolved to a stale or in-flux address hit a DNS-propagation blip, not a wrong target -- re-resolving and retrying clears it.
The problem
A deploy step fails because a hostname resolved to a stale or changing address during a cutover -- a record was being updated, or a cached entry pointed at an endpoint that was draining. The intended target is correct; DNS was briefly in flux. A human re-runs after propagation and the deploy reaches the right endpoint unchanged.
dial tcp: lookup api.example.com: no such host
connect to old endpoint failed (record was being updated mid-deploy)Why it happens
During a deploy that changes a DNS record, resolvers and caches converge on the new value over a short window, so a lookup in that window can return a stale address or briefly fail even though the intended target is correct.
It is a propagation/caching blip, not a wrong target: once DNS settles, re-resolving the name reaches the correct endpoint, so a retry succeeds.
The manual fix
Manual mitigations for a DNS rebind during deploy:
- Re-run the step after a short wait so DNS converges.
- Use low TTLs during cutovers and add retry-with-re-resolution around the affected calls.
- Drain old endpoints gracefully so in-flight traffic to the old address still succeeds.
How this gets automated
A DNS rebind during deploy has a recognizable transient signature -- a resolution failure or a stale address during a cutover rather than a permanently wrong target -- and the safe response is to re-resolve and retry. A self-healing CI pipeline detects the resolution/connect failure, retries after DNS converges, and only escalates if the name stays unresolvable or wrong, distinguishing a propagation blip from a real misconfiguration.