Self-Healing CI: Recovering a Kubernetes Rollout That Is Briefly Not Ready
A rollout that has not gone ready by the deadline is usually still converging, not failing -- giving it a moment longer lets the deploy succeed.
The problem
A deploy step fails because kubectl rollout status (or an equivalent readiness check) timed out before the new pods became ready. The manifests are valid and the image is fine; image pulls, readiness probes, or scheduling simply took longer than the deadline on this run. A human re-runs the deploy or extends the wait and the rollout completes unchanged.
error: deployment "api" exceeded its progress deadline
# or
Waiting for deployment rollout to finish: 2 of 3 updated replicas are available...Why it happens
A rollout becomes ready only after new pods are scheduled, images are pulled, and readiness probes pass. Slow image pulls, brief node pressure, or a probe that needs a few extra seconds can push readiness just past the deploy step’s deadline even though the rollout is healthy and converging.
The same rollout reaches ready shortly after, so a deadline crossed by a small margin is a transient timing miss, not a failed deployment.
The manual fix
Manual handling for a not-yet-ready rollout:
- Re-run the deploy or re-check rollout status -- it has usually converged.
- Increase the readiness/progress-deadline timeout to absorb normal variance.
- Tune readiness probes and resource requests so pods become ready faster and more predictably.
kubectl rollout status deploy/api --timeout=300sHow this gets automated
A "rollout not ready in time" failure is distinguishable from a real rollout failure by whether the deployment converges shortly after the deadline. A self-healing CI pipeline detects the readiness timeout, waits for the rollout to finish converging and re-checks, and only escalates if the rollout genuinely fails to progress, which is the real signal of a bad deploy rather than a slow one.