Self-Healing CI: Recovering a Flapping Service Container Healthcheck
A healthcheck that flaps unhealthy then healthy during startup is a service still stabilizing, not a broken dependency -- waiting for it to settle clears it.
The problem
A job fails because a service container was marked unhealthy at the moment the dependent step started, even though its healthcheck flaps and settles healthy moments later. The service is fine; its healthcheck oscillated during startup stabilization. A human re-runs or waits for the healthcheck to settle and the step passes unchanged.
service container "db" is unhealthy (healthcheck failing)
# healthcheck oscillates unhealthy -> healthy during startup, then stays healthyWhy it happens
During startup a service can briefly fail its healthcheck -- still loading data, warming connections, or running migrations -- so the healthcheck oscillates between unhealthy and healthy before settling, and a step that samples it at the wrong instant sees unhealthy.
It is a startup-stabilization flap, not a broken service: the healthcheck settles healthy shortly after, so waiting for it to stabilize lets the same step succeed.
The manual fix
Manual mitigations for a flapping healthcheck:
- Add a start-period/grace window so early failures during startup do not count.
- Poll for a stable healthy state (several consecutive passes) before the dependent step.
- Re-run the job -- the healthcheck has usually settled by the retry.
How this gets automated
A flapping healthcheck during startup is distinguishable from a persistently failing one by whether it settles healthy shortly after. A self-healing CI pipeline detects the unhealthy-at-start condition, waits for the healthcheck to stabilize and re-checks, and only escalates if the service stays unhealthy, which is the real signal of a broken dependency rather than startup flapping.