Self-Healing CI: Recovering a Load Balancer Warmup 502
A 502 from the load balancer while new targets are still registering is a warmup readiness blip, not a broken deploy -- once targets are healthy, the same request succeeds.
The problem
A post-deploy smoke check fails with a 502 Bad Gateway because the load balancer had no healthy targets yet -- new instances were still registering and passing health checks. The deploy is fine; the check raced target warmup. A human re-runs after warmup and the endpoint returns 200 unchanged.
HTTP/1.1 502 Bad Gateway
# load balancer has no healthy targets yet; new targets still registering / health-checkingWhy it happens
After a deploy, new targets must register with the load balancer and pass health checks before they receive traffic, and during that brief window the LB can have no healthy target and return a 502, even though the new instances are coming up fine.
It is a warmup/readiness race, not a broken deploy: once targets pass their health checks, the same request succeeds, so a retry after warmup returns 200.
The manual fix
Manual mitigations for a warmup 502:
- Re-run the smoke check after the targets are healthy, or poll for a healthy target group first.
- Wait for the deployment’s health checks to pass before asserting on the endpoint.
- Tune health-check thresholds so targets become healthy promptly and predictably.
How this gets automated
A warmup 502 is distinguishable from a real outage by whether the endpoint recovers once targets become healthy. A self-healing CI pipeline detects the 502-during-warmup condition, waits for healthy targets and re-checks, and only escalates if the endpoint stays unhealthy, which is the real signal of a broken deploy rather than a slow target registration.