Self-Healing CI: Recovering a Transient 429 from an API in Tests
A test that fails because an API rate-limited it hit a quota, not a regression -- the same test passes once the limit resets or with a short backoff.
The problem
A test that calls a real API fails because the API returned a 429 (rate limited). The test logic and assertions are correct; the suite briefly exceeded the API’s request rate. A human re-runs the suite and it goes green with no code change.
AssertionError: expected 200 but got 429 Too Many Requests
# from the API the test exercisesWhy it happens
Tests that hit a real API share its rate limit. A burst of test requests -- or other suites and tenants calling the same API -- can briefly exceed the cap, so a valid test fails on a 429 that has nothing to do with the behavior under test.
The limit resets on a short window, so blindly retrying every failure is risky -- a genuine 4xx contract break must still surface -- but a 429 specifically has a transient signature that can be retried with backoff.
The manual fix
Manual handling for a rate-limited test:
- Re-run the suite after the rate window resets.
- Add retry-with-backoff around the API call for 429 responses specifically.
- Reduce request volume in tests, or stub/mock the API so the suite does not depend on its live quota.
How this gets automated
Self-healing CI treats a test 429 conservatively: it retries failures whose signature matches a transient rate limit while leaving consistent, reproducible assertion failures to surface as real. The aim is to remove rate-limit noise with backoff without ever masking a genuine contract or logic regression.