# Self-Healing CI: Auto-Retrying a gcloud Quota Exceeded Error

> RESOURCE_EXHAUSTED and "Quota exceeded for quota metric" from gcloud are usually a rate quota that refills, not a hard capacity wall. See the manual fix and how self-healing CI retries.

Source: https://latchkey.dev/learn/self-healing-ci/self-healing-gcloud-quota-exceeded  
Updated: 2026-09-02

Most gcloud quota errors in CI are rate quotas measured per minute, not allocation quotas: the same command succeeds shortly afterwards without anything being raised.

## What makes a failure safely retryable

Automatic retry is only correct for failures that are genuinely transient. Retrying a deterministic failure wastes minutes and hides a real defect, so the classification matters more than the retry mechanism.

- Safe to retry: network timeouts, registry 5xx, transient DNS failures, a service container that was not ready, a spot instance reclaimed mid-run.
- Not safe to retry: assertion failures, compile errors, lint violations, anything that fails identically on every attempt.
- Ambiguous, and worth investigating rather than retrying: out-of-memory kills, disk exhaustion, and flaky tests. These repeat under load and a retry only hides the trend.
- Always record that a retry happened. A pipeline that silently retries is a pipeline whose real failure rate you do not know.

## FAQ

### How do I tell a rate quota from an allocation quota?

A rate quota refills on a clock and clears on a retry a minute later. An allocation quota is a ceiling on resources in existence and will keep failing until something is released or the limit is raised.

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt