Self-Healing CI: Auto-Retrying a gcloud Quota Exceeded Error
Most gcloud quota errors in CI are rate quotas measured per minute, not allocation quotas: the same command succeeds shortly afterwards without anything being raised.
The problem
A gcloud command fails during a deploy or provisioning step reporting a quota. The project has capacity and the command is correct. Google enforces both rate quotas, which refill continuously, and allocation quotas, which do not, and CI overwhelmingly hits the first kind by issuing many API calls in a short burst.
ERROR: (gcloud.compute.instances.create) Quota exceeded for quota metric 'CPUs' and limit 'CPUS-per-project-region'Why it happens
Pipelines call cloud APIs in tight bursts rather than at a human pace, so they reach per-minute rate quotas that interactive use never approaches.
The two quota kinds are reported in similar language, so a refilling rate quota reads exactly like a hard capacity limit that needs a support request.
The manual fix
Manual mitigations for a gcloud quota error:
- Re-run the step after a short wait, which is sufficient for a rate quota.
- Check the quota page to see whether the limit is a rate or an allocation before requesting an increase.
- Serialize or throttle bursts of API calls so a single job does not consume the per-minute budget.
gcloud compute project-info describe --format="value(quotas)"How this gets automated
The value of automating this one is diagnostic as much as it is recovery. A rate quota and an allocation quota look alike in the log, and the usual human response to either is to open a quota increase request that is not needed. A self-healing pipeline retries with a long backoff first: if the command then succeeds, the answer was a rate quota and nobody had to investigate. If it keeps failing, the failure is real and reaches you having already been triaged.