gcloud resource_exhausted quota ci: rate quota or allocation

By Kaveh Alemi·Latchkey

A gcloud resource_exhausted quota ci failure is one of at least three different lines Google Cloud produces for the same underlying condition, and which one you get depends on the service and the transport rather than on what went wrong. The distinction that decides your fix is not in the status token at all: it is whether the quota you crossed resets on a timer or does not reset until somebody raises it.

What this error means

A gcloud step exits 1 and prints one line starting ERROR: , followed by the command path in brackets, followed by a status token, a colon, and a sentence from the service. The status token is the part people key on and it is the least stable part of the line. Google documents three different outcomes for exceeding a quota: an HTTP 429 for a REST request, an HTTP 403 for Compute Engine whether the call came over API, REST or gRPC, and a ResourceExhausted error over gRPC whose appearance "depends on the service". Its Compute Engine wording is more specific still: 403 QUOTA_EXCEEDED for an allocation quota and 403 RATE_LIMIT_EXCEEDED when the quota is a rate quota. So RESOURCE_EXHAUSTED in your log narrows which door the error came through, and tells you nothing yet about whether waiting will help.

ERROR: (gcloud.compute.instances.create) RESOURCE_EXHAUSTED: The zone 'us-east1-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.

The line is three layers deep

Nothing in gcloud holds this sentence as a literal. The ERROR: prefix is the CLI logger. The bracketed command path comes from _LogKnownError in googlecloudsdk/calliope/exceptions.py, which formats the message as the command path followed by the exception. The exception text comes from _MakeGenericMessage in googlecloudsdk/api_lib/util/exceptions.py, which joins a description and a message with a colon. And the sentence after the colon is the message field the service sent.

The description in front of the colon is worth one more step, because it has two sources and they can disagree. _ExtractResponseAndJsonContent sets it first from the HTTP response reason, which is the reason phrase, and only if that is absent does it fall back to the status field of the JSON error body. RESOURCE_EXHAUSTED is the status field, so seeing it tells you the fallback branch fired. A line that reads Too Many Requests: instead is the same failure reported through the other branch, not a different failure.

Two things about the block at the top follow from that. Its message slot holds the body Google publishes as its own example of a RESOURCE_EXHAUSTED response, which describes a zone without capacity rather than a project over quota, and this page keeps that distinction rather than quietly reading it as a quota message. And no claim is made here about what Google's quota service does internally, because that service is closed and the only honest sources are the values Google publishes and the bytes that arrive.

Common causes

A wide matrix calls one API faster than the rate quota allows

Twenty parallel jobs that each authenticate, list, and deploy generate a burst against one project in a few seconds. Rate quotas are usually expressed per minute, so a burst that would be fine spread over a minute trips the limit when it arrives in five seconds. The signature is that a few jobs in the matrix fail while the rest pass, and that re-running the failed ones alone succeeds.

Preview environments accumulate against an allocation quota

A pipeline that creates a service, a load balancer or an address per branch and relies on a cleanup job that sometimes does not run will walk into a ceiling weeks after the pipeline was written. In our experience this is the version that gets misdiagnosed most often, because the failing deploy is new while the cause is months of accumulation, and retrying the deploy is the first thing anyone tries.

The quota is charged to a project you did not intend

Client libraries and gcloud can bill quota to a quota project that differs from the project holding the resources, particularly under workload identity federation. When that project is small or shared, its limits are lower than the ones you sized for, so the failure names limits that look wrong for your workload. Checking which project the quota was charged to is faster than arguing about the number.

The region genuinely has no capacity, and the code is the same

Compute capacity is not the same thing as your quota, but it arrives through the same status token: Google's own published error example for RESOURCE_EXHAUSTED is a zone without enough resources to fulfil a request. No quota increase fixes that, and the message tells you the fix, which is a different zone. This is the case where reading the sentence rather than the token saves a support ticket.

How to fix it

Classify before you retry

1. Capture the full error line, not just the exit code. Exit 1 is all gcloud gives you, and Google documents it as the CLI outcome for every quota-exceeded case, so it carries no information about which one you hit.
2. Match the sentence for the words that name a ceiling and a region, versus the words that name a metric and an interval. Fail fast on the first, back off on the second.
3. Record which pipelines hit which class. A rate quota that trips every Monday morning and an allocation quota that trips once a quarter need different work.

gcloud run deploy api --image "$IMAGE" --region us-central1 --format=json 2>err.txt || {
  cat err.txt
  grep -q "QUOTA_EXCEEDED" err.txt && exit 1
  exit 75
}

Serialize the deploy rather than backing off inside it

A concurrency group is the cheapest fix for a rate quota, because it removes the burst instead of absorbing it. One deploy at a time per environment costs queueing rather than runner minutes, and unlike a retry ladder it cannot turn one slow failure into five.

concurrency:
  group: gcloud-deploy-${{ github.ref }}
  cancel-in-progress: false

Make cleanup a guaranteed step, not a happy-path step

Allocation quotas are filled by resources nobody deleted. A teardown that runs only when the previous steps passed will miss exactly the runs that created something and then failed, which is the population most likely to leak. Run it with if: always() and give it its own timeout so it is not cancelled with the job.

- name: Tear down preview environment
  if: always()
  timeout-minutes: 10
  run: gcloud run services delete "preview-${{ github.event.number }}" --region us-central1 --quiet || true

Confirm which project is being charged

Before asking for a quota increase, check that the limit you are hitting belongs to the project you think it does. A quota project set by the credential rather than by the command is a common surprise under federated identity, and a one-line check settles it at the start of the job instead of at the end of an escalation.

- run: |
    gcloud config list --format="value(core.project,billing.quota_project)"
    gcloud auth list --format="value(account)"

Rate quotas reset. Allocation quotas do not.

This is the distinction that decides whether a retry is a fix or a way to spend runner minutes. Google's quota documentation states that rate quotas reset after a predefined interval specific to each service. An allocation quota is a ceiling on how much of a thing your project holds at once, and nothing about waiting changes it: if the project is allowed thirty in-use addresses and holds thirty, it will still hold thirty in five minutes.

CI workloads generate both, and they tend to arrive together in the same pipeline. A matrix of twenty jobs each calling the same API in the same second is a rate problem. A pipeline that creates a preview environment per pull request and never tears the old ones down is an allocation problem. The first is fixed with backoff or with a concurrency group, and the second is fixed by deleting things or by asking for more.

The practical test is the sentence rather than the token. A rate message names a metric and an interval, often in the shape of a limit per minute for a consumer. An allocation message names a resource and a region and a number you recognise as a ceiling. If your retry wrapper cannot tell them apart, it will convert a five second failure into a fifteen minute one.

- name: Deploy
  run: |
    set -euo pipefail
    for attempt in 1 2 3 4 5; do
      if out=$(gcloud run deploy api --image "$IMAGE" --region us-central1 2>&1); then
        echo "$out"; exit 0
      fi
      echo "$out"
      case "$out" in
        *QUOTA_EXCEEDED*) echo "allocation quota, retrying will not help"; exit 1 ;;
      esac
      sleep $(( attempt * 15 ))
    done
    exit 1

Why this page carries no recorded run

Reproducing this on a runner means deliberately exhausting a real Google Cloud quota on a real project, which either costs money, degrades a project somebody depends on, or requires a project created to be broken. None of those produce a better answer than the documentation already gives, and the first two are irresponsible on infrastructure we do not own.

There is also a limit on what any single reproduction could prove. The sentence after the colon is written by the service, and there are hundreds of services, so a capture from one of them would be an example rather than a specification. The specification is what Google publishes about which status arrives for which kind of quota, and that is what this page quotes. The block at the top is assembled from the two gcloud functions that build the line plus a message body from Google's own documented example, and the label says so. content/heal-evidence.mjs has no record for this slug, so nothing here claims Latchkey repairs it.

How to prevent it

• Put a concurrency group on every job that deploys, so a matrix cannot burst against one rate quota.
• Run teardown with if: always() so failed runs cannot leak resources into an allocation quota.
• Log the full gcloud error line, since exit code 1 is the same for every quota class.
• Review which project your CI credential charges quota to when you set the credential up, not when it fails.

Frequently asked questions

Related guides

References

• Google Cloud: troubleshoot quota errors, and which status each surface returns
• Google API Design Guide: the error model, and a populated RESOURCE_EXHAUSTED response
• googleapis/googleapis: google/rpc/code.proto, where RESOURCE_EXHAUSTED is defined
• Google Cloud SDK source: HttpErrorPayload, which builds the text after the command path
• GitHub Actions documentation