GKE Workload Identity "PermissionDenied" - Fix Pod GCP Auth in CI

GKE Workload Identity lets a pod authenticate to Google Cloud as a GCP service account by binding a Kubernetes ServiceAccount (KSA) to a GCP service account (GSA). If the binding or the GSA’s IAM roles are missing, the pod’s GCP calls are denied.

What this error means

A pod’s GCP SDK calls fail with PermissionDenied or unable to impersonate / 403, even though the cluster has Workload Identity enabled. It is deterministic for a KSA that is not correctly bound or under-privileged.

pod log

google.api_core.exceptions.PermissionDenied: 403 Permission 'storage.objects.get'
denied on resource ... (or it is not in your project)
# or: compute.googleapis.com ... unable to impersonate service account

Common causes

KSA not bound to a GSA

The Kubernetes ServiceAccount lacks the iam.gke.io/gcp-service-account annotation, or the GSA has no roles/iam.workloadIdentityUser binding for that KSA, so impersonation fails.

GSA missing the needed IAM roles

The binding works but the GSA does not have the GCP IAM roles (e.g. roles/storage.objectViewer) the workload needs, so the actual API call is denied.

How to fix it

Bind the KSA to the GSA

Annotate the KSA and grant the workloadIdentityUser role on the GSA.

Terminal

kubectl annotate serviceaccount api -n app \
  iam.gke.io/gcp-service-account=api@PROJECT.iam.gserviceaccount.com
gcloud iam service-accounts add-iam-policy-binding \
  api@PROJECT.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT.svc.id.goog[app/api]"

Grant the GSA the roles the workload needs

Add the least-privilege IAM roles for the GCP resources the pod accesses.
Confirm the pod uses the annotated KSA (serviceAccountName: api).
Verify from inside the pod: gcloud auth list should show the GSA.

How to prevent it

Manage the KSA↔GSA binding and IAM roles in IaC, not by hand.
Grant the GSA least-privilege roles for exactly what the workload needs.
Confirm pods reference the annotated KSA, not the default ServiceAccount.