GitLab Runner Kubernetes Executor - Pod Pending / ImagePullBackOff / OOMKilled

By Kaveh Alemi·Latchkey

With the Kubernetes executor, each job runs in a build pod. Jobs fail when that pod cannot be scheduled, cannot pull its image, or is OOMKilled - all cluster-side problems, not your script.

What this error means

The job fails in preparation or mid-run with a Kubernetes pod error surfaced in the log: the pod stays Pending (unschedulable), hits ImagePullBackOff, or is OOMKilled. Your script may never start.

Job log

ERROR: Job failed (system failure): prepare environment:
waiting for pod running: pod status is Pending
(0/3 nodes are available: 3 Insufficient memory)

Diagnose it: which rule matched, and on which runner?

GitLab evaluates rules: top to bottom and the first match wins, including one that sets when: never. A job that does not run, or runs when you did not expect it to, is nearly always matching an earlier rule than the one you are reading.

.gitlab-ci.yml

# validate the definition against the project
curl -s --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/ci/lint" \
  --data-urlencode "content=$(cat .gitlab-ci.yml)"

# what the job actually sees
script:
  - env | grep -E "^CI_(PIPELINE_SOURCE|COMMIT_REF_NAME|RUNNER)" | sort

Common causes

Pod cannot be scheduled

Insufficient CPU/memory on nodes, taints without tolerations, or a missing node selector leave the build pod Pending until capacity appears.

Image pull fails in the cluster

The build or helper image cannot be pulled - wrong registry, missing imagePullSecrets, or a transient registry outage - yielding ImagePullBackOff.

Container OOMKilled

Memory limits set too low in the runner’s Kubernetes config cause the build container to be OOMKilled when the job exceeds them.

How to fix it

Set sensible pod resources and scheduling

Configure requests/limits and any node selectors/tolerations in the runner’s Kubernetes executor config.

config.toml

[runners.kubernetes]
  image = "alpine:3.20"
  cpu_request = "500m"
  memory_request = "512Mi"
  memory_limit = "2Gi"
  [runners.kubernetes.node_selector]
    "kubernetes.io/arch" = "amd64"

Diagnose with kubectl

Find the build pod and run kubectl describe pod <pod> to see scheduling or pull events.
For ImagePullBackOff, fix the image reference or add imagePullSecrets for the registry.
A transient unschedulable/pull blip on an autoscaling cluster clears once a node or the registry recovers - retry the job.

How to prevent it

Right-size pod requests/limits to node capacity and real job usage.
Configure imagePullSecrets for private images used by jobs.
Use cluster autoscaling and retry: runner_system_failure for transient capacity blips.

Frequently asked questions

What causes GitLab runner Kubernetes executor?

There are 3 common causes: pod cannot be scheduled, image pull fails in the cluster, and container oomkilled. Insufficient CPU/memory on nodes, taints without tolerations, or a missing node selector leave the build pod Pending until capacity appears.

How do I fix GitLab runner Kubernetes executor?

There are 2 fixes depending on which cause you have: set sensible pod resources and scheduling and diagnose with kubectl. Work through them in order, since the first is the most common.

What does GitLab runner Kubernetes executor actually mean?

The job fails in preparation or mid-run with a Kubernetes pod error surfaced in the log: the pod stays Pending (unschedulable), hits ImagePullBackOff, or is OOMKilled.

How do I stop GitLab runner Kubernetes executor happening again?

Right-size pod requests/limits to node capacity and real job usage. The prevention section lists 3 changes that keep it from recurring.

Can Latchkey fix this automatically?

Yes. Latchkey runs your GitHub Actions on managed runners that detect this failure, apply the fix, and retry the job automatically - self-healing is on by default.