Skip to content
LatchkeyLatchkey home

kubectl taint nodes: Usage, Options & Common CI Errors

Repel pods from a node unless they explicitly tolerate it.

kubectl taint nodes marks a node so only pods with a matching toleration may schedule there. It dedicates nodes to specific workloads - and is a frequent cause of mysteriously Pending pods in a test cluster.

What it does

kubectl taint nodes NODE key=value:EFFECT adds a taint; the trailing key=value:EFFECT- (or key:EFFECT-) form removes it. EFFECT is NoSchedule (block new pods), PreferNoSchedule (soft preference), or NoExecute (also evict running pods that do not tolerate it). Pods need a matching tolerations entry to land on a tainted node.

Common usage

Terminal
kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule
kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule-      # remove it
kubectl taint nodes node-1 maintenance=true:NoExecute   # evict + block
kubectl taint nodes node-1 maintenance:NoExecute-       # clear by key

Common errors in CI

After tainting a node, ordinary pods land Pending with "node(s) had untolerated taint {dedicated: gpu}" - expected, but it breaks pipelines that assumed scheduling, so either taint a dedicated node or give the test pod a matching toleration. A NoExecute taint evicts non-tolerating pods immediately, which can knock out workloads you did not mean to move. "error: at least one taint update is required" means you passed no taint spec. Forgetting to remove a test taint leaves the node effectively cordoned for normal work; clean it up with the key=value:EFFECT- form.

Using this in CI

A runner has no kubeconfig, no cached context, and no interactive auth. Every kubectl invocation in CI needs the context supplied explicitly, and most confusing CI failures here are the command running against the wrong cluster or no cluster at all.

Terminal
# never rely on the ambient context on a runner
kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" get pods

# confirm what you are actually connected to before mutating anything
kubectl config current-context
kubectl cluster-info

# fail fast instead of hanging on an unreachable API server
kubectl --request-timeout=30s get nodes

Frequently asked questions

kubectl taint nodes: Usage, Options & Common CI Errors?
kubectl taint nodes marks a node so only pods with a matching toleration may schedule there. It dedicates nodes to specific workloads - and is a frequent cause of mysteriously Pending pods in a test cluster.
What it does?
kubectl taint nodes NODE key=value:EFFECT adds a taint; the trailing key=value:EFFECT- (or key:EFFECT-) form removes it. EFFECT is NoSchedule (block new pods), PreferNoSchedule (soft preference), or NoExecute (also evict running pods that do not tolerate it). Pods need a matching tolerations entry to land on a tainted node.
Common errors in CI?
After tainting a node, ordinary pods land Pending with "node(s) had untolerated taint {dedicated: gpu}" - expected, but it breaks pipelines that assumed scheduling, so either taint a dedicated node or give the test pod a matching toleration.

Related guides

References

Run this faster and cheaper on Latchkey managed runners - self-healing included. Start free → 30-day trial · No credit card