Skip to content
Latchkey

Kubernetes DaemonSet Not Running on All Nodes - Fix DESIRED < Nodes in CI

A DaemonSet should run one pod per eligible node. When DESIRED is lower than your node count, or its pods sit Pending, the nodes are being excluded by taints the pods do not tolerate, a nodeSelector/affinity, or a lack of resources.

What this error means

kubectl get daemonset shows DESIRED/CURRENT below the number of nodes, or DaemonSet pods are Pending on some nodes. The agent (logging, CNI, monitoring) is missing exactly where those nodes are excluded.

kubectl output
$ kubectl get ds -n logging
NAME       DESIRED   CURRENT   READY   NODE SELECTOR   AGE
fluentbit  3         3         3       <none>          1d
# but the cluster has 5 nodes → 2 nodes are excluded

Common causes

Untolerated node taints

Tainted nodes (control-plane, dedicated pools, condition taints) are excluded unless the DaemonSet pod tolerates them. DESIRED drops to only the nodes whose taints are tolerated.

nodeSelector / affinity excludes nodes

A nodeSelector or nodeAffinity on the DaemonSet limits it to labeled nodes; unlabeled nodes are not counted in DESIRED.

Insufficient resources on some nodes

If the DaemonSet pod’s requests do not fit a node’s free capacity, its pod stays Pending there even though DESIRED includes it.

How to fix it

Compare DaemonSet placement to node taints/labels

Terminal
kubectl get ds <ds> -n <ns> -o jsonpath='{.spec.template.spec.tolerations}'; echo
kubectl get nodes -o json | jq '.items[] | {name:.metadata.name, taints:.spec.taints}'
kubectl get pods -n <ns> -l <ds-selector> -o wide

Add tolerations/labels or free resources

  1. Add tolerations for the taints on nodes the agent must cover (control-plane agents often tolerate the control-plane taint).
  2. Adjust or remove a nodeSelector/affinity that excludes intended nodes (or label those nodes).
  3. Right-size the DaemonSet pod’s requests so it fits on every target node.

How to prevent it

  • Give cluster-wide DaemonSets the tolerations to cover tainted nodes you intend them to run on.
  • Keep node labels and DaemonSet selectors aligned as pools change.
  • Keep DaemonSet resource requests small enough to fit every node.

Related guides

Tired of flaky CI? Latchkey auto-heals failed jobs and retries them for you. Start free →