kubectl exec "error dialing backend" / WebSocket Upgrade Failed in CI

kubectl exec/attach/port-forward upgrades the HTTP connection to a streaming (WebSocket/SPDY) channel through the API server to the kubelet. A momentary network blip, a brief kubelet hiccup, or proxy congestion can fail that upgrade - and a retry usually succeeds.

What this error means

kubectl exec fails with error dialing backend or unable to upgrade connection. Re-running the same command shortly after often works, which points to a transient connectivity issue rather than a config error.

error: unable to upgrade connection: error dialing backend: dial tcp 10.0.1.5:10250:
i/o timeout
# or
error: unable to upgrade connection: pod does not exist

Common causes

Transient API-server↔kubelet connectivity blip

The connection upgrade tunnels to the kubelet on the node. A brief network/route hiccup or a momentary kubelet stall fails the dial, and it clears once connectivity recovers.

Proxy/load-balancer dropping the upgrade

An intermediary that does not cleanly pass WebSocket/SPDY upgrades, or is briefly overloaded, can intermittently fail the connection without a persistent cause.

How to fix it

Retry the exec with a bounded backoff

Because the failure is usually transient, a short bounded retry often clears it.

for i in 1 2 3; do
  kubectl exec deploy/api -- echo ok && break
  sleep $((i*3))
done

Rule out a non-transient cause

1. Confirm the pod is Running and the node is Ready (kubectl get pod -o wide, kubectl get nodes).
2. Check whether the node’s kubelet (port 10250) is reachable from the API server.
3. Verify any ingress proxy/LB in front of the API server forwards WebSocket upgrades.

How to prevent it

• Wrap CI exec/port-forward calls in a bounded retry to absorb transient blips.
• Keep nodes and kubelets healthy so the streaming backend is reachable.
• Ensure proxies in front of the API server pass WebSocket/SPDY upgrades.

Related guides