Self-Healing CI: Recovering a Transient SSH Connection Drop

An SSH session that drops mid-command is a network blip, not a rejected login -- reconnecting and re-running the command clears it.

The problem

A step that runs commands over SSH (a deploy, a remote build, an rsync) fails with a dropped connection -- broken pipe, Connection reset by peer, or Connection closed by remote host. The host, key, and command are all valid; the session was cut by a transient network problem. A human re-runs the job and the SSH step completes unchanged.

Typical symptom

client_loop: send disconnect: Broken pipe
# or
Connection closed by remote host
kex_exchange_identification: Connection reset by peer

Why it happens

An SSH session holds a single long-lived connection, and a brief network interruption, an idle-timeout, or upstream congestion can sever it mid-command even though authentication succeeded and the remote host is healthy.

It is transport flakiness, not an auth or host problem: the key still works and the host is up, so reconnecting and re-running the command succeeds.

The manual fix

Manual mitigations for SSH drops:

Re-run the job to re-establish the connection.
Enable SSH keepalives (ServerAliveInterval) so idle sessions are not dropped.
Wrap the SSH command in a bounded retry, and make remote operations idempotent.

Manual retry

ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=4 host './deploy.sh'   || (sleep 5 && ssh -o ServerAliveInterval=15 host './deploy.sh')

How this gets automated

A dropped SSH connection has a recognizable transient signature -- a reset or closed session rather than an auth rejection -- and the safe response is to reconnect and retry. A self-healing CI pipeline detects the connection drop, retries with backoff, and only escalates if the host is genuinely unreachable or authentication fails, so a flaky session does not fail the build.