Self-Healing CI: Recovering a Message Queue Connection Drop
A step that loses its connection to a message broker hit a transient transport blip or a broker rebalance, not a permanent outage -- reconnecting and retrying clears it.
The problem
A step that publishes to or consumes from a message queue (AMQP, Kafka, etc.) fails because the broker connection dropped -- a reset, a heartbeat miss, or a brief rebalance. The broker and credentials are fine; the connection was severed transiently. A human re-runs and the step reconnects and completes unchanged.
Connection reset by peer (broker heartbeat missed)
kafka: client has run out of available brokers to talk to (rebalance in progress)Why it happens
A message-queue client holds a long-lived connection to the broker, and a brief network interruption, a heartbeat miss, or a broker rebalance/restart can sever it mid-operation even though the broker is healthy and the credentials are valid.
It is transport flakiness, not a broker outage: the broker is up and the same publish/consume succeeds once the client reconnects and retries.
The manual fix
Manual mitigations for a queue connection drop:
- Re-run the step to re-establish the connection.
- Enable automatic reconnection and retry-with-backoff in the client.
- Make publishes idempotent and consumers resumable so a reconnect is safe.
How this gets automated
A dropped broker connection has a recognizable transient signature -- a reset or rebalance rather than an auth or routing error -- and the safe response is to reconnect and retry. A self-healing CI pipeline detects the connection drop, retries with backoff after reconnecting, and only escalates if the broker is genuinely unreachable, distinguishing a momentary drop from a real broker outage.