Self-Healing CI: Recovering a Database Failover During Migration
A migration that drops its connection because the database failed over is an infrastructure event, not a bad migration -- reconnecting to the new primary and retrying an idempotent migration completes it.
The problem
A migration step fails because the managed database failed over to a new primary mid-run, dropping the connection or briefly going read-only. The migration itself is valid; a failover event interrupted it. A human re-runs after the new primary is promoted and an idempotent migration completes unchanged.
server closed the connection unexpectedly (failover in progress)
ERROR: cannot execute ... in a read-only transaction (replica promoted)Why it happens
Managed databases fail over to a standby for maintenance or recovery, briefly dropping connections and promoting a replica, so a migration running across that window can lose its connection or hit a momentary read-only state even though the migration is correct.
It is a transient infrastructure event, not a bad migration: once the new primary is promoted, reconnecting and re-running an idempotent migration completes the same work.
The manual fix
Manual mitigations for a failover during migration:
- Re-run the migration once the new primary is promoted -- ensure migrations are idempotent/resumable.
- Use a connection string that follows the writer endpoint and reconnects on failover.
- Wrap migration runs in retry-with-backoff that reconnects before retrying.
How this gets automated
A failover during migration has a recognizable transient signature -- a dropped connection or a momentary read-only state during a promotion, not a SQL/logic error in the migration -- and the safe response is to reconnect to the new primary and retry an idempotent migration. A self-healing CI pipeline detects the failover-driven failure, retries after reconnecting, and only escalates if the migration itself genuinely fails, distinguishing an infrastructure event from a bad migration.