Self-Healing CI: Recovering When a Database Service Container Is Slow to Accept Connections

When tests start before the database container finishes booting, the failure is a timing race -- a short readiness wait clears it with no code change.

The problem

A job that uses a database service container fails with connection refused because the test step started before the database finished initializing and opened its port. The credentials and config are correct; the job simply raced the container’s startup. A human adds a readiness wait or re-runs and it connects cleanly.

psql: error: connection to server at "localhost", port 5432 failed: Connection refused
# or
Error: connect ECONNREFUSED 127.0.0.1:3306

Why it happens

A database container reports "started" when its process launches, but it needs additional time to initialize storage, run migrations, and begin listening. A test that connects in that window gets a connection refused even though the service comes up moments later.

It is a pure startup race, not a misconfiguration: the same connection succeeds once the database is actually accepting connections, with no change to the job.

The manual fix

The manual fix is to wait for readiness before the first query:

1. Add a healthcheck/readiness loop that polls the database until it accepts connections.
2. Use the service’s wait-for-ready tooling (e.g. pg_isready, mysqladmin ping) before running tests.
3. Re-run the job -- by the retry the container is usually ready.

for i in $(seq 1 30); do pg_isready -h localhost -p 5432 && break; sleep 1; done

How this gets automated

A "service not ready" failure at job start has a clear readiness-race signature, and the safe response is to wait for the dependency to come up and retry. A self-healing CI pipeline detects the connection-refused-at-startup condition, waits for the service to become ready, retries the step, and only escalates if the service never comes up, which is the real signal of a broken container rather than a slow start.

Related guides