Elasticsearch waiting for green/yellow cluster health in CI

The Elasticsearch port accepts connections before the cluster has finished initializing shards and reached a serviceable state. Tests that start the moment the TCP port opens race the cluster and fail intermittently. The fix is to poll _cluster/health and block until it reports yellow or green.

What this error means

Early requests fail with 503 or "master_not_discovered_exception", and the failures are intermittent: the job passes when the cluster happens to be ready and fails when the test starts too early.

elasticsearch

curl: (52) Empty reply from server
{"error":{"type":"master_not_discovered_exception"},"status":503}

Common causes

The port opens before the cluster is ready

A container healthcheck on the raw TCP port passes as soon as the socket binds, well before shards allocate and a master is elected.

Tests connect on the first successful TCP handshake

Client code treats an open socket as "ready", but the cluster still returns 503 until it reaches yellow or green.

How to fix it

Block on cluster health with wait_for_status

Call the health API with wait_for_status=yellow and a timeout before any test runs.
The request only returns once the cluster reaches at least that status.
Run this as a dedicated setup step, not inside the tests.

Terminal

until curl -s "http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s" \
  | grep -qE '"status":"(yellow|green)"'; do
  echo "waiting for elasticsearch..."; sleep 2
done

Use a health-aware container healthcheck

Point the service healthcheck at the cluster health endpoint so the job only proceeds once the cluster is serviceable.

.github/workflows/ci.yml

options: >-
  --health-cmd "curl -sf http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=1s"
  --health-interval 10s --health-timeout 5s --health-retries 10

How to prevent it

Gate tests on _cluster/health?wait_for_status=yellow, not on an open port.
Set a generous health retry count so slow cluster startup does not fail the job.
Keep readiness polling in a setup step separate from the test run.