Elixir "mix test" failures / non-zero exit in CI

mix test exits non-zero when any test fails, and the job fails with it. When tests pass locally but fail in CI, the difference is usually shared state, async DB coupling, or missing environment setup.

What this error means

The test step ends with "N tests, M failures" and exit code 1. Failures may be flaky (order-dependent) or consistent (missing env/DB).

mix test

  1) test creates a user (MyApp.AccountsTest)
     Assertion with == failed
     code:  assert user.role == "admin"
     left:  "member"
     right: "admin"

Finished in 2.1 seconds
12 tests, 1 failure

Common causes

Async tests coupling through shared state

Tests marked async: true that share global or DB state pass alone but interfere under CI parallelism, causing order-dependent failures.

Missing environment or unmigrated DB in CI

An env var or a migrated database present locally is absent in CI, so a subset of tests fail there.

How to fix it

Isolate DB tests with the Ecto Sandbox

Use Ecto.Adapters.SQL.Sandbox so each async test gets an isolated connection.
Only mark tests async: true when they truly share no mutable global state.
Migrate the test DB in CI before running the suite.

test/support/data_case.ex

setup do
  :ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
end

Reproduce the CI ordering to find flakes

Run with a fixed seed (or the seed from the failing run) to reproduce order-dependent failures.

Terminal

mix test --seed 0
mix test --seed 123456   # the seed printed by the failing CI run

How to prevent it

Isolate DB-backed tests with the SQL Sandbox.
Only use async: true for tests with no shared mutable state.
Provide the same env and migrated DB in CI as locally.