What Is a Non-Deterministic Test? Explained
A non-deterministic test can return different results on identical inputs because it depends on something uncontrolled, such as time, randomness, ordering, or external state.
Determinism is the property that the same inputs always produce the same output. A test that lacks it is unreliable by construction: even when the code is correct, the test result can vary. Non-determinism is the underlying mechanism behind most flaky tests.
Determinism, defined
A deterministic test, given the same code and inputs, always produces the same result. A non-deterministic test does not, because its outcome is influenced by a factor it does not control. That hidden dependency is the root of the flakiness.
Sources of non-determinism
- Real wall-clock time, time zones, or dates that change between runs.
- Unseeded randomness or hash ordering that varies per run.
- Concurrency and test ordering that interleave differently each time.
- Real network, filesystem, or third-party services that return varying data.
Why it is worse than a plain bug
A deterministic bug fails consistently and is straightforward to reproduce and fix. A non-deterministic test fails only sometimes, so it is hard to reproduce, easy to dismiss as "just flaky", and erodes confidence in the whole suite. The intermittency is the problem.
Making tests deterministic
Inject and freeze time. Seed randomness explicitly. Stub external services. Isolate state and avoid shared globals. Wait on real conditions instead of fixed sleeps. Each change removes one uncontrolled input until the test produces the same result every time.
The Latchkey angle
Non-determinism in your test is a code-side problem that retries cannot truly fix. Latchkey self-healing managed runners instead remove the environmental non-determinism around your tests by retrying transient and mechanical failures, so a one-off blip does not fail your build and the remaining variation points clearly at the test.
Key takeaways
- A non-deterministic test gives different results on identical inputs.
- Time, randomness, ordering, and external state are the usual culprits.
- Its intermittency makes it harder to fix than a consistent bug.
- Control each uncontrolled input to restore determinism.