pytest/Hypothesis "Flaky" - Inconsistent Test Behavior in CI

Hypothesis found an example that failed, replayed it to confirm, and got a different result - so it raises Flaky. The test depends on hidden state, ordering, or nondeterminism rather than only its generated inputs.

What this error means

A property-based test fails with Flaky: Hypothesis ... produced unreliable results (e.g. "Falsified on the first call but did not on a subsequent one"). The same example does not reproduce deterministically.

hypothesis.errors.Flaky: Hypothesis test_roundtrip(value=0) produced
unreliable results: Falsified on the first call but did not on a subsequent one

Common causes

Hidden state between examples

Module/global state, a shared cache, or a fixture mutated by the test leaks across Hypothesis’ many invocations, so the same input behaves differently.

Nondeterminism inside the test

Unseeded randomness, real time/clock use, ordering of sets/dicts, or external I/O makes the outcome depend on more than the generated example.

How to fix it

Remove hidden state and nondeterminism

1. Reset or avoid global/shared state between examples; build fresh objects inside the test.
2. Seed any randomness and avoid wall-clock/time-dependent assertions.
3. Isolate external I/O behind deterministic fakes.

Make the property self-contained

A property test should depend only on its generated inputs. Move setup inside the test so each example starts clean.

@given(st.integers())
def test_roundtrip(value):
    obj = make_fresh()        # no shared state across examples
    assert decode(encode(obj, value)) == value

How to prevent it

• Keep property tests pure functions of their generated inputs.
• Avoid global state, real clocks, and unseeded randomness in tests.
• Build fresh fixtures per example rather than sharing across the run.

Related guides