nbval cell output mismatch (output differs) in CI

nbval is a pytest plugin that re-runs each notebook cell and compares the new output against the output saved in the notebook. When they differ it fails the test with a "Mismatch" report showing the reference output versus the freshly executed output.

What this error means

pytest --nbval reports failures like "Notebook cell execution failed" / "Mismatch" with a diff of "saved output" against "executed output" for a specific cell.

_______ test notebook.ipynb::Cell 3 _______
Mismatch in cell 3 output:
  reference output:  0.142857
  executed output:   0.1428571428571428

Common causes

Non-deterministic or environment-dependent output

Timestamps, random values, floating-point formatting, or object memory addresses differ between when the notebook was saved and when CI re-runs it.

Stale saved output after a code change

The code changed but the committed cell outputs were not regenerated, so the saved reference no longer matches.

How to fix it

Use nbval-lenient or sanitize volatile output

Run with --nbval-lenient to ignore unimportant differences, or add a sanitize file to normalize timestamps and random values.

pytest --nbval-lenient notebook.ipynb
# or, with a sanitize config:
pytest --nbval --sanitize-with sanitize.cfg notebook.ipynb

Regenerate and commit current outputs

1. Re-execute the notebook to refresh saved outputs.
2. Commit the updated .ipynb so the reference matches the code.
3. Tag inherently volatile cells to skip nbval checks.

jupyter nbconvert --to notebook --execute --inplace notebook.ipynb

How to prevent it

• Sanitize timestamps and random output, or seed RNGs in notebooks under nbval.
• Regenerate committed outputs whenever the code changes.
• Use --nbval-lenient when exact output equality is not required.

Related guides