DVC "experiment ... failed" (dvc exp) in CI
dvc exp run reproduces the pipeline inside an experiment workspace and it failed. The cause is usually the same as a repro failure, a stage command error, or a workspace that is not clean enough for DVC to create the experiment.
What this error means
dvc exp run stops with "ERROR: Failed to reproduce experiment" after a stage error or a message about an unexpected workspace state.
dvc
ERROR: Failed to reproduce experiment 'quick-lynx':
failed to reproduce 'train': failed to run: python train.py, exited with 1Common causes
A pipeline stage failed inside the experiment
The experiment reruns the DAG; a stage command that errors fails the whole experiment.
Missing data or a dirty workspace
Inputs were not pulled, or uncommitted changes block DVC from creating a clean experiment commit.
How to fix it
Fix the failing stage, then re-run
- Read the stage error inside the experiment output.
- Pull inputs and install deps so the stage can run.
- Re-run
dvc exp run.
Terminal
pip install -r requirements.txt
dvc pull
dvc exp runRun experiments on a clean checkout
Ensure the CI checkout is clean so DVC can snapshot the experiment.
Terminal
git status --porcelain
dvc exp runHow to prevent it
- Pull data and install deps before
dvc exp run. - Keep the CI workspace clean so experiments can be recorded.
- Reproduce the pipeline with
dvc reprobefore running experiments.
Related guides
DVC "dvc repro ... failed" pipeline stage in CIFix DVC "ERROR: failed to reproduce ... stage" in CI - a dvc repro pipeline stage command exited non-zero. Th…
DVC "dvc.lock ... does not match" in CIFix DVC "dvc.lock does not match" / stages out of date in CI - the committed dvc.lock is stale relative to dv…
DVC Studio token (DVC_STUDIO_TOKEN) missing or invalid in CIFix DVC Studio integration failures in CI - sharing experiments or live metrics to DVC Studio needs a valid D…