Skip to content
Latchkey

DVC "experiment ... failed" (dvc exp) in CI

dvc exp run reproduces the pipeline inside an experiment workspace and it failed. The cause is usually the same as a repro failure, a stage command error, or a workspace that is not clean enough for DVC to create the experiment.

What this error means

dvc exp run stops with "ERROR: Failed to reproduce experiment" after a stage error or a message about an unexpected workspace state.

dvc
ERROR: Failed to reproduce experiment 'quick-lynx':
failed to reproduce 'train': failed to run: python train.py, exited with 1

Common causes

A pipeline stage failed inside the experiment

The experiment reruns the DAG; a stage command that errors fails the whole experiment.

Missing data or a dirty workspace

Inputs were not pulled, or uncommitted changes block DVC from creating a clean experiment commit.

How to fix it

Fix the failing stage, then re-run

  1. Read the stage error inside the experiment output.
  2. Pull inputs and install deps so the stage can run.
  3. Re-run dvc exp run.
Terminal
pip install -r requirements.txt
dvc pull
dvc exp run

Run experiments on a clean checkout

Ensure the CI checkout is clean so DVC can snapshot the experiment.

Terminal
git status --porcelain
dvc exp run

How to prevent it

  • Pull data and install deps before dvc exp run.
  • Keep the CI workspace clean so experiments can be recorded.
  • Reproduce the pipeline with dvc repro before running experiments.

Related guides

Tired of flaky CI? Latchkey auto-heals failed jobs and retries them for you. Start free →