What Is Experiment Tracking? Logging ML Runs Explained
Experiment tracking is the practice of recording the inputs and results of every ML training run so experiments are comparable, searchable, and reproducible.
Machine learning is empirical: you try dozens of configurations and keep the best. Without a system, those runs vanish into scrollback and forgotten file names. Experiment tracking captures every run - its parameters, code, data, and metrics - so you can compare and reproduce them.
What experiment tracking is
Experiment tracking logs the details of each training run to a central store: the hyperparameters used, the code version, the dataset version, the resulting metrics, and often artifacts like the model and plots. Each run becomes a queryable record rather than a transient result.
Why it matters
Tracking makes experiments comparable ("which learning rate gave the best F1?"), reproducible (you know exactly what produced a result), and collaborative (the whole team sees the run history). It is the foundation that model registries and continuous training build on.
Tools
MLflow, Weights and Biases, and Neptune are common trackers. They provide a logging API you call from training code and a UI to browse, filter, and compare runs, plus integration with model registries for promoting the best run.
Experiment tracking in CI
When training runs in CI, log each run to the tracker tagged with the commit and dataset version, so every automated run is recorded and comparable.
import mlflow
mlflow.set_tag("git_sha", "${GITHUB_SHA}")
mlflow.log_params(params)
mlflow.log_metric("f1", score)Latchkey note
Tracked CI runs upload artifacts and metrics to the tracking server. On Latchkey, caching the training data and dependencies keeps each tracked run fast, and auto-retry covers a transient blip when the run uploads its artifacts to the tracker.
Key takeaways
- Experiment tracking records the parameters, code, data, and metrics of every training run.
- It makes experiments comparable, reproducible, and visible to the whole team.
- Logging CI training runs to a tracker, tagged with commit and data version, keeps automated runs recorded and comparable.