Skip to content
Latchkey home
Documentation menu

Pipeline performance

Workflow health, build durations, failures and recovery (MTTR, rebuild success rate), and where failures concentrate.

In short

  1. Workflow health, durations, failures, and recovery metrics (MTTR, rebuild success rate) in one place.
  2. The page counts Latchkey CI runs rather than listing them: each run, and Healed beside a pass self-heal repaired, is on the Runs page.
  3. Trends catch slow creep; Most failed repositories and Most failed workflows find the active fires.
app.latchkey.dev / acme / performance

The Performance page for the last 30 days. Performance overview: workflow health 85% successful, average build duration 4m 12s, 254 successful and 46 failed builds, with 21 cancelled. Most failed repositories: api, web, worker; most failed workflows: build, release, e2e. Recovery and comparison: build fail rate 15.3%, mean time to recovery 3m 2s from failure to the next passed run, rebuild success rate 68%. A build in acme/api fails: failed builds go to 47, health to 84%, the fail rate to 15.6%, and api and build each gain one. The next build passes: successful builds go to 255 of 323 runs, and mean time to recovery becomes 3m 4s.

FIG. 2.4-AWatch one failed build move workflow health, the failed count, the fail rate and both Most failed lists together; the next green build ends the recovery that mean time to recovery averages. Health and fail rate count only runs that passed or failed, so the 21 cancelled runs leave them alone.

The Performance page tracks how healthy and how fast your workflows are. It counts only Latchkey CI runs; Latchkey on GH jobs and CLI jobs are listed on the Runs page. When no Latchkey CI run concluded in the selected dates and filters, the page reads "No runs in this window", with View runs to open the Runs page.

Runners, in the filter row (in Easy, under Filters), narrows the page to the runs that had a job on the sizes chosen. A run counts under a runner size when any of its jobs had that size. A run that created no job is in no size, so a pick of runner sizes leaves it out.

There are two ways to use it. As a monitor, you glance at the overview and trends to answer "is CI getting better or worse?" As an investigation tool, you filter down to one repository or workflow and use the failure lists and recovery metrics to answer "what exactly happened, and how did we handle it?" The sections below are laid out roughly in that order.

Performance overview#

  • Workflow health: the share of runs that passed, as a meter (in Pro, with its 50% threshold marked).
  • Average build duration, Successful builds and Failed builds for the selected dates, each with its last 14 days as a sparkline.
  • Most failed repositories and Most failed workflows: the top three of each by failed runs. A repository links to the repository on GitHub, and a workflow opens its workflow file on the default branch, so you can jump from the hotspot to the fix.
  • Repositories monitored (in Pro, of those connected) and Pipelines that ran (pipelines with at least one concluded run in the selected dates).

Guidance for reading the meter: a single bad day rarely means much; one broken merge can sink a day's success rate on its own. What deserves attention is a sustained drop across the window, or a gap between repositories: if one repo's health sits far below the rest, Most failed workflows will usually name the workflow responsible.

  • Successful builds: the daily trend of successful builds, one line for each of the four repositories with the most and an Other line for the rest.
  • Average build duration and Workflow duration: bar charts ranking repositories and workflows by average duration, each with a Top 5 / Bottom 5 switch.
  • Build duration trend: the daily average across workflows, to spot pipelines that are getting slower.

Duration trends come in two shapes worth telling apart. A step change (duration jumps on a specific day and stays there) almost always traces back to a concrete edit: a new step, a dependency change, a different runner. A slow creep is the quieter problem: growing test suites and accumulating steps that nobody notices week to week. The trend charts exist to make the creep visible. Latchkey's nightly reliability detectors watch the same run history: a sustained run-time regression surfaces as a reliability finding on AI Insight, with the evidence one click away.

Runs and healed runs#

This page counts runs rather than listing them. Each run, its jobs and their logs are on the Runs page.

When self-heal repairs a Latchkey CI run that then passes, the Runs page shows its result with Healed beside it; the word never replaces the result. On a Latchkey CI job, What Latchkey did names the heal, and Open the heal shows what failed, the diagnosis and the exact action taken, in plain language. A healed run passed, so it counts toward Workflow health here, but read it as a rescue rather than a clean pass: if the same workflow keeps needing heals, the underlying flakiness is still there to fix. See Self-healing for how healing works.

Recovery & comparison#

  • Build fail rate and Rebuild success rate, each a meter: how often builds fail, and how often a re-run goes green.
  • Mean time to recovery (MTTR): the average time between a failed build and the next successful build, in the strip and as a daily trend in its own chart.
  • Build status: successful, failed and cancelled builds per repository in a stacked bar chart, most failed first, with a Top 5 / Bottom 5 switch.

What each recovery metric tells you#

MTTR is the average time between a failed build and the next successful build. It measures your team's whole recovery loop: noticing the red build, diagnosing it, and landing the fix. There is no universal "good" number; a team deploying many times a day needs MTTR measured in minutes, while a weekly-release team can tolerate hours. What is universally bad is a rising Mean time to recovery trend: it means failures are getting harder to diagnose, or the team is getting slower to respond, and either one compounds.

Rebuild success rate is the flakiness detector. It tells you what fraction of failures disappear when you simply run the build again, and the two extremes point at very different problems:

High rebuild success rate
  • Most failures vanish on re-run: they were transient
  • Points at flaky tests, network hiccups, or fragile setup steps
  • Convenient day to day, but retries are masking reliability debt
  • The fix is stabilizing the flaky thing, not retrying harder
Low rebuild success rate
  • Failures stay red on re-run: they are real
  • Points at genuine code or configuration regressions
  • Your fail rate is telling the truth about code quality
  • The fix lives in the changes being merged, not in CI

Build fail rate only becomes meaningful next to rebuild success rate. A 10% fail rate made of real regressions and a 10% fail rate made of flaky re-run-and-forget failures are entirely different problems, and this page gives you both numbers precisely so you can tell them apart.

Export#

Export, in the title bar, downloads the figures the page draws as a CSV file named for the first and last days it covers, one row per figure under the columns Panel, Name, Figure, Value and Unit. A chart with a Top 5 / Bottom 5 switch is written in the order it shows. Export is unavailable while the page loads, when its figures did not load, and when no run concluded in the selected dates and filters.

Metric glossary#

MetricWhat it measuresHow to read it
Workflow healthOverall success rate for the selected window, shown as a meter: the runs that passed out of those that passed or failed. A run Latchkey stopped counts toward neither, and neither does a trial run; a live run that ran a pipeline on trial counts by its live pipelines aloneWatch sustained movement, not single-day dips
Average build durationMean run duration across your selected scope. A run Latchkey stopped is left out of this average, even one stopped after some of its jobs had run, and still counts among the runs. A held pull request from a fork is in no figure until it is releasedThe trend matters more than the absolute number
Build fail rateHow often builds fail: the runs that failed out of those that passed or failedInterpret it together with rebuild success rate
Rebuild success rateHow often a re-run of a failed build goes greenHigh = transient/flaky failures; low = real regressions
MTTRAverage time between a failed build and the next successful buildMeasures your detect-diagnose-fix loop end to end
MTTR trendMTTR as a daily trend across the windowDirection matters most; a rising line compounds
A suggested weekly review (adapt to taste)
Q.01What does Healed on the Runs page mean?

It means self-heal repaired something and what it repaired then passed: a Latchkey CI run, or on Latchkey on GH a job. Healed sits beside the result on the Runs page and never replaces it. Open the heal, on a Latchkey CI job, shows what failed, what was applied and whether the step then succeeded, so a heal is auditable rather than something that quietly happened.

Q.02Which metric should I watch to catch CI getting slower?

Build trends, not the current duration. Trends catch slow creep over weeks, which is how most pipelines degrade; a single slow run is noise. Use Most failed repositories and Most failed workflows for the opposite problem, the active fires that need attention today.

Q.03What do MTTR and rebuild success rate mean here?

MTTR is the mean time from a failing run to the next successful one on the same workflow, so it measures recovery rather than failure. Rebuild success rate is the share of reruns that pass; a low rate means reruns are not fixing anything and the failure is real, not flaky.