# Parallelize tests by timing, not by file count

> Parallelize tests by timing and the slowest shard stops setting your wall clock: where durations come from, which packer to use, what it is worth.

Source: https://latchkey.dev/learn/speed/parallelize-tests-by-timing-in-ci  
Updated: 2026-09-21

Parallelize tests by timing rather than by file name and the shards stop finishing at different times, which is the only thing that decides what a split is worth. On a runner we measured, a twelve-file suite split four ways by the test runner's own file order produced shards of 5.24, 6.24, 8.25 and 9.25 seconds, so a quarter of the work was already done while the last shard still had four seconds to go.

Splitting a suite is easy and balancing it is not. Every test runner that shards will happily give you four groups; almost none of them promise the four groups take the same time, because most of them divide by file in whatever order they enumerate them.

The wall clock of a split is its slowest shard, so an unbalanced split throws away part of what you paid for. This page is about the data that fixes it: where per-test durations come from, what to do with them, and how much the fix is actually worth.

## What a file split produced at four shards

We can be specific about the cost, because a script committed for the sharding page measured it. Twelve files with deliberately uneven durations, eight of one second and four of two, three, five and six, run through the test runner's own `--shard` flag at four. The runner assigned three files to each shard, which looks balanced and is not: what matters is the weight of the files, not the count.

The spread is the story. The slowest shard took 9.25 seconds and the fastest 5.24, a difference of 4.0 seconds on a suite whose whole serial runtime was 26.8. Every shard also paid about 1.25 seconds of overhead before its first test, consistently enough across all four that it can be subtracted: a perfectly balanced quarter of the declared work would be about 7.25 seconds. So the imbalance cost roughly 2.0 seconds of wall clock, a fifth of the split, for no reason except which files landed together.

| Shard | Files the runner assigned | Declared test work | Measured |
| --- | --- | --- | --- |
| `--shard=1/4` | `test-01`, `test-04`, `test-11` | 7,000 ms | 8,252 ms |
| `--shard=2/4` | `test-02`, `test-08`, `test-10` | 5,000 ms | 6,240 ms |
| `--shard=3/4` | `test-03`, `test-06`, `test-12` | 8,000 ms | 9,249 ms |
| `--shard=4/4` | `test-05`, `test-07`, `test-09` | 4,000 ms | 5,243 ms |

> Measured by `job-n.sh`, under content/repro/timings/shard-tests-across-github-actions-runners/, on a Latchkey `latchkey-small` runner on 2026-09-20: 2 vCPU, Node 20.20.2, vitest 3.2.4, one pass per row, `--no-file-parallelism` on every row so the split is what is being measured. The file assignments are the `FILES` lines that script prints for each shard. The declared column is the durations the script wrote into the files, not a measurement. The balanced target of about 7.25 seconds is arithmetic on the measured per-shard overhead and was not run.

## Where the durations come from

You need a number per test, from a run that already happened. Three places have one and you almost certainly already produce one of them. A JUnit XML report carries a `time` attribute per test case and nearly every runner can emit one. Some runners have a durations file of their own: `pytest --store-durations` writes `.test_durations`, which its plugin documentation says should be stored in the repo in order to have it available during future test runs. Playwright's blob report carries results and attachments for everything that ran, and is designed to be merged across shards.

The decision that follows is where the file lives, and both answers are defensible. Committing it makes the split reproducible and puts a diff in front of a reviewer when the timings move, at the cost of a file that churns. Uploading it as an artifact and downloading it on the next run keeps the repository clean, at the cost of a split that silently degrades when the artifact expires and nobody notices.

## The two packing algorithms, and which one you want

Once you have durations, assigning them to bins is a scheduling problem with two standard answers, and pytest-split is unusual in shipping both and documenting the tradeoff. Its default, `duration_based_chunks`, finds boundaries in the existing list so that each group contains all the tests between a start and an end boundary. Its alternative, `least_duration`, walks the list of tests and assigns each test to the group with the smallest current duration.

The plugin's own table says `least_duration` gives better split quality, and the reason it is not the default is ordering: the contiguous algorithm preserves the absolute order of your tests and the greedy one does not. If your suite has order-dependent tests, that is not a stylistic preference, it is whether the suite passes. There is a related trap the documentation calls out: `duration_based_chunks` is incompatible with test-order randomization, because selection happens after randomization, so some tests get picked in several groups and others in none.

```Terminal
# record once, from a full run
pytest --store-durations

# then split, with the greedy packer
pytest --splits 4 --group 1 --splitting-algorithm least_duration

# playwright: shard, then merge the blob reports
npx playwright test --shard=1/4
npx playwright merge-reports --reporter html ./all-blob-reports
```

> Flags and algorithm descriptions read from the pytest-split README at release 0.11.0, published 2026-02-03, and from the Playwright sharding guide, both read 2026-09-21. Playwright does not balance by duration: its documentation says it defaults to file-level granularity, and that `fullyParallel: true` distributes individual tests instead, which is a different and usually better-balanced unit.

## Timings drift, and the good tools already handle it

The obvious objection to a stored durations file is that tests are added and renamed constantly, so the data is wrong within a week. In practice the good implementations degrade gently rather than breaking. pytest-split documents that it assumes average test execution time, calculated from the stored information, for every test which does not have duration information stored, so a new test is treated as typical rather than as zero.

That means a refresh is maintenance, not a prerequisite. The plugin's own advice is to update the durations when there are major changes in the suite compared with what is stored, and otherwise to leave it. A reasonable rule is to re-record on the default branch on a schedule, or whenever the spread between your slowest and fastest shard passes some threshold you are willing to name, which is a number your CI already has.

## Wiring it into a matrix

The matrix leg number is the shard index, and taking the total from the `strategy.job-total` context rather than writing `4` twice is what stops the two numbers drifting apart when you widen the matrix later. Two other settings do real work here. `fail-fast: false` keeps the remaining shards running after one fails, which is the difference between knowing one test broke and knowing the suite is broken. And every shard needs a unique artifact name, or the uploads collide.

Do not start from four. Start from the arithmetic: wall clock is the per-shard fixed cost plus the work divided by the shards, while the bill is the fixed cost times the shards plus the work. Balancing removes the waste inside a given width; it does not change the shape of that curve, which is what [test sharding across runners](/learn/speed/shard-tests-across-github-actions-runners) works through in detail.

```.github/workflows/ci.yml
jobs:
  test:
    runs-on: latchkey-small
    strategy:
      fail-fast: false
      matrix:
        group: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
        with:
          python-version: '3.13'
          cache: pip
      - run: pip install -r requirements.txt
      - run: |
          pytest --splits ${{ strategy.job-total }} \
                 --group ${{ matrix.group }} \
                 --splitting-algorithm least_duration \
                 --junitxml=junit-${{ matrix.group }}.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: junit-${{ matrix.group }}
          path: junit-${{ matrix.group }}.xml
```

## What balancing cannot fix

Balance is worth exactly the spread it removes and not a second more. On the four-way split above that was about 2.0 seconds out of 9.25, which is real, and it is smaller than the 1.25 seconds of fixed cost every shard paid before running anything. Widen far enough and you are paying that fixed cost four, eight, sixteen times while the balanced work per shard shrinks toward it.

The other term is queue time, and nothing on this page touches it. Our shards ran one after another inside one job, so these numbers contain no waiting for a machine. On a real matrix each shard queues on its own, and once the width crosses your plan ceiling the later shards wait for the earlier ones to finish, which no amount of duration data improves.

## What we ran, so you can disagree with it

No new runner job was run for this page. Every measured number above is read out of `job-n.log`, committed beside the script that produced it and a status file carrying the job id, the runner size and the exit code, from a run on 20 September 2026. The suite is synthetic on purpose: the durations are declared in the script, so the split arithmetic can be checked against the intended weights rather than against something we have to trust.

Three honest limits. One pass per row means a difference under about half a second is not a result. The twelve files are evenly shaped in a way real suites are not, which makes the file split look better than it usually is rather than worse. And the balanced target of 7.25 seconds is arithmetic on the measured overhead, not a fifth run: we did not run a duration-balanced four-way split, only the two-way one that the sharding page reports.

## FAQ

### How do I record per-test timings for a balanced split?

Use whatever your runner already emits. A JUnit XML report has a time attribute per test case. pytest has a purpose-built one: `pytest --store-durations` writes a `.test_durations` file that the plugin documentation says to store in the repository so it is available on future runs. Playwright takes a different route and merges blob reports from every shard after the fact.

### Which splitting algorithm gives the most balanced shards?

The greedy one. pytest-split documents two: `duration_based_chunks`, the default, which cuts the existing list at boundaries and preserves absolute order, and `least_duration`, which walks the tests and assigns each to the group with the smallest current duration. Its own comparison rates `least_duration` as the better split quality, at the cost of reordering, which matters only if your suite has order-dependent tests.

### What happens to a new test that has no recorded duration?

Good implementations assume it is average rather than free. pytest-split documents that it uses the average test execution time calculated from the stored information for every test that has no duration recorded, which is why the durations file does not need regenerating every time someone adds a test. Refresh it when the suite changes substantially, or when your shard spread widens past a threshold you pick.

### How much wall clock does balancing actually buy?

Only the spread between your slowest and fastest shard. On the four-way split we measured, the shards came in at 5.24, 6.24, 8.25 and 9.25 seconds, and a balanced quarter would have been about 7.25, so the imbalance was costing about 2.0 seconds out of 9.25. That is worth having and it is a smaller lever than the number of shards or the per-shard setup cost.

## References

- [pytest-split: storing durations, the splits and group flags, and both splitting algorithms (verified 2026-09-21)](https://github.com/jerry-git/pytest-split)
- [Playwright: sharding, file-level granularity, fullyParallel and merge-reports (verified 2026-09-21)](https://playwright.dev/docs/test-sharding)
- [Vitest CLI: the shard flag and file parallelism (verified 2026-09-21)](https://vitest.dev/guide/cli.html)
- [GitHub Docs: running variations of jobs in a workflow, including the strategy context (verified 2026-09-21)](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations)

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt