Parallelize tests by timing, not by file count

By Kaveh Alemi·Latchkey

Parallelize tests by timing rather than by file name and the shards stop finishing at different times, which is the only thing that decides what a split is worth. On a runner we measured, a twelve-file suite split four ways by the test runner's own file order produced shards of 5.24, 6.24, 8.25 and 9.25 seconds, so a quarter of the work was already done while the last shard still had four seconds to go.

Splitting a suite is easy and balancing it is not. Every test runner that shards will happily give you four groups; almost none of them promise the four groups take the same time, because most of them divide by file in whatever order they enumerate them.

The wall clock of a split is its slowest shard, so an unbalanced split throws away part of what you paid for. This page is about the data that fixes it: where per-test durations come from, what to do with them, and how much the fix is actually worth.

What a file split produced at four shards

We can be specific about the cost, because a script committed for the sharding page measured it. Twelve files with deliberately uneven durations, eight of one second and four of two, three, five and six, run through the test runner's own --shard flag at four. The runner assigned three files to each shard, which looks balanced and is not: what matters is the weight of the files, not the count.

The spread is the story. The slowest shard took 9.25 seconds and the fastest 5.24, a difference of 4.0 seconds on a suite whose whole serial runtime was 26.8. Every shard also paid about 1.25 seconds of overhead before its first test, consistently enough across all four that it can be subtracted: a perfectly balanced quarter of the declared work would be about 7.25 seconds. So the imbalance cost roughly 2.0 seconds of wall clock, a fifth of the split, for no reason except which files landed together.

Where the durations come from

You need a number per test, from a run that already happened. Three places have one and you almost certainly already produce one of them. A JUnit XML report carries a time attribute per test case and nearly every runner can emit one. Some runners have a durations file of their own: pytest --store-durations writes .test_durations, which its plugin documentation says should be stored in the repo in order to have it available during future test runs. Playwright's blob report carries results and attachments for everything that ran, and is designed to be merged across shards.

The decision that follows is where the file lives, and both answers are defensible. Committing it makes the split reproducible and puts a diff in front of a reviewer when the timings move, at the cost of a file that churns. Uploading it as an artifact and downloading it on the next run keeps the repository clean, at the cost of a split that silently degrades when the artifact expires and nobody notices.

The two packing algorithms, and which one you want

Once you have durations, assigning them to bins is a scheduling problem with two standard answers, and pytest-split is unusual in shipping both and documenting the tradeoff. Its default, duration_based_chunks, finds boundaries in the existing list so that each group contains all the tests between a start and an end boundary. Its alternative, least_duration, walks the list of tests and assigns each test to the group with the smallest current duration.

The plugin's own table says least_duration gives better split quality, and the reason it is not the default is ordering: the contiguous algorithm preserves the absolute order of your tests and the greedy one does not. If your suite has order-dependent tests, that is not a stylistic preference, it is whether the suite passes. There is a related trap the documentation calls out: duration_based_chunks is incompatible with test-order randomization, because selection happens after randomization, so some tests get picked in several groups and others in none.

# record once, from a full run
pytest --store-durations

# then split, with the greedy packer
pytest --splits 4 --group 1 --splitting-algorithm least_duration

# playwright: shard, then merge the blob reports
npx playwright test --shard=1/4
npx playwright merge-reports --reporter html ./all-blob-reports

Timings drift, and the good tools already handle it

The obvious objection to a stored durations file is that tests are added and renamed constantly, so the data is wrong within a week. In practice the good implementations degrade gently rather than breaking. pytest-split documents that it assumes average test execution time, calculated from the stored information, for every test which does not have duration information stored, so a new test is treated as typical rather than as zero.

That means a refresh is maintenance, not a prerequisite. The plugin's own advice is to update the durations when there are major changes in the suite compared with what is stored, and otherwise to leave it. A reasonable rule is to re-record on the default branch on a schedule, or whenever the spread between your slowest and fastest shard passes some threshold you are willing to name, which is a number your CI already has.

Wiring it into a matrix

The matrix leg number is the shard index, and taking the total from the strategy.job-total context rather than writing 4 twice is what stops the two numbers drifting apart when you widen the matrix later. Two other settings do real work here. fail-fast: false keeps the remaining shards running after one fails, which is the difference between knowing one test broke and knowing the suite is broken. And every shard needs a unique artifact name, or the uploads collide.

Do not start from four. Start from the arithmetic: wall clock is the per-shard fixed cost plus the work divided by the shards, while the bill is the fixed cost times the shards plus the work. Balancing removes the waste inside a given width; it does not change the shape of that curve, which is what test sharding across runners works through in detail.

jobs:
  test:
    runs-on: latchkey-small
    strategy:
      fail-fast: false
      matrix:
        group: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
        with:
          python-version: '3.13'
          cache: pip
      - run: pip install -r requirements.txt
      - run: |
          pytest --splits ${{ strategy.job-total }} \
                 --group ${{ matrix.group }} \
                 --splitting-algorithm least_duration \
                 --junitxml=junit-${{ matrix.group }}.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: junit-${{ matrix.group }}
          path: junit-${{ matrix.group }}.xml

What balancing cannot fix

Balance is worth exactly the spread it removes and not a second more. On the four-way split above that was about 2.0 seconds out of 9.25, which is real, and it is smaller than the 1.25 seconds of fixed cost every shard paid before running anything. Widen far enough and you are paying that fixed cost four, eight, sixteen times while the balanced work per shard shrinks toward it.

The other term is queue time, and nothing on this page touches it. Our shards ran one after another inside one job, so these numbers contain no waiting for a machine. On a real matrix each shard queues on its own, and once the width crosses your plan ceiling the later shards wait for the earlier ones to finish, which no amount of duration data improves.

What we ran, so you can disagree with it

No new runner job was run for this page. Every measured number above is read out of job-n.log, committed beside the script that produced it and a status file carrying the job id, the runner size and the exit code, from a run on 20 September 2026. The suite is synthetic on purpose: the durations are declared in the script, so the split arithmetic can be checked against the intended weights rather than against something we have to trust.

Three honest limits. One pass per row means a difference under about half a second is not a result. The twelve files are evenly shaped in a way real suites are not, which makes the file split look better than it usually is rather than worse. And the balanced target of 7.25 seconds is arithmetic on the measured overhead, not a fifth run: we did not run a duration-balanced four-way split, only the two-way one that the sharding page reports.

Frequently asked questions

Related guides

References

• pytest-split: storing durations, the splits and group flags, and both splitting algorithms (verified 2026-09-21)
• Playwright: sharding, file-level granularity, fullyParallel and merge-reports (verified 2026-09-21)
• Vitest CLI: the shard flag and file parallelism (verified 2026-09-21)
• GitHub Docs: running variations of jobs in a workflow, including the strategy context (verified 2026-09-21)
• GitHub Actions documentation