How to run tests in parallel for coding agents
To run tests in parallel for coding agents, fan the work out into separate jobs, then wait on all of them and reduce the results to one verdict the agent can branch on. On GitHub Actions that is a matrix or one workflow_dispatch run per sub-agent, watched one by one; with the Latchkey CLI it is latchkey run --detach under one batch and a single latchkey wait.
An agent that splits a task across sub-agents soon wants several test runs at the same moment: one per shard of a slow suite, one per candidate fix, one per worktree. Starting them is easy. Knowing when all of them are done, which ones failed, and keeping them from canceling each other or swallowing the team's concurrency is the actual work.
Below, the fan-out and the wait on GitHub Actions come first, with the limits that bite, and then the same shape with detached Latchkey jobs.
Fan-out inside one run: the matrix
If every parallel piece runs the same workflow with a different parameter, a matrix is the cheapest fan-out GitHub has: one dispatch, one run id to watch, and GitHub expands the legs. A matrix generates at most 256 jobs per workflow run, and max-parallel caps how many legs run at once.
The default that hurts an agent is fail-fast. It defaults to true, and when it is on GitHub will cancel all in-progress and queued jobs in the matrix if any job in the matrix fails. A human wants that. An agent sharding a suite usually wants every shard's verdict, because the second failing shard is the one it would otherwise discover on the next attempt. Turn it off.
on:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
max-parallel: 4
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- run: npm ci
- run: npm test -- --shard=${{ matrix.shard }}/4Fan-out across runs: one dispatch per sub-agent
Sub-agents rarely share a parameter list. Each has its own branch or its own task, so each dispatches its own run. The REST endpoint behind workflow_dispatch now answers 200 with workflow_run_id, run_url and html_url, so the old dance of dispatching and then guessing the run id from gh run list is gone. Inputs are capped at 25 properties.
The join is where this gets slow. gh run watch takes one run id, so an agent with four sub-agents watches four runs in sequence. That is fine, because a run that already finished returns at once, and with --exit-status a finished run whose conclusion is not success exits non-zero. Collect every id first, then watch them all.
ids=()
for task in auth billing search export; do
ids+=("$(gh api -X POST repos/{owner}/{repo}/actions/workflows/agent-check.yml/dispatches \
-f ref="agent/$task" -f "inputs[task]=$task" --jq .workflow_run_id)")
done
failed=0
for id in "${ids[@]}"; do
gh run watch "$id" --exit-status --compact >/dev/null || failed=1
done
exit $failedConcurrency groups: how parallel runs cancel each other
This is the failure that looks like flakiness. Copy the usual concurrency block into an agent workflow, keyed on github.ref, and two sub-agents dispatching on the same branch now share a group. GitHub allows at most one running and one pending run per group, and by default a new pending run cancels the pending one before it. With cancel-in-progress: true the running one goes too. The agent reads a canceled run as a failed check and starts again.
Key the group on what makes the work distinct. The workflow-level concurrency expression may use the github, inputs and vars contexts, so a task input works. If the runs genuinely must take turns, queue: max lets up to 100 runs wait in the group instead of replacing each other; GitHub rejects it combined with cancel-in-progress: true.
Above all of this sits the plan's ceiling: on standard GitHub-hosted runners, 20 concurrent jobs on Free, 40 on Pro, 60 on Team and 500 on Enterprise, shared with every human push in the account. Give your coding agent a CI runner covers that ceiling and the macOS sub-limit in full.
on:
workflow_dispatch:
inputs:
task:
required: true
concurrency:
group: agent-${{ github.workflow }}-${{ inputs.task }}
cancel-in-progress: trueThe Latchkey way: detached jobs in one batch
With the Latchkey CLI there is no workflow to dispatch. latchkey run --detach packs the directory it runs from, submits the job to a fresh runner and prints the job id without waiting. A sub-agent in its own worktree therefore ships its own uncommitted edit, and --context <dir> packs another directory instead.
A batch is a name you put on every job of one piece of work, with --batch or the LATCHKEY_BATCH environment variable, which sub-agents inherit from the shell that starts them. Give the batch a budget and a limit on jobs running at once before the burst: jobs past that limit, or past the workspace's own limit, wait in the queue instead of being refused. latchkey wait --batch then waits on every job the batch holds when the wait starts, prints each outcome as it ends, then a table of state, exit code, run time, cost and batch, and exits 0 only when every job's exit code is 0.
export LATCHKEY_BATCH="refactor-auth-$(date +%Y%m%d%H%M%S)"
latchkey budget set "$LATCHKEY_BATCH" 3 --max-concurrent 4
for shard in 1 2 3 4; do
latchkey run --detach "npm ci && npm test -- --shard=$shard/4"
done
latchkey wait --batch "$LATCHKEY_BATCH"
verdict=$? # 0 only if every job in the batch exited 0Reading the result without losing it
Two details decide whether the join is trustworthy. First, --batch on wait takes the jobs the batch holds when the wait starts, and it never reads LATCHKEY_BATCH, so start every job before you wait and pass the name explicitly. Second, read the verdict from the right field. With --output json the wait prints one ended event per job and then a summary; each carries job_id, state, exit_code, run_seconds, cost_usd, batch and failure_reason, and here exit_code is the verdict latchkey run would have given the job.
If the parallel pieces are Latchkey CI runs rather than one-off jobs, the same command waits on them by commit or pull request: latchkey wait --pr <number> --repository <owner/name>.
latchkey wait --batch "$LATCHKEY_BATCH" --output json >wait.ndjson
verdict=$?
jq -r 'select(.event == "ended") | "\(.job_id) \(.state) \(.exit_code) \(.failure_reason // "")"' <wait.ndjsonThe two fan-outs side by side
The limits differ more than the commands do.
| GitHub Actions | Latchkey CLI | |
|---|---|---|
| Unit of fan-out | A matrix leg, or one dispatched run | One detached job |
| Join | gh run watch once per run id | latchkey wait --batch, one command |
| What cancels what | Concurrency groups and fail-fast | Nothing by default; latchkey cancel per job |
| Ceiling on jobs at once | Plan limit: 20 Free, 40 Pro, 60 Team, 500 Enterprise | Workspace limit: 20 Developer, 40 Launch, 80 Scale; batch --max-concurrent |
| Over the ceiling | Jobs queue | Jobs wait in the queue with a queue_reason |
| Spend bound for the burst | None per burst | latchkey budget set on the batch |
When not to fan out
Every parallel job pays its own dependency install and is billed in its own whole minutes, so a two-minute suite split four ways can cost more and finish no sooner. The workspace limit is shared by every agent and every workflow run, and Latchkey limits job creation per workspace per clock hour (120 on Developer, 240 on Launch, 600 on Scale). For a short suite, one job with several checks chained, latchkey run 'npm ci && npm run lint && npm test', is the better shape. For the spend side of a burst, see cap agent CI spend.
Where the numbers come from
Every limit and flag here is quoted from a reference, not measured: docs.github.com for the matrix, concurrency and plan limits, the gh source and help for gh run watch, and latchkey help from CLI 0.16.0 with Latchkey's limits page for the batch side. No recorded run backs the page, and neither evidence file in this repository lists it. A timing captured from one burst would reflect that day's queue more than the shape of the fan-out, so none is offered.
Key takeaways
- Turn
fail-fastoff for agent matrices: it defaults to true and cancels every other leg on the first failure. - Key a concurrency group on the sub-agent's task, not the branch, or parallel runs cancel each other.
- The dispatch API now returns
workflow_run_id, so collect ids at dispatch time and watch each one. latchkey wait --batchturns a burst of detached jobs into one table and one exit code.
Frequently asked questions
How do I wait for multiple workflow runs to complete in GitHub Actions?
workflow_run_id), then run gh run watch <id> --exit-status for each one. A run that already finished returns immediately, so watching in sequence costs no extra time. Inside a single workflow, express the dependency with needs: instead.Why do my GitHub Actions runs get canceled when I start a new one?
cancel-in-progress: true also cancels the running one. Give each parallel run its own group, or set queue: max if they must take turns.How do I limit the number of parallel jobs in a GitHub Actions matrix?
strategy.max-parallel to the most legs you want running at once; the rest wait until a slot frees. It limits one matrix only, so two runs of the same workflow can still overlap unless a concurrency group keeps them apart.Does fail-fast cancel other matrix jobs?
strategy.fail-fast defaults to true, and then GitHub cancels all in-progress and queued jobs in the matrix when any one fails. Set fail-fast: false when you want every leg's result.