Flaky Playwright tests in CI: find them, record them, fix them

By Kaveh Alemi·Latchkey

Flaky Playwright tests in CI are usually tests that wait on the clock instead of the page, share state across workers, or depend on file order. Playwright gives you the tools to see them: retries mark a test flaky when it fails then passes, trace: 'on-first-retry' records what happened, and --repeat-each plus --fail-on-flaky-tests reproduce and gate them. Use retries to collect evidence, not to turn the build green.

The e2e job passes on the second attempt often enough that nobody looks anymore. Playwright already knows which tests are doing it, because a retry that passes is a distinct outcome in its model. The work is making that outcome reach a place someone reads.

Everything here was checked against @playwright/test 1.65.0, the current release on 2026-11-12, with a test written to fail on its first attempt and pass on its retry.

What Playwright calls flaky

Playwright sorts every test into three outcomes: passed on the first run, flaky (failed on the first run but passed when retried), and failed on the first run and every retry. By default failing tests are not retried, so there is no flaky outcome until you set retries.

A retry is not a rerun in place. After a failure Playwright discards the worker process and its browser and starts a new one, so the retry starts clean in memory. State outside the process, such as a database row or a shared account, survives, which is why a test that flakes from shared state often passes on retry.

The config below follows the docs for CI and adds two lines. failOnFlakyTests (added in 1.52) makes a flaky result fail the run, and the reporters write flaky results to files you can query. In the 1.65.0 check, the list reporter ended with 1 flaky, and the run exited 0 by default and 1 with --fail-on-flaky-tests.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : '50%',
  failOnFlakyTests: !!process.env.NIGHTLY,
  reporter: process.env.CI
    ? [['list'], ['json', { outputFile: 'results.json' }],
       ['junit', { outputFile: 'junit.xml', includeRetries: true }]]
    : 'html',
  use: { trace: 'on-first-retry' },
});

The default JUnit report hides flaky tests

This is the setting that decides whether anything downstream ever sees a flake. With retries on and the JUnit reporter at its defaults, a test that failed then passed is written as an ordinary passing testcase, and the suite reports failures="0". Any tool reading that file, your own script or a flaky-test service, counts it as green.

Set includeRetries: true on the reporter, or PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1 in the environment, and the same test carries a flakyFailure element with the first attempt's error. The docs mention includeRetries only in passing, in the onlyFailures row of the reporter table. The JSON reporter needs no setting: the test's status is flaky and stats.flaky counts them.

# default JUnit: the flaky test looks like any pass
<testcase name="checkout total" classname="flaky.spec.ts" time="0.003">

# with includeRetries: the first attempt is kept
<testcase name="checkout total" classname="flaky.spec.ts" time="0.002">
<flakyFailure message="expect(received).toBeGreaterThan(expected)" type="expect.toBeGreaterThan" time="0.002">

# JSON reporter: list the flaky tests
jq -r '.. | objects | select(has("specs")) | .specs[]
  | select(any(.tests[]; .status == "flaky")) | "\(.file):\(.line) \(.title)"' results.json

Wait on the page, not the clock

Most Playwright flakes in our experience are a wait that guesses. The page.waitForTimeout documentation is blunt: "Tests that wait for time are inherently flaky." A runner with fewer cores than your laptop turns a sleep that was long enough locally into one that is not.

Web-first assertions wait for the condition instead. await expect(locator).toBeVisible() retries until the element is visible or the expect timeout runs out, 5000 ms by default. expect(await locator.isVisible()).toBe(true) checks once, immediately; the best practices page says such a test "won't wait a single second."

// flaky: guesses how long the request takes, then checks once
await page.getByRole('button', { name: 'Pay' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Payment received').isVisible()).toBe(true);

// stable: waits for the outcome, up to the expect timeout
await page.getByRole('button', { name: 'Pay' }).click();
await expect(page.getByText('Payment received')).toBeVisible();

Workers, fullyParallel and shards change who runs together

By default, test files run in parallel and tests in one file run in order in the same worker. That order is a dependency nobody wrote down: a test that only passes after its neighbor set something up breaks the day you turn on fullyParallel, or the day sharding splits the file. With fullyParallel, shards split by test; without it, by file, so adding files reshuffles which tests share a worker.

Workers default to half the logical CPU cores, and the CI docs recommend one worker on CI for stability. If a test fails only when workers run together, it shares something: a user account, a fixture file, a port. Give each worker its own with testInfo.workerIndex, which the parallelism docs suggest for exactly this.

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - name: Install Chromium
        run: npx playwright install --with-deps chromium
      - name: Run tests
        run: npx playwright test --shard=${{ matrix.shard }}/4
      - if: ${{ !cancelled() }}
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: playwright-${{ matrix.shard }}-attempt-${{ github.run_attempt }}
          path: |
            test-results/
            results.json
            junit.xml

Reproduce before you fix

Run the suspect test many times with retries off and the CI worker count, so a flake has room to show. --repeat-each runs each test N times; --last-failed reruns only what failed last time. The trace from the first retry, in test-results/ under a directory ending in -retry1, shows what the page looked like when it failed.

npx playwright test tests/checkout.spec.ts --repeat-each=30 --retries=0 --workers=1
npx playwright test --last-failed
npx playwright show-trace test-results/<test-dir>-retry1/trace.zip

Browser install time is not test time

Install only the browsers you test, as in the workflow above: the best practices page says npx playwright install chromium --with-deps saves download time and disk space on CI. The CI docs advise against caching browser binaries, because restoring the cache takes about as long as downloading them, and the Linux system dependencies cannot be cached at all.

Keep the install in its own step. A download that fails there is a network failure, covered in Playwright browser download failed in CI, and should never be counted as a flaky test.

Key takeaways

• A Playwright test that fails then passes on retry is flaky, and the list reporter prints the count.
• The default JUnit report records a flaky test as a pass. Set includeRetries: true.
• Replace waitForTimeout and one-shot isVisible() checks with web-first assertions.
• Gate a nightly run with --fail-on-flaky-tests, and reproduce with --repeat-each and retries off.

Frequently asked questions

Related guides

References

• Playwright: test retries (read 2026-11-12)
• Playwright: command line, --fail-on-flaky-tests and --repeat-each
• Playwright: continuous integration
• Playwright: parallelism and sharding
• Playwright: best practices
• GitHub Actions documentation