# Flaky Playwright tests in CI: find them, record them, fix them > Fix flaky Playwright tests in CI with the flaky status, traces on first retry, workers and shards, and the JUnit setting that keeps flakes visible. Source: https://latchkey.dev/learn/flaky-tests/flaky-playwright-tests-in-ci Updated: 2026-11-12 Flaky Playwright tests in CI are usually tests that wait on the clock instead of the page, share state across workers, or depend on file order. Playwright gives you the tools to see them: retries mark a test flaky when it fails then passes, `trace: 'on-first-retry'` records what happened, and `--repeat-each` plus `--fail-on-flaky-tests` reproduce and gate them. Use retries to collect evidence, not to turn the build green. The e2e job passes on the second attempt often enough that nobody looks anymore. Playwright already knows which tests are doing it, because a retry that passes is a distinct outcome in its model. The work is making that outcome reach a place someone reads. Everything here was checked against @playwright/test 1.65.0, the current release on 2026-11-12, with a test written to fail on its first attempt and pass on its retry. ## What Playwright calls flaky Playwright sorts every test into three outcomes: passed on the first run, flaky (failed on the first run but passed when retried), and failed on the first run and every retry. By default failing tests are not retried, so there is no flaky outcome until you set `retries`. A retry is not a rerun in place. After a failure Playwright discards the worker process and its browser and starts a new one, so the retry starts clean in memory. State outside the process, such as a database row or a shared account, survives, which is why a test that flakes from shared state often passes on retry. The config below follows the docs for CI and adds two lines. `failOnFlakyTests` (added in 1.52) makes a flaky result fail the run, and the reporters write flaky results to files you can query. In the 1.65.0 check, the list reporter ended with `1 flaky`, and the run exited 0 by default and 1 with `--fail-on-flaky-tests`. ```playwright.config.ts import { defineConfig } from '@playwright/test'; export default defineConfig({ fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : '50%', failOnFlakyTests: !!process.env.NIGHTLY, reporter: process.env.CI ? [['list'], ['json', { outputFile: 'results.json' }], ['junit', { outputFile: 'junit.xml', includeRetries: true }]] : 'html', use: { trace: 'on-first-retry' }, }); ``` > Outcome definitions from playwright.dev/docs/test-retries; config lines from playwright.dev/docs/test-configuration and the TestConfig API page; the 1.45 and 1.52 dates from the Playwright release notes. All read 2026-11-12. `NIGHTLY` is a variable you set yourself, so pull requests keep retries and a scheduled run fails on any flake. ## The default JUnit report hides flaky tests This is the setting that decides whether anything downstream ever sees a flake. With retries on and the JUnit reporter at its defaults, a test that failed then passed is written as an ordinary passing `testcase`, and the suite reports `failures="0"`. Any tool reading that file, your own script or a flaky-test service, counts it as green. Set `includeRetries: true` on the reporter, or `PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1` in the environment, and the same test carries a `flakyFailure` element with the first attempt's error. The docs mention `includeRetries` only in passing, in the `onlyFailures` row of the reporter table. The JSON reporter needs no setting: the test's `status` is `flaky` and `stats.flaky` counts them. ```Output captured locally, @playwright/test 1.65.0 # default JUnit: the flaky test looks like any pass # with includeRetries: the first attempt is kept # JSON reporter: list the flaky tests jq -r '.. | objects | select(has("specs")) | .specs[] | select(any(.tests[]; .status == "flaky")) | "\(.file):\(.line) \(.title)"' results.json ``` ## Wait on the page, not the clock Most Playwright flakes in our experience are a wait that guesses. The `page.waitForTimeout` documentation is blunt: "Tests that wait for time are inherently flaky." A runner with fewer cores than your laptop turns a sleep that was long enough locally into one that is not. Web-first assertions wait for the condition instead. `await expect(locator).toBeVisible()` retries until the element is visible or the expect timeout runs out, 5000 ms by default. `expect(await locator.isVisible()).toBe(true)` checks once, immediately; the best practices page says such a test "won't wait a single second." ```tests/checkout.spec.ts // flaky: guesses how long the request takes, then checks once await page.getByRole('button', { name: 'Pay' }).click(); await page.waitForTimeout(2000); expect(await page.getByText('Payment received').isVisible()).toBe(true); // stable: waits for the outcome, up to the expect timeout await page.getByRole('button', { name: 'Pay' }).click(); await expect(page.getByText('Payment received')).toBeVisible(); ``` ## Workers, fullyParallel and shards change who runs together By default, test files run in parallel and tests in one file run in order in the same worker. That order is a dependency nobody wrote down: a test that only passes after its neighbor set something up breaks the day you turn on `fullyParallel`, or the day sharding splits the file. With `fullyParallel`, shards split by test; without it, by file, so adding files reshuffles which tests share a worker. Workers default to half the logical CPU cores, and the CI docs recommend one worker on CI for stability. If a test fails only when workers run together, it shares something: a user account, a fixture file, a port. Give each worker its own with `testInfo.workerIndex`, which the parallelism docs suggest for exactly this. ```.github/workflows/e2e.yml jobs: e2e: runs-on: ubuntu-latest strategy: fail-fast: false matrix: shard: [1, 2, 3, 4] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: node-version: lts/* cache: npm - run: npm ci - name: Install Chromium run: npx playwright install --with-deps chromium - name: Run tests run: npx playwright test --shard=${{ matrix.shard }}/4 - if: ${{ !cancelled() }} uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: name: playwright-${{ matrix.shard }}-attempt-${{ github.run_attempt }} path: | test-results/ results.json junit.xml ``` ## Reproduce before you fix Run the suspect test many times with retries off and the CI worker count, so a flake has room to show. `--repeat-each` runs each test N times; `--last-failed` reruns only what failed last time. The trace from the first retry, in `test-results/` under a directory ending in `-retry1`, shows what the page looked like when it failed. ```Terminal npx playwright test tests/checkout.spec.ts --repeat-each=30 --retries=0 --workers=1 npx playwright test --last-failed npx playwright show-trace test-results/-retry1/trace.zip ``` ## Browser install time is not test time Install only the browsers you test, as in the workflow above: the best practices page says `npx playwright install chromium --with-deps` saves download time and disk space on CI. The CI docs advise against caching browser binaries, because restoring the cache takes about as long as downloading them, and the Linux system dependencies cannot be cached at all. Keep the install in its own step. A download that fails there is a network failure, covered in [Playwright browser download failed in CI](/learn/failures/playwright-browser-download-failed-in-ci), and should never be counted as a flaky test. ## FAQ ### Why are my Playwright tests flaky in CI but not locally? CI runners usually have fewer cores, run your tests next to other workers, and start from a cold browser install. Fixed waits that were long enough locally stop being long enough, and tests that share accounts or data collide under parallel workers. Replace waits with web-first assertions and give each worker its own data. ### How do I make Playwright fail on flaky tests? Pass `--fail-on-flaky-tests` on the command line, available since 1.45, or set `failOnFlakyTests: true` in the config, added in 1.52. A run with a test that failed then passed then exits nonzero. Many teams keep retries on pull requests and turn this on for a scheduled run. ### What does trace on first retry do in Playwright? With `trace: 'on-first-retry'`, Playwright records a trace only when a failed test is retried for the first time, so a passing suite pays no tracing cost. The trace lands in the retry's directory under `test-results/` and opens with `npx playwright show-trace`. ### Should I cache Playwright browsers in GitHub Actions? Playwright's CI documentation recommends against it: restoring the cache takes about as long as downloading the browsers, and the Linux system dependencies cannot be cached. Install only the browsers you test instead. ## References - [Playwright: test retries (read 2026-11-12)](https://playwright.dev/docs/test-retries) - [Playwright: command line, --fail-on-flaky-tests and --repeat-each](https://playwright.dev/docs/test-cli) - [Playwright: continuous integration](https://playwright.dev/docs/ci) - [Playwright: parallelism and sharding](https://playwright.dev/docs/test-parallel) - [Playwright: best practices](https://playwright.dev/docs/best-practices) --- Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt