How to Quarantine a Flaky Test in CI

Quarantine means moving a flaky test out of the blocking suite while still running it separately, so the pipeline is stable but the test is not forgotten.

Tag the test as flaky, exclude that tag from the required job, and run the quarantined tests in a second non-blocking job. The goal is a tracked, time-boxed exclusion, not a permanent skip.

Steps

Add a flaky tag or mark to the suspect test.
Exclude the tag from the required test job.
Run the quarantined tests in a non-blocking job that still reports.
File a tracking issue so quarantine is temporary.

Mark and split (pytest)

pytest.ini

# pytest.ini
[pytest]
markers =
    flaky: known-flaky, quarantined pending a fix

Two jobs in the workflow

.github/workflows/ci.yml

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: pytest -m 'not flaky'   # blocking
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - run: pytest -m 'flaky'       # non-blocking, still visible

Gotchas

A quarantine with no owner or deadline becomes a permanent blind spot.
Keep the quarantined job running so a test that starts failing consistently is caught.