How to Build a Flaky Test Dashboard

A flaky test is one that both passes and fails on the same commit; a dashboard just ranks tests by how often they flip.

Parse the JUnit XML your test runner emits, push a pass/fail counter per test id, and rank by the ratio of failures to total runs. Tests with both outcomes on unchanged code are the flaky ones.

Steps

• Emit JUnit XML from the test run.
• Push a test_result_total counter labeled by test id and outcome.
• Rank tests by failure share on a Grafana panel.

Parse and push

- name: Record test outcomes
  if: always()
  run: |
    python - <<'PY'
    import xml.etree.ElementTree as ET, os, urllib.request
    for case in ET.parse("junit.xml").getroot().iter("testcase"):
        outcome = "fail" if case.find("failure") is not None else "pass"
        print(f'test_result_total{{test="{case.get(\"name\")}",outcome="{outcome}"}} 1')
    PY

Rank query

topk(10,
  sum(increase(test_result_total{outcome="fail"}[7d])) by (test)
  / sum(increase(test_result_total[7d])) by (test))

Related guides