Self-Healing CI: Containing an Intermittent Segfault in a Native Test
A native test that segfaults on one run but passes on the next may be hitting a race or resource blip -- a retry that passes points to flakiness, while a consistent crash points to a real bug.
The problem
A test exercising native code dies with a segfault (SIGSEGV) on one run and passes on the next with no code change. A race, an uninitialized resource under load, or a transient memory condition triggered the crash intermittently. A human re-runs the suite and it goes green.
Segmentation fault (core dumped)
##[error] Process completed with exit code 139.Why it happens
Native code can crash on conditions that only sometimes occur -- a data race, a use-after-free that is timing-dependent, or memory pressure that varies with load -- so the same test can pass or segfault depending on conditions unrelated to a single, deterministic defect.
Intermittence is the key signal but also the trap: a segfault that reproduces consistently is a real bug that must surface, while one that vanishes on a clean retry behaves like a transient flake. Blindly retrying every crash would hide genuine defects.
The manual fix
Manual handling for an intermittent native crash:
- Re-run the suite and check whether the segfault reproduces.
- If it does not reproduce, quarantine the test and file a ticket to fix the underlying race or memory issue.
- Run under sanitizers/valgrind to find the root cause; add a bounded retry only for known-flaky native tests.
How this gets automated
Self-healing CI treats an intermittent segfault conservatively: it retries a crash whose signature matches a transient, non-reproducing flake while surfacing a segfault that reproduces consistently as the real bug it is. The point is not to retry every crash -- it is to separate genuine native defects from one-off, condition-dependent flakes so a red build still means something.