Flaky Test
A test whose result flips between pass and fail across runs without the product changing — measured from result history, never guessed, and never silently deleted.
Definition
A flaky test is one that does not give the same verdict twice: it passes, then fails, then passes again on an unchanged product. Flaky tests are dangerous in both directions — a real defect can be dismissed as 'just flaky', and a healthy product can be blocked by noise. MC Testing therefore treats flakiness as a measured property of a case's recorded history, not an excuse attached to a single failure.
The measurement is exact: over a sliding window of the case's 10 most recent results, only PASS and FAIL count as decisive (NEEDS REVIEW, NOT RUN, SKIPPED and BLOCKED carry no pass/fail signal). Count how often the decisive outcome flips between consecutive runs; a case is flagged FLAKY when it has at least 3 decisive results and either at least 3 flips or a flip rate of 30% or more. The minimum sample keeps one flip in two runs from flagging a case on noise alone.
How MC Testing applies it
A flagged case can be quarantined by the owner. A quarantined case is excluded from scoring and from coverage denominators — so it cannot distort the readiness score — but it still executes on every run and its results are still reported. Nothing is hidden: the case shows a FLAKY or QUARANTINED badge with its flip statistics (for example '4 flips / 9 decisive'), and quarantine can be removed at any time, returning the case to scoring.
A failure is never silently deleted in MC Testing: every executed result stays in the test history, and flakiness is computed from that history. See also Readiness Score and Coverage.