Flaky tests: Test strategy and fixtures
Flaky tests enter an owned quarantine record with deadline and retry budget; required CI gates stay honest by reporting quarantined failures separately instead of silently turning green.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
This block defines test strategy and fixtures for a required CI suite. Quarantine is one controlled containment mechanism within that strategy—not a substitute for fixing or retiring bad tests. When a test is classified as flaky, it is not deleted from the repository and it is not silently excluded from required CI. Instead, the owning team creates a quarantine record that binds the test identifier, a named owner, a remediation deadline, and a bounded retry budget for diagnostic reruns. Required merge gates continue to evaluate non-quarantined tests and must surface quarantined failures as a distinct signal rather than reporting an all-green required suite.
Scope
- Test suite strategy for required CI, fixture conventions, and quarantine as a bounded sub-mechanism.
- Quarantine record required fields, state lifecycle, and registry semantics in
quarantine-record.yaml. - CI reporting semantics for required versus optional suites when quarantine is active.
- Remediation paths: fix, deflake with evidence, or retire the test through an explicit removal review.
- Metrics and alerts on quarantine age, owner backlog, and retry-budget exhaustion.
Outside this block
- Root-cause analysis techniques for race conditions or environmental drift.
- Test framework plugin implementation details.
- Organization-wide quality scorecards unrelated to quarantine records.
Contract
- A quarantine record requires owner, deadline, retry budget, and test identifier before CI may treat the test as quarantined.
- Required CI gates never turn green solely because a failing required test is quarantined; the gate reports quarantined-required separately from passing.
- Quarantine without an owner or past deadline blocks merge for teams under the strict profile in
example.yaml. - Retry budget consumption is recorded per quarantine id; exhausted budget stops automatic reruns and escalates to the owner.
- Removing quarantine requires either a linked fix commit or an audited retirement decision; silent config deletion is rejected.
- Quarantined tests remain visible in test listings and dashboards with quarantine status, not hidden from developers.
- Duplicate quarantine records for the same test id merge into one active record with preserved audit history.
- Approved waiver allows merge when quarantined-required is nonzero only with audited waiver id and expiry; waivers never hide overdue records from dashboards.
Implementation guidance
- Store quarantine records in version-controlled metadata or a durable registry queried by CI orchestration.
- Emit a machine-readable
quarantined-requiredoutcome on merge gates so bots cannot misread yellow as green. - Tie retry budget to scheduled reruns or manual flake-detection jobs, not unbounded developer reruns.
- Require owner rotation workflow when deadlines slip twice for the same test id.
Failure handling and safeguards
- If CI cannot read quarantine state, fail closed for required suites rather than skipping unknown tests.
- Misclassified stable tests must be revocable within one business day without rewriting history.
- Do not use quarantine to bypass security or compliance tests marked non-quarantinable in the adoption profile.
Verification and operations
- Track count of active quarantines, mean age, overdue records, and required-suite honesty violations.
- Alert when any required gate reports success while quarantined-required count is nonzero without an approved waiver.
- Run acceptance scenarios in the target repository; packaged scenarios are not executed evidence.
Adoption assumptions
- Names and numeric values in
example.yamlandquarantine-record.yamlare profile choices, not universal defaults. - The adopting team maps owners, CI systems, and waiver policy to its organization.