Health checks: Instrumentation design
Treats liveness as process-deadlock detection only, includes critical dependencies in readiness, and forbids recycling healthy processes because a downstream dependency is unavailable.
Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.
Decision
Expose two probes. Liveness fails only when this process cannot make progress: deadlock, stuck event loop, or a missed watchdog tick. Readiness fails when the instance must not receive traffic, including a failed critical dependency probe. Never report a downstream outage as liveness failure. Restarting a healthy process does not restore Postgres or object storage; it multiplies connection storms.
Scope
- Probe endpoints consumed by a container orchestrator.
- Critical versus noncritical serving-path dependencies.
- Timeouts and thresholds in
probe-matrix.yaml. - Rolling updates, endpoint removal, and container restart.
Outside this block
- User-journey SLOs and error-budget freezes.
- Multi-step synthetic monitoring.
- Autoscaling from CPU or queue depth.
Contract
- Liveness succeeds whenever the in-process watchdog ticked within the profile interval, even if every downstream dependency is down.
- Readiness succeeds only when every
criticaldependency probe succeeds inside its timeout. - A noncritical failure leaves readiness successful and emits a warning that names the dependency.
- Probe handlers must not acquire unbounded locks, run migrations, or invoke non-idempotent commands.
- Liveness failures equal to
failure_thresholdrestart the container; readiness failures only remove the instance from endpoints. - Probe bodies contain no credentials, connection strings, or row-level payloads.
Implementation
Use distinct paths such as /livez and /readyz. Keep liveness in-process. Fan out readiness using the matrix, with each dependency timeout below the orchestrator timeout. Cache a result only for cache_ms. The matrix is the source of truth for stores on the serving path.
Failure handling
A critical timeout fails readiness closed. A wedged watchdog is recovered by restart. After a flap, require success_threshold successes before advertising ready. Honor the matrix interval instead of tight-loop retries. While a startup probe has not succeeded, the orchestrator suppresses liveness and readiness. Once startup succeeds, liveness and readiness take over; exhausting the startup failure threshold causes the container to be killed and restarted.
Verification
Alert when liveness restarts during a dependency-only outage. Track ready churn against the profile. Re-review the matrix when a datastore joins the serving path. Execute the packaged scenarios on a local orchestrator fixture.
The executable-looking examples in this package are fixtures and acceptance contracts. Run
python tools/validate.py from the collection root to check package structure and metadata; then
implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - The adopting team must map actors, data classes, error vocabulary, and ownership to its system.