Graceful degradation: Reference architecture

Places a breaker per downstream dependency with an explicit fallback, a bounded half-open probe budget, and no shared breaker that hides a tenant-specific outage.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Each downstream dependency has its own circuit breaker. When open, callers receive an explicit fallback rather than blocking on timeouts. Half-open state allows only a budgeted number of probes. Breakers are not shared across tenants in a way that lets one tenant's error rate open the circuit for everyone, and they are not shared across unrelated dependencies.

Scope

• Breaker identity, failure thresholds, open timeout, half-open probe budget, fallback contracts, tenant isolation, and metrics for outbound calls to downstream dependencies.
• Interaction with request deadlines: a breaker trip is not an excuse to ignore an already expired deadline.
• Operator reset of a stuck open breaker.

Outside this block

• Rate-limit policy for inbound product APIs.
• Retry budgets that belong to the deadline-propagation block.

Contract

• Breaker keys are dependency name plus isolation partition (global or tenant when the dependency is tenant-scoped); payment and email cannot share a breaker.
• Open state fails fast into a declared fallback: cached stale, degraded feature, or error code; an empty catch that returns success is forbidden.
• Half-open allows at most the probe budget concurrent calls; excess calls continue to fallback until probes succeed.
• Successes in half-open close the breaker only after the success threshold; a single success does not reopen full traffic if the threshold is greater than one.
• Tenant partitions cannot be collapsed into one global counter when the runbook marks the dependency tenant-isolated.
• Deadline-expired calls count as caller-cancelled, not as downstream failures, unless the dependency already returned an error.

Implementation guidance

• Name breakers after the dependency client module, not after the inbound HTTP route.
• Size open duration from dependency SLO, not from a copied default; too short causes flap, too long extends outage.

Failure handling

• Fallback failure (empty cache, secondary down) returns a dependency-unavailable error with retry guidance, not a 200.
• Clock jump that skips open timeout: still require at least one probe cycle before closing.

Verification and operations

• Measure open time, probe rejects, fallback use, and false global trips during a single-tenant fault injection.
• Alert when a breaker stays open beyond the dependency's expected recovery bound.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• Per-dependency breakers, half-open probe budgets, and tenant partitions are project decisions recorded in this package; they are not a public standard.