Under load, services expose bounded queues, explicit admission decisions, retry hints, and load-shedding states so callers degrade predictably instead of timing out against unbounded buffers.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
Overload is handled as an explicit contract between producers and consumers. Internal work queues are bounded; when bounds are reached the service returns a typed overload response with retry guidance instead of accepting work that will expire in an invisible buffer. Request tiers split into critical, non-sheddable, and deferrable. Critical work receives reserved capacity and is rejected only after its own hard budget is exhausted. Non-sheddable means accepted work is durably retained; it does not mean admission is unlimited. Deferrable traffic is rejected first, then lower-priority admitted traffic is capped so overload protection itself cannot become an unbounded queue. Recovery returns to normal only after queue depth stays below a stable threshold for a profile duration and the ramp budget completes. Clients must honor retry-after or equivalent hints to avoid retry storms.
Scope
- Queue depth limits for ingress, worker pools, and downstream fan-out.
- Admission control outcomes: accept, defer, reject-with-hint, shed.
- Response fields for overload: error code, retry-after, optional degraded-mode flag.
- Client obligations for exponential backoff and jitter when hints are present.
- Metrics and alerts on time-in-queue and shed rate.
Outside this block
- End-user marketing degradation copy.
- Global CDN caching strategy.
- Capacity planning forecasts.
Contract
- Every async ingress path declares a maximum queue depth; beyond it new work receives reject-with-hint, not silent buffering.
- Overload responses include machine-readable retry hints and a stable error code distinguishable from application validation failures.
- Each tier declares a hard admission budget. Shedding preserves reserved capacity for critical work but rejects even critical traffic with an explicit overload response after that capacity is exhausted.
- Accepted non-sheddable work is never silently discarded. When its queue budget is exhausted, new work is rejected or redirected before acknowledgement rather than admitted without a bound.
- Congested state defers deferrable tiers first; non-sheddable work uses its dedicated queue and admission budget rather than sharing the deferrable backlog.
- Transition from recovering to normal requires stable queue depth below the profile threshold for the configured interval and completed ramp, not an immediate flip on first dip.
- Time-in-queue above SLO triggers early reject rather than late timeout inside workers.
- Downstream callers propagate overload signals instead of wrapping them as generic 500 errors.
- Bounded retry from clients respects retry-after max; unbounded immediate retries are rate-limited at the edge.
- Transition from normal to shedding and back is observable through overload state metrics.
Implementation guidance
- Prefer small per-tenant sub-queues to isolate noisy neighbors before global shed.
- Document canonical error schema in API specs alongside rate-limit responses.
- Run load tests that verify queue bounds hold under sustained overload.
Failure handling and safeguards
- If overload controller data is stale, prefer reject-with-hint over unbounded accept.
- Shedding must not discard operations already acknowledged as non-sheddable. Admission still fails explicitly when the dedicated hard cap is reached.
- Recovery reduces shed rate gradually to avoid flip-flop thundering herds.
Verification and operations
- Monitor queue depth, shed rate, hint compliance by top clients, and p99 time-in-queue.
- Alert when shed rate exceeds profile threshold for more than five minutes.
- Execute acceptance scenarios in the target repository; packaged scenarios are not executed evidence.
Adoption assumptions
- Queue sizes and tier definitions in
example.yamlandoverload-states.yamlare profile choices. - Teams map error vocabularies and client SDK behavior locally.