Selects models by declared capability, data-handling policy, and cost budget with explicit forbidden fallback paths so sensitive or high-assurance requests never downgrade to disallowed tiers silently.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
Model routing is a policy decision, not an implicit SDK default. Each request declares a class, data sensitivity, and required capabilities such as structured output, context length, or approved PII handling. The router selects a tier that satisfies capability and data policy, respects tenant cost budgets, and may upgrade when capability gaps appear. Fallback to a lower tier is allowed only on explicit allow-list edges; forbidden fallbacks fail closed rather than silently serving a cheaper or non-compliant model. High-assurance and confidential workloads must never downgrade to economy tiers.
Scope
- Tier definitions with capabilities and data-handling policies.
- Request-class to tier mapping, upgrade triggers, and forbidden fallback matrix.
- Per-tenant or per-workload cost budgets and fail-closed behavior on exceedance.
- Observability for route decisions and fallback reasons.
Outside this block
- Structured output validation after model response.
- Prompt injection controls and output guardrails.
- Fine-tuning, hosting, or vendor contract negotiation.
Contract
- Every request resolves to a tier before model invocation; ad hoc per-call overrides require an audited break-glass role when enabled.
- Capability requirements such as structured output, minimum context, or PII-approved handling must be satisfied by the selected tier or an allowed upgrade target.
- Confidential or high-assurance classes forbid routing to economy tiers even under cost pressure or provider outage unless an explicit human-approved contingency profile exists.
- Forbidden fallback edges in the policy matrix are hard errors with logged fallback_reason; they never silently proceed on a disallowed tier.
- Daily or monthly budgets scoped in the profile fail closed when exceeded; budget exceedance never downgrades tier and the router does not borrow from other tenants.
- Successful and failed routing decisions emit selected tier, fallback reason if any, and remaining budget snapshot for operations review.
Implementation guidance
- Encode tiers and fallback edges in configuration consumed by a single router module shared by batch and online paths.
- Separate data_class from feature flags so product teams cannot accidentally mark confidential work as public for cheaper routing.
- Pre-flight capability checks before sending payloads to avoid charging for calls destined to fail validation downstream.
Failure handling and safeguards
- Provider outage on the selected tier attempts only allowed upgrade or lateral fallback targets; otherwise return unavailable.
- Do not log prompt content when the profile disables it, even when routing fails.
- Cache route decisions only when capability and data class match; never cache across tenants.
Verification and operations
- Measure route distribution, forbidden fallback attempts, budget fail-closed rate, and upgrade frequency.
- Test confidential requests against simulated economy-only availability and expect fail-closed behavior.
- Audit break-glass overrides with immutable records when contingency profiles are used.
The executable-looking examples in this package are fixtures and acceptance contracts. Run static validation on the package structure and metadata, then implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - OWASP LLM and cloud responsible-AI guidance inform policy axes but do not mandate a vendor.
- Teams map tier names to concrete model endpoints in deployment configuration.