Defines the ranking boundary, objectives, hard constraints, tie-breaking, explanation evidence, and offline-to-online gates needed to change result order safely and reproducibly.
Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.
Decision
Rank only authorized candidates produced by retrieval. Separate non-negotiable eligibility and policy constraints from the scored objective, version every feature and model contract, and use deterministic tie-breaking.
Scope
- Ranking inputs, objectives, constraints, feature availability, output, ties, evaluation, rollout, and rollback.
- A result-list ranker operating after candidate retrieval.
- Segmented relevance and guardrail metrics.
Outside this block
- Candidate generation or business-policy definition.
- Assuming clicks are unbiased relevance labels.
Contract
- The ranker never reintroduces filtered or unauthorized items.
- Hard constraints are applied before or around scoring and are named in the response profile.
- Every request records ranker version, feature schema version, candidate-set identity, and experiment assignment.
- Missing features use a documented value or fallback path; they do not inherit memory from a previous candidate.
- Ties resolve by an explicit stable key after score tolerance is applied.
- A release passes offline relevance thresholds and online safety guardrails for each protected segment.
Implementation guidance
- Create a typed feature contract with source, freshness, null semantics, range, and leakage review.
- Use counterfactual or randomized data when estimating effects from biased interaction logs.
- Keep an interpretable rule baseline for fallback and comparison.
- Separate user utility metrics from marketplace, revenue, or policy objectives and record their precedence.
Failure handling and safeguards
- If the model or feature service is unavailable, invoke the declared baseline and expose its ranker version.
- If feature schema is incompatible, reject the model load rather than filling unknown fields silently.
- If guardrails breach, roll back assignment to the last accepted profile while preserving experiment evidence.
Verification and operations
- Track nDCG/MRR/success@k, coverage, calibration where applicable, feature missingness, latency, stability, and guardrails by segment.
- Audit large position changes and unexpected concentration.
- Maintain frozen evaluation sets plus time-based sets to detect drift.
The executable-looking examples in this package are fixtures and acceptance contracts. Run
python tools/validate.py from the collection root to check package structure and metadata; then
implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - The adopting team must map actors, data classes, error vocabulary, and ownership to its system.