Defines how to evaluate, deploy, observe, and fall back from a RAG reranker using query-document judgments, citation coverage, latency budgets, and versioned candidate contracts.
Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.
Decision
Rerank a bounded, authorization-filtered candidate set and return a smaller evidence set. Promote a reranker only when it improves segmented retrieval quality within latency and cost budgets and does not reduce source diversity below policy.
Scope
- Input contract, truncation, scoring, output, offline judgments, shadowing, rollout, fallback, cost, and drift.
- Reranking text passages before answer generation.
- Evaluation of answer-support coverage as well as ranking metrics.
Outside this block
- The first-stage retriever, answer generation, or factuality guarantees.
- Assuming a vendor score is calibrated across models or versions.
Contract
- Input candidates already satisfy tenant, visibility, and document-state filters.
- Each candidate carries stable chunk ID, document version, source identity, text boundaries, language, and first-stage rank.
- Truncation is deterministic and preserves the fields used during evaluation.
- Scores are comparable only within one query and one reranker profile unless calibration is demonstrated.
- Output preserves provenance and includes an explicit reason when fewer than the requested passages survive.
- Promotion gates cover retrieval quality, support for reference answers, latency, cost, error rate, and protected query segments.
Implementation guidance
- Maintain a no-reranker baseline and a lightweight fallback profile.
- Evaluate at the final context budget, not only at an arbitrary rank cutoff.
- Shadow new models on sampled, privacy-approved traffic before user exposure.
- Version model, prompt or instruction, tokenizer, truncation, candidate depth, and output depth together.
Failure handling and safeguards
- On timeout or provider failure, use the declared first-stage order or fallback; never return an empty context solely because reranking failed.
- Reject candidates whose source authorization or version cannot be verified.
- If input exceeds limits, apply the documented per-candidate and total-budget truncation rather than silently dropping tail candidates.
Verification and operations
- Track nDCG/MRR/recall at context budget, answer-support coverage, diversity, timeout, fallback, tokens, cost, and p95/p99 latency by segment.
- Keep hard queries and temporal holdouts in evaluation sets.
- Re-evaluate after model, tokenizer, corpus, chunking, or first-stage profile changes.
The executable-looking examples in this package are fixtures and acceptance contracts. Run
python tools/validate.py from the collection root to check package structure and metadata; then
implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - The adopting team must map actors, data classes, error vocabulary, and ownership to its system.