Defines a hybrid retrieval contract that combines lexical and semantic candidates while preserving filters, authorization, explainable contribution signals, and measurable relevance behavior.
Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.
Decision
Retrieve lexical and vector candidate lists independently under the same hard filters, normalize them through rank-based fusion, then pass the fused pool to a separate ranking stage. Treat fusion parameters as evaluated profile choices.
Scope
- Query normalization, hard filters, candidate generation, fusion, empty paths, explanation fields, and evaluation.
- Search over items with both an inverted representation and embeddings.
- A relevance contract independent of a specific search vendor.
Outside this block
- Final learned ranking, answer generation, or embedding-model training.
- Using semantic similarity to bypass permissions or mandatory filters.
Contract
- Lexical and vector retrieval receive the same tenant, visibility, status, and other hard-filter predicates.
- Fusion uses stable document identity and records which retrievers contributed and at what rank.
- A document returned by only one retriever remains eligible; absence from one list is not an automatic penalty beyond the fusion rule.
- Query paths handle blank, identifier-like, quoted, rare-term, natural-language, and unsupported-language inputs explicitly.
- When embedding generation or a retriever fails, the declared degraded mode and user-visible behavior apply.
- Offline judgments define success by query segment, including zero-result and known-item behavior.
Implementation guidance
- Use reciprocal rank fusion or another rank-based method when raw score scales are not comparable.
- Version query normalization, lexical analyzer, embedding model, candidate depths, and fusion settings as one retrieval profile.
- Apply post-filtering only for non-security presentation rules; oversample and measure its loss when unavoidable.
- Log privacy-reduced query identity, profile version, candidate contributions, filters, latency, and clicked item.
Failure handling and safeguards
- If vector retrieval times out, return lexical results only when the degraded-mode contract permits it and mark the response profile.
- If either candidate set cannot enforce authorization, do not include it.
- If fused identities collide across tenants or entity types, fail the request and fix identity construction.
Verification and operations
- Evaluate nDCG, MRR or success@k, recall@candidate-depth, zero-result rate, latency, and contribution share by query segment.
- Compare profile changes on fixed judgments before an online experiment.
- Monitor retriever failure, filter mismatch, duplicate identity, and embedding-version coverage.
The executable-looking examples in this package are fixtures and acceptance contracts. Run
python tools/validate.py from the collection root to check package structure and metadata; then
implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - The adopting team must map actors, data classes, error vocabulary, and ownership to its system.