Access-controlled RAG: Pipeline implementation and metadata
Enforces authorization before retrieval and aggregation across index, cache, count, and delete surfaces so RAG pipelines never rely on post-generation filtering alone to protect restricted knowledge.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
Access control in RAG is enforced where data enters and leaves retrieval paths, not only by trimming model answers after generation. Ingest attaches document ACLs; search and vector retrieval apply the principal filter before ranking; facet and source counts use the same predicate; cache keys include tenant and principal scope; delete requires ownership evidence. Post-generation filtering alone is forbidden as the sole control because it cannot remove information already passed to the model context window. When no authorized chunks exist, generation stops with an explicit no-context outcome.
Scope
- Authorization enforcement on index write, retrieval query, aggregation counts, cache read/write, generation context assembly, and delete.
- Tenant isolation and principal-scoped cache behavior.
- Audit logging for denials and index mismatches.
- Empty authorized retrieval set handling.
Outside this block
- Embedding model version migrations.
- Chunking strategy and citation formatting.
- Model routing and structured output validation.
Contract
- Retrieval queries apply the principal ACL filter as part of the index query before ranking or embedding similarity scoring executes.
- Facet counts, source counts, and analytics aggregates use authorization-before-count semantics identical to result retrieval.
- Cache entries are keyed by tenant and principal scope; cross-tenant cache reuse is forbidden.
- Generation receives only chunk identifiers from the authorized allow list; post-generation redaction does not satisfy this block as the only control.
- Delete operations on indexed documents require ownership evidence or an audited admin role; denials write immutable audit records.
- When the authorized allow list is empty, the pipeline returns no-authorized-context and skips model invocation unless a separate public-knowledge profile is explicitly selected.
Implementation guidance
- Encode ACL predicates once and inject into ingest pipelines, search clients, and cache middleware.
- Store ACL version on chunks so permission revocations trigger re-filtering even on cache hits.
- CI policy checks can detect pipelines that retrieve global top-k before application filtering.
Failure handling and safeguards
- Index rows missing ACL metadata fail closed at ingest rather than defaulting to public.
- Cache poisoning tests verify two tenants with identical queries never share chunk payloads.
- Log blocked chunk identifiers at summary level without leaking restricted document titles to unauthorized support viewers.
Verification and operations
- Measure denied retrieval rate, empty allow-list rate, cache scope violations, and delete denial counts.
- Penetration-test with cross-tenant principals on shared indexes.
- Prove counts and results exclude the same unauthorized document set in integration fixtures.
The executable-looking examples in this package are fixtures and acceptance contracts. Run static validation on the package structure and metadata, then implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - NIST AI RMF and OWASP authorization guidance inform the threat model but do not certify compliance.
- Teams map principal scopes and ownership evidence to their identity and document systems.