Guardrails: System design

Specifies that untrusted retrieved or user content cannot authorize tools or exfiltrate secrets, that the model follows a documented instruction hierarchy, and that outputs are filtered before privileged side effects.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Language-model features treat user turns, retrieved chunks, and tool results as untrusted. Those channels cannot grant tools, mint session capabilities, or place secrets into context. A documented instruction hierarchy helps the model interpret instructions, while runtime capability checks, tool isolation, least privilege, and secret isolation form the primary security boundary. Output filtering is defense in depth for recognizable leakage patterns, not proof that arbitrary sensitive content cannot be exfiltrated. Tool hops are capped.

Scope

• Copilot and RAG answer paths that may call tools.
• Prompt packaging, capability checks, output filters, and hop limits in the adoption profile.
• Mapping in threat-model.md.

Outside this block

• Model quality, evaluation datasets, or citation formatting; see the related grounding block.
• Full tool-calling product policy beyond injection; see the related authorization block.
• Training-time alignment of foundation models.

Contract

• Untrusted channels are labeled in the packaged prompt and cannot set cap-export or cap-egress.
• A tool listed as privileged executes only when the session already holds the matching capability.
• Instruction rank is system-policy, then developer-tool-manifest, then user-turn, retrieved-chunk, and tool-result.
• API tokens and connection strings are forbidden in model context; fixtures must not embed live secrets.
• Output that matches block_patterns is not delivered to the user as-is, and further tools are denied.
• The agent stops after max_tool_hops even if the model requests more calls.

Implementation guidance

• Package prompts with a machine-checked hierarchy. Authorize tools in runtime after the model proposes a call, minimize each tool's reachable data and side effects, and keep secrets outside model-visible context.
• Keep retrieval text out of system-policy. Unknown tools fail closed.

Failure handling and safeguards

• Missing capabilities mean refuse privileged tools. Filter failure fails closed for those tools.
• Oversized retrieved chunks are dropped. Crashed workers must not resume the hop loop.

Verification and operations

• Replay threat-model.md fixtures in CI without executing privileged tools.
• Monitor prompt_injection_blocked; do not log raw secret material from filter hits.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• OWASP AI Agent Security Cheat Sheet
• OWASP LLM Prompt Injection Prevention Cheat Sheet