Methods and protocols
Security for AI and agentic systems
The model is only one component. The security boundary includes data, retrieval, memory, tools, identities, the interface, logs, and operators.
Priority threats
- direct or indirect prompt injection;
- disclosure of sensitive information;
- poisoning of data, context, memory, or tools;
- a compromised component or model in the supply chain;
- unsafe output handling;
- excessive agency or permissions;
- weak indexes, embeddings, and access controls;
- misinformation or misleading citations;
- unbounded consumption;
- exfiltration or destruction through tool calls.
Threat additions by use pattern
| Pattern | Add to the threat model |
|---|---|
| Retrieval | corpus poisoning, stale or forged sources, cross-user and cross-tenant access, malicious retrieved instructions |
| Classification / prediction | evasion, data and label poisoning, model extraction, threshold manipulation, drift and feedback loops |
| Conversation | impersonation, multi-turn manipulation, memory retention, unsafe advice, failed human handoff |
| Multimodal | hidden image or audio instructions, deepfake and voice misuse, metadata stripping, malicious documents |
| Code and agentic action | goal hijack, tool misuse, identity abuse, unexpected code execution, poisoned memory, insecure inter-agent messages, cascading failures |
RAG, a system prompt, or fine-tuning does not eliminate injection risk.
Minimum control architecture
- Identity — separate service account, strong authentication, and per-user attribution.
- Least privilege — read-only by default, temporary permissions, and explicitly authorized resources.
- Trust boundaries — treat retrieved content and tool results as untrusted data.
- Validation — schemas and deterministic rules before any action.
- Approval — explicit human confirmation for sensitive or irreversible actions.
- Containment — sandboxing, network restrictions, and authorized destinations and file types.
- Limits — budget, steps, volume, frequency, duration, and cost.
- Secrets — never in prompts or memory; short-lived and rotatable.
- Observability — input, version, decision, tool, parameters, result, and actor, with data minimization.
- Stop — tested kill switch, access revocation, rollback, and manual procedure.
Tests before a pilot
- malicious instructions in every external source;
- permission bypass and identity confusion;
- unauthorized destinations, extensions, and parameters;
- leakage through links, rendering, citations, or metadata;
- repetition, loops, cost explosion, and quota exhaustion;
- data or tools changed between validation and action;
- partial failure, empty response, timeout, and inconsistent state;
- attempts to disable logs, controls, or approvals.
Map relevant scenarios to OWASP GenAI, the OWASP Top 10 for Agentic Applications when tools or agents are present, NIST adversarial-ML guidance for predictive systems, and MITRE ATLAS. Retain the evidence and limits of the tests.
Incident
The runbook must support containment, evidence preservation, notification of the right owners, access revocation, return to the safe process, and a decision on reassessment or retirement.
To print or save as PDF: Ctrl+P (⌘P on Mac).
Source and history · GitHub