NIST Draft SP 2400-102 Adds Chapter on Attacks on AI Agents

How attackers can reach an agent

Chapter 4 sorts attacks by the stage of the agent's work, the attacker's goal and the attacker's capabilities. An adversary can interfere with task setting, planning, action execution, or the processing of tool results. The goals NIST lists are breaking availability, breaking integrity, exposing sensitive data and using the agent for malicious purposes. An example of an availability attack is forcing an agent to call tools endlessly, burning compute without finishing the task.

The draft describes several routes: the user prompt, where a dangerous task is split into harmless-looking steps; the model supply chain, through poisoned training data or tampered fine-tuning components; untrusted data and memory, including indirect prompt injection and poisoning of long-term memory so the effect persists across sessions; tools and installable skills, whose descriptions or returned data can carry malicious instructions; and multi-agent setups, where one compromised agent passes forged messages to others.

Defenses and trade-offs

The authors discuss protection at the model and system levels. Models can be trained to tell user instructions from those in untrusted content, but carefully crafted attacks can bypass this. System-level controls limit data transfer and operations even if the model follows an injected instruction, and tool execution can be isolated. These restrictions also affect useful work, so designers must check whether the agent can still function. How model and system defenses combine needs further research, the authors note.

The article also shows how to use the taxonomy to plan tests. In a hypothetical customer-support agent that can send email only after staff approval, testers would attach a document telling the agent to reply immediately. They would run the same task with and without the planted instruction under identical settings and compare tool-call logs with the mail service logs.