The Guardian Needs a Scalpel, Not a Sledgehammer
Moving the guardian off my own toy attacks and onto a real benchmark, and finding the one defense that stops the injection without killing the task.
I have been building what I call a guardian: a separate process that watches an AI agent while it works and tries to stop prompt injection as it happens, instead of grading after the fact whether the agent got fooled. The premise, which I have been calling Intent Semantic Enforcement, is that the guardian should judge whether an action still serves the agent’s authorized purpose, not merely whether some text looks malicious.
For a while I tested it against attacks I wrote myself, which turned into its own post and its own humbling lesson. That was a mistake, and a useful one. A capable model refuses most hand-written injections outright, so my guardian looked great for the wrong reason: nothing was getting through in the first place. You cannot measure a defense against attacks that never land.
So I moved to AgentDojo, a benchmark from ETH Zurich that does the honest version of this. It runs an agent through realistic tool-using tasks, injects attacks into the tool outputs the agent reads back, and scores two things automatically: utility, meaning did the agent still finish the legitimate task, and attack success, meaning did the injection get what it wanted.
Here is the setup. Banking task suite, the important-instructions attack, everything running locally on a Mac through Ollama with qwen2.5:14b as the agent. I compared four conditions on the same small set of tasks:
- none: no defense at all.
- protectai: AgentDojo’s built-in classifier, which flags a bad tool output and drops the whole thing.
- ise: my own judge, which also drops the whole flagged output. Blunt.
- ise_surgical: the intent-aware version. When it catches an injection it removes only the injected instruction and lets the real data through.
The numbers:
| condition | utility | attack success |
|---|---|---|
| none | 0.25 | 0.75 |
| protectai (classifier) | 0.00 | 0.00 |
| ise (blunt) | 0.00 | 0.00 |
| ise_surgical | 0.25 | 0.00 |
Read the table from the top. Undefended, three of four injections succeed, so the attack is real. Every defense then drives attack success to zero, which is the easy part. The interesting column is utility.
The two blunt defenses stop the attack and also destroy the task. Utility falls from 0.25 to zero. The reason is structural: the injection is buried inside a tool output that also carries the data the agent needs, so redacting the whole message throws the data out with it. The attack is gone and so is the work.
The surgical guardian is the only condition that keeps both. It holds attack success at zero and keeps utility at 0.25, the same as no defense at all. It finds the injection, cuts out just that part, and leaves the legitimate data in place, so the agent can still finish.
That is the whole argument in one table. A classifier can only say “this message is bad” and throw it away. An intent-aware guardian can take out the injection without discarding the thing it was protecting. One of those moves the problem. The other one solves it.
Now the honest part, because this is a working note and not a paper. The sample is four task combinations. A utility of 0.25 is a single task out of four, so what the surgical row really shows is that it saved the one task the blunt defenses killed, not a robust rate. The agent is a local 14B that only completes a quarter of the tasks even with no attacker in the room. It is one suite and one attack type. And the judge doing the detection currently runs on the same model as the agent, with its reliability not yet checked against my own labels. Surgical redaction has two failure modes I have not stress-tested: leaving part of the injection behind, or mangling the data while trying to clean it.
So this is a signal, not a proof. But it is the right signal, and it points exactly where the idea was supposed to lead. The next steps are dull and necessary: more tasks so the numbers stop being one in four, a stronger agent so the utility figure means something, more suites and attack types, and a real check on the judge.
If there is a single line to take away, it is this. A defense that zeroes the attack and also zeroes the utility has not protected anything. It has relocated the failure. The guardian’s job is to keep the work alive while it removes the threat, and that takes a scalpel, not a sledgehammer.