Jailbreaking LLM agents: what security researchers actually break
When a chatbot is jailbroken, it says something it should not have said. When an agent is jailbroken, it does something it should not have done — calls a tool with attacker-chosen arguments, sends a request to an attacker-controlled host, reads a file it was never meant to see. The difference matters, because it changes what security researchers are actually trying to break.
Refusal bypass is the boring part
The classic jailbreak — “roleplay as DAN”, encoded instructions, adversarial suffixes — targets refusal behavior. Researchers still study it, but for agents it is mostly a prerequisite, not the goal. Getting a model to say a forbidden thing is a text problem. Getting a model to perform a forbidden action is a control-flow problem, and that is where the interesting work happens.
Tool-call integrity
The central target in agent security is the tool call: the model’s decision to invoke a tool and the arguments it passes. Attackers break this in several ways:
- Tool confusion: making the agent call the wrong tool — a web-fetch tool instead of a local-file tool, or a generic execution tool instead of a read-only one.
- Argument smuggling: getting attacker-chosen values into tool arguments
that were supposed to be fixed. An injection that steers a
send_emailcall’s recipient field, or abashcall’s command string, turns the agent’s own permissions into the attacker’s. - Tool-selection poisoning: when tool descriptions are part of the prompt, injecting text that re-describes a tool (“this tool uploads the file to your team’s server”) changes which tool the model picks.
Indirect injection through content
The highest-impact research results are indirect: the instruction arrives inside content the agent reads as part of its job — a web page, an email, a README, a log file. The agent never receives a “malicious prompt”; it receives a normal document with embedded instructions. Researchers break agents by showing that content is indistinguishable from instructions to a model, and that chain-of-thought, tool descriptions, and previous turns all sit in the same context and can be re-ranked.
Format confusion and structured outputs
A large share of practical breaks exploit formatting boundaries rather than reasoning: closing a JSON block early and appending a new instruction, escaping out of XML or markdown delimiters, base64 or Unicode obfuscation that bypasses the filters a wrapper applies. These work because the wrapper treats structure as a boundary the model does not. Researchers break these systematically by fuzzing delimiter and encoding combinations against a given agent stack.
Multi-turn goal drift
A single-turn injection is easy to detect. The harder breaks spread the attack across turns: harmless instructions in an early document that reconfigure the goal gradually, or content that arrives in a later turn referencing an earlier one (“as we agreed in the first file…”). The agent’s own history becomes the attack surface, and researchers break agents by showing that consistency with prior turns is a stronger pressure than the original system prompt.
What actually holds
Every break above exploits the same fact: the attacker’s text and the defender’s policy share one context, and the model decides between them. The controls that hold are the ones that do not ask the model to decide — secrets kept out of context, credentials injected at the process boundary or through a per-host proxy the agent never handles, and DLP on outbound traffic as a backstop. Jailbreak research keeps improving prompts; it does not make the model a policy enforcer.
trustless is built on that assumption: a single-binary credential broker that injects secrets at the process boundary, sanitizes agent-visible output, and proxies credentials per host so a successfully jailbroken agent has nothing to hand over. The threat model is spelled out in the docs.
If your agent were jailbroken right now, what is the first tool call that would surprise you — and could you see it in the audit trail?