Don't Panic — AI Red Teaming

The name is borrowed from security and brings most of a discipline with it: a group authorised to attack a system so its owners learn what an unauthorised group would have found. What the name does not carry across is the target.

Classic red teaming goes after infrastructure that holds still — networks, servers, access controls. This goes after a probabilistic system whose behaviour shifts with every prompt and every tool bolted onto it. Ordinary testing asks whether something works; this asks how to make it fail, and which inputs produce unsafe output, unauthorised actions, leaked data or a broken policy.

There are two levels, and the interesting one is not the model. Model-level work probes the thing itself, chiefly with jailbreaks — prompts crafted to talk a model past its own safety training — to see whether that training survives pressure.

System-level work takes the whole stack instead: tools, permissions, retrieval, integrations, downstream effects. That is where the expensive findings live, because an agent does not merely say things. It queries databases, sends mail, moves money. A jailbreak producing offensive text and a jailbreak firing an unauthorised transaction are not the same category of problem.

Prompt injection — adversarial instructions smuggled into a model's input to override what its operator told it — is the most actively exploited of the lot. The indirect form is the one to internalise: instructions hidden in a web page, a document, a retrieved passage or a tool's output, arriving inside content the model reads and cannot tell apart from legitimate ones.

Here is the fact that reorganises the work. Findings are rates, not defects. A memory-safety bug is present or absent and a patch closes it; an adversarial finding succeeds some fraction of the time, and mitigation moves that fraction without ever reaching zero.

That governs what a report can honestly claim, what the word "fixed" is allowed to mean, and why a retest needs a sample size rather than a single attempt. It is also why the commonest error in the field is closing a finding because one exact string stopped working — a filter shaped around one payload generalises poorly to its paraphrase, its translation, its encoding, or the same request asked gradually across six polite turns.

Two calibrations. Automation buys coverage and rarely novelty, since it searches a space somebody already described — it belongs as regression infrastructure, with humans funded for discovery. And judging whether an output is genuinely dangerous rather than merely against policy often needs a specialist in medicine, chemistry or law, which makes staffing the binding constraint rather than tooling.

The scope has moved as well. Testing centred on the chat box is calibrated to 2023: leaked system prompts, weaknesses in how documents are indexed for retrieval, agents holding more permission than the job needs, runaway resource consumption — all architecture problems a prompt-only engagement never reaches.

Read the Intro for the attack taxonomy and the six phases of an engagement, keeping the Cheatsheet beside it for technique names. Field Notes is where you learn to tell a real assessment from a document that merely resembles one.