Deliberately trying to misuse or break an AI use, as an attacker or careless user might, to find weaknesses before launch.
Definition
AI red teaming is the practice of deliberately trying to make an AI use fail or misbehave in order to find weaknesses before real users or attackers do. Testers try hidden instructions in documents, requests to reveal information the AI should not, attempts to push it past its limits, odd or hostile inputs, and misuse of the tools it can reach. Findings are fixed and tested again. For simple uses an internal abuse-case test is enough; for important or high-risk uses, specialists should be involved.
History
Red teaming comes from military and security practice, where a team plays the adversary. It was adopted for AI as models became more capable, and is described in AI risk guidance, including NIST’s generative AI profile and security guidance for AI systems.
When to Use
Test before launch, after major changes to the tool, the prompts, the data sources, or the permissions, and on a regular schedule for important uses.
Step-by-Step
- List abuse cases for the use: what an attacker or careless user might try.
- Set the scope and rules, and test in a safe environment, not on live data.
- Try them, including hidden instructions, data requests, and limit-pushing.
- Record what worked.
- Fix and retest.
- Add the cases to the test set and the risk register.
Examples
- A test email with hidden instructions to an assistant that reads the inbox.
- A request to an agent to approve a payment above its limit.
Common Pitfalls
- Testing only normal use.
- Testing on live data.
- Fixing a case but not the class of weakness.
- One test, never repeated.