An event in which the use of AI causes, or nearly causes, harm or a breach of rules, such as a wrong output reaching a customer or an agent acting outside its limits.
Definition
An AI incident is an event in which an AI use causes harm, or nearly does, or breaks a rule or limit. Examples are a wrong output that reaches a customer or a record, confidential data entered into an unapproved tool, an agent that acts beyond its permissions or repeats an action, hidden instructions that change an AI’s behavior, an unexplained change in results after a vendor update, and a concern that a use treated people unfairly. A near miss, such as a wrong output that a reviewer caught, is also worth recording.
History
Incident management comes from safety, quality, and IT operations. Public collections of AI incidents began to appear in the 2020s, and AI risk frameworks, including the NIST AI Risk Management Framework, treat incident response and disclosure as part of managing risk over the life of a system.
When to Use
Define what counts as an AI incident before launch, and use the definition for training, reporting, and review. Apply it to any AI use, from a drafting assistant to an agent that acts.
Step-by-Step
- Define the term and the severity levels, and tell everyone how to report.
- Contain first: pause the use, revoke access, switch to the manual fallback, or pin a version.
- Assess what happened, how bad it was, and who is affected, and preserve logs.
- Notify the owner and the right leaders, and decide whether outside parties must be told.
- Fix the cause, repair the damage, and test before restart.
- Learn without blame: run a postmortem, update the controls, and track actions.
Examples
- Duplicate orders: an agent retries after a timeout and sends 14 order emails; a rate limit raises the alert and a kill switch stops it.
- Data in a public tool: an engineer pastes a customer drawing into a public chatbot; a colleague reports it.
- Silent drift: a vendor changes the model and the error rate rises over weeks.
Common Pitfalls
- No kill switch.
- Unclear definition: people do not know what to report.
- Blame: reports dry up.
- Fixing the output only, not the control that failed.
- Unpracticed fallback.