Every control lowers the chance of an AI failure; none removes it. What decides the harm is how fast you notice, how fast you can stop the AI, and whether you learn. Organizations that run incident processes for safety, quality, and IT already have most of what they need.
This guide defines what counts as an AI incident, gives four severity levels with responses, sets out a five-step flow (detect, contain, assess, fix, learn and report), lists containment tools such as a kill switch, revoked credentials, a manual fallback, and a pinned version, and walks through a worked example of a purchasing agent that sent duplicate orders.
Reporting duties vary by place and sector. Check what applies to you. Example data are illustrative.
Before You Start
Plan for the Day It Goes Wrong
Every control in this hub lowers the chance of an AI failure; none removes it. What decides the harm is how fast you notice, how fast you can stop it, and whether you learn. Organizations that run incident processes for safety, quality, and IT already have most of what they need. The additions for AI are a clear idea of what counts as an AI incident, a way to stop an AI use quickly, and attention to causes that are new: a changed vendor model, injected instructions, or an agent that exceeded its limits.
Define What Counts
If nobody knows that a wrong output or a data leak to a public tool is reportable, nobody reports it.
Build the Stop Button First
A kill switch, a manual fallback, and a way to revoke access must exist before launch.
Make Reporting Safe
People report what they believe will be treated as learning. See Safety Culture and Human Performance.
Learn From Near Misses
A wrong output caught by a reviewer is a free lesson about the control that worked.
What Counts as an AI Incident
- An AI output that was wrong and reached a customer, a supplier, a regulator, or a record that matters.
- Personal, confidential, or restricted data entered into a tool that is not approved for it, or shown to someone who should not see it.
- An agent that acted outside its permissions, limits, or approval gates, or that repeated an action.
- Instructions hidden in a document, web page, or message that changed the AI’s behavior (prompt injection).
- A sudden change in results after a vendor update, or a monitoring signal that is not explained.
- A complaint or concern that an AI use treated people unfairly.
- A long outage of a tool that a process depends on, if the fallback failed.
Sort by Severity
| Level | Examples | Response |
|---|---|---|
| S1 Critical | Harm to a person, a safety system affected, a serious customer or legal impact, or a large data exposure | Stop the AI use at once. Notify the owner, safety, and security leads within the hour. Report as your incident policy and the law require |
| S2 High | A wrong output reached a customer, a supplier, or a record that matters; sensitive data went to an unapproved tool; the AI took an action outside its limits | Contain the same day. Owner and risk lead review. Decide whether to suspend |
| S3 Moderate | An error caught before it left the team, a pattern of errors above the tolerance, or a service outage that people worked around | Contain within the week. Add to the AI risk register and the monitoring review |
| S4 Minor | A single wrong output that a reviewer caught, or a near miss with no effect | Log it. Review the log monthly for patterns |
Adapt the levels to your own risk scale, such as the severity scale in the AI Risk Register. Check what your contracts, your regulators, and the law require for reporting; for example, data breach notification laws and sector rules set deadlines, and they vary by place.
Five Steps
Contain first, understand second. The first question is not “why” but “how do we stop more harm?” Containment tools worth having ready:
- A kill switch that a named person on every shift can use without asking for permission.
- The ability to revoke the AI’s credentials or tokens at once.
- A manual fallback that people have practiced.
- A way to pin or roll back to a known model version.
- A way to quarantine outputs produced during the problem period, so they can be re-checked.
- A contact list for owner, safety, security, legal, communications, and the vendor.
Worked Example: The Duplicate Orders
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, ran a purchasing agent at autonomy level 2: it prepared purchase-order emails and a buyer approved each one. After a change meant to save the buyers time, routine orders under a set value were sent without approval. A network timeout then made the agent retry. It sent 14 duplicate order emails to one supplier before a limit on messages per hour tripped an alert. The example is illustrative.
| Time | Event |
|---|---|
| 0:00 | Timeout in the supplier connection; the agent starts to retry |
| 0:09 | Alert: messages to one supplier above the hourly limit. The on-duty buyer sees it |
| 0:12 | Buyer uses the kill switch; the agent’s send permission is revoked. Duplicates stop at 14 |
| 0:40 | Buyer phones the supplier, who agrees to ignore the duplicates. No goods were shipped. Severity set at S2 |
| Next day | Blameless postmortem. Causes: no check for an order already sent (no idempotency check); the approval step removed for low-value orders without a risk review; the retry rule had no cap |
| Within the week | Fixes: a unique order key checked before sending; a cap of one retry; approval restored above a lower value; the change added to the risk register and the monitoring plan |
Learn Without Blame
Use the method in the Incident Management and Postmortems Guide and the Incident Investigation and Near-Miss Reporting Guide. Ask what the system allowed, not who erred. Update the AI Risk Register, the guardrails in the Agent Charter and Runbook, and the monitoring plan. Track the actions to completion. Count near misses, because they are the cheapest data you will get.
Common Mistakes
No Kill Switch
The only way to stop the AI is to find the developer.
Unclear Reporting
People do not know that a data leak to a chat tool is an incident.
Blame
Reports dry up and the pattern stays hidden.
Fixing the Output, Not the Cause
The wrong answer is corrected and the control that failed is not.
No Practice
The first test of the fallback is the real failure.
Changes Without Review
A convenient change to a control removes a safeguard.
Self-Assessment Questions
- Do people know what counts as an AI incident and how to report it?
- Can a named person on every shift stop each AI use at once?
- Have we practiced the manual fallback?
- Do we know our reporting duties to customers, regulators, and others?
- Do postmortems change controls, and do we track the actions?
AI Incidents and Failure Response: Frequently Asked Questions
What is an AI incident?
An event in which an AI use causes or nearly causes harm or a breach of rules: a wrong output that reached a customer or a record, data entered into an unapproved tool, an agent acting outside its limits, injected instructions that changed behavior, or an unexplained change in results.
What is a kill switch for an AI use?
A fast, simple way to stop the AI from acting, such as revoking its credentials or switching a flag, that a named person on every shift is allowed to use without waiting for approval. It should be tested, and a manual fallback should be ready.
Do we have to report AI incidents to a regulator?
It depends on the sector, the data, and the place. Data breach notification laws, product and safety rules, and contracts can all set duties and deadlines. Ask your legal and compliance leads, and write the answers into the playbook.
Should near misses be logged?
Yes. A wrong output caught by a reviewer shows that a control worked and what the AI gets wrong. Counting them also shows patterns before they become incidents.
Who should run the postmortem?
Someone trained to facilitate, who is not blamed and does not own the system, with the people who know the work. Focus on what the system allowed and track the actions to completion.
Sources and Further Reading
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1, 2023) and Generative AI Profile (NIST AI 600-1, 2024), on incident disclosure and response as part of managing AI risk.
- National Institute of Standards and Technology, Computer Security Incident Handling Guide, SP 800-61 (check the current revision), for a general incident-handling lifecycle.
- Google, Site Reliability Engineering (O’Reilly, 2016), chapter on postmortem culture, for blameless postmortems.
- OWASP, Top 10 for Large Language Model Applications (check the current version), for prompt injection and excessive agency.
- ISO/IEC 42001:2023, on managing AI system incidents and nonconformities (check the current edition).