A pilot shows that an AI tool worked on the cases you tested, in the weeks you tested them. It does not show that the tool will keep working. Data, processes, and users change, and a purchased tool can change when its vendor changes the model. Monitoring is the control plan for an AI use.
This guide lists what to watch across outcomes, inputs, the model or vendor, the human check, operations, and safety; shows how to chart the sampled error rate on a p-chart; explains why to compare results with the tolerable error rate and not only with control limits; sets out triggers and responses; and says who reviews what and when. A worked example shows slow drift after an unannounced model change.
It does not cover how to build monitoring software. Example data are illustrative.
Before You Start
An AI Tool Is Not Finished When It Goes Live
A pilot proves that a tool worked on the cases you tested, in the weeks you tested them. It does not prove the tool will keep working. The data changes, the process changes, people change how they use it, and with a purchased tool the vendor can change the model underneath you, sometimes without telling you. Quality people know this pattern: a process that was capable at launch drifts unless someone watches it. The control plan for an AI use is monitoring.
Drift Is Normal
Inputs, rules, products, and the vendor’s model all change over time. Plan for it rather than hope against it.
Watch the Outcome, Not Only the Uptime
A tool can be up and fast and wrong. Measure the error rate in a checked sample, not only availability.
Decide the Response in Advance
Write down what result triggers what action, before anyone is under pressure.
Name an Owner
Monitoring that belongs to everyone belongs to no one.
What to Monitor
| Layer | What to watch | How | How often |
|---|---|---|---|
| Outcomes | Error rate in a sample of checked outputs, critical errors, seeded-error catch rate | The Human Review Sampling Log and a p-chart | Weekly chart, monthly review |
| Inputs | Volume, mix of categories, new terms or products, share of missing fields | Counts and a simple comparison with the validation period | Weekly |
| The model or vendor | Model and version in use, vendor change notices, outages | A version field in the log; a named person who reads the notices | Every release and at least monthly |
| The human check | Override rate, review time, share approved without comment | Review log; look for approval that is too fast or too uniform (automation bias) | Monthly |
| Operations | Cost per item, response time, escalations | System reports | Monthly |
| Safety and security | Incidents, near misses, data-handling breaches | The AI incident log | As they happen; reviewed monthly |
A Control Chart for the Error Rate
The error rate in a weekly sample is a proportion, so the right chart is a p-chart. Use the first stable weeks after launch to set the center line and the limits, then keep adding points. The limits show how much week-to-week wobble to expect from sampling alone: a point beyond them, or a long run on one side of the center line, is a signal that something changed. See the SPC Control Charts Guide for the method.
Compare With the Tolerance, Not Only the Limits
Control limits describe the process’s own history. The tolerable error rate from your plan is a different line: it is what the business can accept. In the example it is 10%. In the last four weeks, 51 of 480 sampled items were wrong, which is 10.6%, with a 95% interval of 8.2% to 13.7%. The interval straddles the tolerance, so the data cannot yet prove that the tool is over it, nor can anyone claim it is safely under it. Both facts matter: the signal says “investigate now”, and the interval says “do not wait for certainty before acting”.
Make the response rules before launch, and link them to the autonomy level from the Human-in-the-Loop Process Design Guide.
Triggers and Responses
| Trigger | Response |
|---|---|
| A point above the upper limit, or a run of eight on one side of the center line | Investigate within the week: check the model version, the inputs, and the reviewers. Do not wait for the next monthly review |
| The upper end of the interval for the last four weeks is above the tolerance | Raise the sampling rate and step autonomy down one level until the cause is known |
| Any critical error | Treat as an incident (see the AI Incidents and Failure Response Guide) |
| The vendor announces or makes a model change | Re-run the golden test set before the change reaches live work, if you can; if not, re-run it at once and watch the next two weeks |
| A change to the process, the products, the forms, or the data feeding the tool | Treat it as a change: assess, test, and approve (see Management of Change) |
| Scheduled re-validation (for example every six months) | Re-test on a fresh set of real cases, including new awkward ones, and compare with the last result |
| Results stay worse than the manual process | Retire or redesign the use. Say so openly; retiring a tool is a success of the process, not a failure |
Who Does What, and When
- Daily: reviewers record results as part of the work; anyone can flag an odd output.
- Weekly: the use owner reads the p-chart and the input counts, and logs any signal.
- Monthly: a short review of outcomes, overrides, incidents, vendor notices, and cost, with a decision recorded.
- At each change and at set intervals: re-validation on the golden test set, signed off by the owner.
- Yearly: a look at the whole portfolio: which uses to continue, scale, change, or retire.
Common Mistakes
Monitoring Only Uptime
The tool is available and wrong.
Limits Set Too Late
Starting the chart after the problems have begun gives limits that include the drift.
No Version Field
Without the model version in the log, a change in results cannot be tied to a vendor change.
Sampling Stops After Launch
Sampling is cut to save time just when the tool is least understood.
Alarms Nobody Owns
A signal with no named person to act on it.
Never Retiring Anything
Tools that no longer pay for themselves stay on.
Self-Assessment Questions
- For each live AI use, do we measure the error rate in a checked sample, and chart it?
- Do we log the model or tool version with the results?
- Do we have written triggers and responses, and a named person for each use?
- Do we re-test on a golden test set at set intervals and after any known change?
- Have we ever retired or scaled back an AI use because the evidence told us to?
Monitoring AI in Production: Frequently Asked Questions
Why do AI tools get worse after launch?
Because the world they work in changes. The mix of inputs shifts, products and forms change, people adapt how they use the tool, and with a purchased tool the vendor may update the model. Without monitoring, the first sign is often a customer complaint.
How often should I sample AI output?
Often enough to see a problem soon after it starts, and more often early in a rollout. A fixed number each week, such as 100 to 150 items, is easy to chart. Raise the rate after any change or signal, and reduce it only when the evidence supports it.
What is the difference between control limits and the tolerable error rate?
Control limits come from the process’s own history and show what variation to expect from sampling alone. The tolerable error rate comes from the business and shows what error you can accept. A tool can be stable and still above the tolerance, or drifting and still below it.
Do I need to re-validate if nothing seems wrong?
Yes, at set intervals and after known changes. Re-test on fresh real cases, including new awkward ones, and compare with the earlier result. Treat it like calibration.
What if the vendor will not tell me when the model changes?
Record the model version where you can, pin a version if the vendor allows it, ask for change notices in the contract, and rely on your own sampled error rate. See the guide on buying and evaluating AI tools.
Sources and Further Reading
- National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023), on measuring and managing risk over the life of a system.
- ISO/IEC 42001:2023, Information technology: Artificial intelligence: Management system, on monitoring, measurement, and performance evaluation (check the current edition).
- Montgomery, D. C., Introduction to Statistical Quality Control, Wiley, for p-charts and run rules.
- Western Electric Company, Statistical Quality Control Handbook (1956), and Nelson, L. S., “The Shewhart control chart: tests for special causes,” Journal of Quality Technology 16 (1984), for run rules.
- Wilson, E. B., “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association 22 (1927), for the interval on a proportion.