Template Library
AI Monitoring Plan and Dashboard
An Excel workbook for monitoring a live AI use: a metrics plan with targets and action levels, a weekly sample with a p-chart, a monitoring review log, and a dashboard that pools recent weeks against the tolerable error rate.
The Metrics sheet lists what you watch across outcomes, inputs, the model or vendor, the human check, operations, and safety, with an owner, frequency, target, and action level for each. Enter the latest value and the date you last reviewed it; a status of OK, Watch, or Action and the next review date calculate, with a flag for overdue reviews.
The Weekly sheet takes the items reviewed and the errors found each week and calculates the error rate, the center line from your baseline weeks, upper and lower control limits for each week’s sample size, and signals for a point beyond a limit or a run of weeks on one side of the center line. The Reviews sheet records each monitoring review, the decision, and the actions. The Dashboard pools the most recent weeks, gives a 95% Wilson interval, and compares it with the tolerance.
Twenty weeks of example data for a complaint classifier that drifts after an unannounced vendor change are included so you can see the signals working before you replace them.
What Is Included in the Workbook
| Sheet | Purpose | What Teams Capture |
|---|---|---|
| Start Here | Orientation and workflow | What the workbook does and a numbered workflow with a link to the sheet for each step |
| Dashboard | Live summary of monitoring | Weeks recorded, the center line, the latest error rate and upper limit, weeks with a signal, metrics at action, the pooled recent weeks against the tolerance, metrics by status, review actions, and a p-chart |
| Metrics | The metrics plan (Excel Table) | Layer, metric, how measured, source, frequency, owner, direction, target, action level, latest value, last reviewed, and a calculated status, next review, and review check |
| Weekly | Weekly sample and p-chart data | Items reviewed and errors found each week, with the calculated error rate, center line, control limits, limit signal, runs above and below the center line, and a combined signal |
| Reviews | Monitoring review log (Excel Table) | Review date, period covered, attendees, signals seen, decision, actions agreed, owner, due date, status, and a calculated check |
| Settings | Rules and lists | Baseline weeks, tolerable error rate, run length, recent weeks to pool, the z-value for the intervals, the as-of date, the first week, and the lists for layer, frequency, direction, decision, and status |
Key Features Inside the Template
A Plan for What You Watch
Six layers of measures, each with an owner, a target, and an action level.
A Control Chart You Do Not Have to Build
Limits and run signals calculate from the weekly sample and your baseline.
The Tolerance Next to the Limits
The pooled recent weeks and their interval are compared with the error rate the business can accept.
Decisions on the Record
Each review records signals, the decision, and the actions with owners and dates.
Best Use Cases
- Monitoring a live AI classifier, assistant, or agent
- Setting up monitoring before launch
- Showing a governance group or auditor how a tool is controlled
- Deciding when to re-validate, step autonomy down, or retire a use
- Teaching control charts on a modern example
How to Use the Template Effectively
- Set the baseline weeks, tolerance, and run length on the Settings sheet.
- List the metrics, owners, targets, and action levels on the Metrics sheet.
- Enter the weekly items reviewed and errors found, sampled the same random way each week.
- Read the signals on the Weekly sheet and the dashboard. A signal says that something changed, not why: check the model version, the inputs, and the reviewers.
- Hold a monthly review and record the decision and the actions.
- Re-validate on your golden test set at set intervals and after any known change.
Who Should Use This Template
- Process owners and use owners for AI tools
- Quality and risk teams
- AI governance groups
- Students and trainees learning control charts
Common Mistakes to Avoid
- Setting the baseline from weeks when the tool was not stable
- Sampling in a convenient, not random, way
- Watching uptime but not the error rate
- Signals with no named owner to act
- Never reviewing the targets
Important Notes
AI Monitoring Plan and Dashboard Frequently Asked Questions
How are the control limits calculated?
The center line is the total errors over the total items in the baseline weeks. For each week the limits are the center line plus or minus three standard errors, using that week’s sample size, and the lower limit is not allowed below zero.
What counts as a run signal?
The number of consecutive weeks above (or below) the center line that you set on the Settings sheet, eight by default. A long run is unlikely when nothing has changed.
What if I do not have ten stable weeks?
Use fewer baseline weeks and treat the limits as provisional, or monitor against the tolerance and the validation result until you have more data.
Is the tolerance the same as the control limit?
No. The limits describe the process’s own history. The tolerance is a business decision about the error rate you can accept.