Calculator Library / AI Operations
Paste your weekly review results and see whether the error rate of an AI tool has changed, with limits, run signals, and the pooled recent weeks against your tolerance.
Tool
Limit = p̄ ± 3√(p̄(1−p̄)/n)
Enter one line per week: the number of items reviewed and the number found wrong, separated by a comma, for example 120,7. If you review the same number each week, you can enter just the error count and set the sample size below. The first weeks set the center line and the limits. Load the example to see a classifier that drifts after a vendor change.
Everything runs in your browser. Nothing is sent to a server, and entries are not saved between visits. The example is made-up data. Intervals are 95% Wilson intervals.
Results
Picture
| Week | Reviewed | Errors | Error rate | Upper limit | Signal |
|---|
The monitor turns the results of your weekly sample of AI output into a p-chart. It sets the center line and the control limits from your first weeks, then tests every week for a point above the upper limit, a point below the lower limit, and a run of consecutive weeks on one side of the center line. It also pools the most recent weeks, gives a 95% Wilson interval, and compares it with your tolerable error rate.
It runs in your browser and does not send your data anywhere. It matches the method in the Monitoring AI in Production Guide; the AI Monitoring Plan and Dashboard and the Human Review Sampling Log hold the plan and the log.
| Signal | Rule |
|---|---|
| Center line | Total errors in the baseline weeks divided by total items reviewed in them |
| Control limits | Center line plus or minus three standard errors, using each week’s own sample size; the lower limit is not allowed below zero |
| Above the limit | A weekly error rate above that week’s upper limit |
| Run signal | The chosen number of consecutive weeks above (or below) the center line, eight by default |
| Pooled recent weeks | Errors and items over the last few weeks, with a Wilson 95% interval compared with the tolerance |
The pre-loaded data are 120 items a week for 20 weeks. The first 10 weeks give a center line of 5.75% and an upper limit of 12.13%. A vendor model change in week 11 is followed by a slow rise. A run of eight above the center line is complete in week 18; the first point above the limit is in week 20. The last four weeks have 51 errors in 480 items (10.6%), with a 95% interval of 8.2% to 13.7%, which reaches above the 10% tolerance. The data are made up.
It draws a p-chart of the weekly error rate in a sample of reviewed AI output, flags weeks above the limit and runs on one side of the center line, and compares the pooled recent weeks with your tolerable error rate using a Wilson 95% interval.
Eight consecutive points on one side of the center line is a common run rule in statistical process control, because it is unlikely when nothing has changed. It often signals a slow shift earlier than a point above the limit. You can change the run length.
At least ten if you can, while the tool was stable and the sample was random. Fewer weeks give less reliable limits.
Control limits show what variation to expect from sampling alone, given the baseline. The tolerance is what the business can accept. A tool can be stable and above the tolerance, or drifting and still within it.
No. It tells you that it changed and roughly when. Check the model or tool version, the input mix, and the reviewers, as the guide describes.