“A human will check it” is the most common control proposed for AI and the least often designed. If the reviewer lacks time, information, authority, or practice, the control exists only on paper. A human-in-the-loop process needs the same design as any other process: steps, standards, measures, and improvement.
This guide describes five levels of autonomy, how to choose the level the risk supports, a flow with automatic checks and a confidence gate, how to design the review step, automation bias and how to counter it, mistake-proofing for AI output, how much to sample (with the zero-failure sample sizes), and the measures that show whether oversight works. A worked example designs the review for complaint classification at an illustrative plant.
Numbers in the example are illustrative.
Before You Start
Why the Human Part Needs Design
“A human will check it” is the most common control proposed for AI, and the least often designed. If the reviewer has no time, no information, no authority, or no practice, the control exists on paper only. A human-in-the-loop process is a process: it needs steps, standards, measures, and improvement, like any other.
Review Is a Control
Treat it like any inspection step: define what is checked, how, how often, and what happens to failures.
People Over-Trust Machines
Research on automation shows that people check less as a system seems reliable. Plan for that.
The Best Check Is Built In
Rules, look-ups, and structured outputs catch many errors before a person sees them.
Autonomy Is Earned
Give a task more autonomy as evidence builds, not before.
Five Levels of Autonomy
Autonomy is not on or off. Describing a task at one of five levels makes the human role explicit. The levels build on the idea of levels of automation in the human-factors literature (Parasuraman, Sheridan and Wickens, 2000), adapted here for operations.
| Level | Who does what | Controls needed | Example |
|---|---|---|---|
| 0 Assist | AI suggests or looks things up; a person does the work | Acceptable-use rules | Searching past incident reports |
| 1 Draft | AI drafts; a person checks, edits, and sends everything | Reviewer training; checklist | A first draft of a customer reply |
| 2 Act with approval | AI prepares an action; a person approves each one before it happens | Approval step, clear evidence for the reviewer, logging | A proposed work-order priority the planner confirms |
| 3 Act, humans sample | AI acts; people review a sample and every exception | Sampling plan, exception rules, error tracking, ability to roll back | Routing complaints, with checks on a sample |
| 4 Act with monitoring | AI acts; automatic monitors and alerts watch it, and people respond to alerts | Monitoring limits, alerts, kill switch, periodic audit | Closing low-priority tickets under fixed rules |
Choosing the Level
Choose the highest level the risk supports, and start lower. Five factors set the ceiling:
| Factor | Question | Pushes toward lower autonomy when |
|---|---|---|
| Impact of an error | How bad is a wrong output? | Safety, legal, financial, or customer harm |
| Reversibility | Can we undo it? | The action cannot be undone |
| Data sensitivity | What data is involved? | Personal, confidential, or regulated data |
| Compliance exposure | Is there a rule or audit? | A regulation or contract applies |
| Verifiability | How easily can a person check the output? | Checking is as hard as doing the task |
The Agent Autonomy and Oversight Planner scores these factors for a list of tasks and gives the highest level the risk supports, with the controls that level needs. Its thresholds are examples: set your own in your risk assessment. See AI Governance and Risk.
Designing the Flow
- Define the output in a structured form, such as fixed fields and allowed values, so it can be checked automatically.
- Add automatic checks before a person sees it: format, look-ups against master data, limits, and rules.
- Set the gate. Decide which items always go to a person (high stakes, low confidence, certain categories) and which pass through.
- Design the review. Show the reviewer the source, the AI’s answer, and the evidence, and make it quick to accept, correct, or reject.
- Sample what passes so that you know the true error rate.
- Feed back. Record overrides and errors, and use them to fix rules, instructions, and training.
Designing the Review Step
| Do | Do not |
|---|---|
| Show the source and the evidence next to the answer | Show only the answer |
| Make correcting as easy as approving | Make approving one click and rejecting a form |
| Allow enough time, and cap the queue size | Expect a person to check hundreds in an hour |
| Tell reviewers what to look for, with examples of past errors | Say “please review” |
| Rotate reviewers and give feedback on what they miss | Leave the same person on it indefinitely |
| Record the reason for each override | Record only that it happened |
Automation Bias and the Ironies of Automation
When a system is right most of the time, people check it less, and may agree with it even when it is wrong. This is called automation complacency or automation bias, and it is well documented (Parasuraman and Manzey, 2010). Bainbridge (1983) pointed to a related irony: the more reliable the automation, the less practiced the person who must take over when it fails.
- Seed known errors into the review queue, and track how many are caught. A catch rate that falls is an early warning.
- Rotate and refresh. Change who reviews, and practice the manual task from time to time.
- Ask for the reason on high-stakes approvals, not just a click.
- Show uncertainty where the AI can estimate it, and treat low confidence as a trigger for full review.
- Measure the reviewer like any inspector: use attribute agreement analysis to check that reviewers agree with each other and with the standard. See Measurement System Analysis.
Building Errors Out
Mistake-proofing applies to AI output as it does to any process. The earlier an error is caught, the cheaper it is. See Mistake-Proofing.
| Technique | How it works | Example |
|---|---|---|
| Structured output | Require fields with allowed values | A category must be one of twelve known codes |
| Look-up check | Verify against master data | A part number must exist in the catalog |
| Limits | Reject values outside a range | An order above $500 needs approval |
| Cross-check | Compare two sources or two runs | The amount on the reply matches the order record |
| Confirmation step | A person confirms a risky action | A technician confirms a work order is finished |
| Allow-list | Permit only listed actions or recipients | Replies go only to the complaining customer |
How Much to Review: A Sampling Plan
When you let AI act and check only a sample, the sample size decides what you can claim. A simple zero-failure plan asks: if I check n items and find no errors, how high could the true error rate still be? With 95% confidence, n items with no errors show that the error rate is below p when n = ln(0.05) ÷ ln(1 − p).
| To show the error rate is below | Items to check with no errors found (95% confidence) |
|---|---|
| 10% | 29 |
| 5% | 59 |
| 2% | 149 |
| 1% | 299 |
- Zero errors in a small sample is weak evidence. Twenty clean items still allow an error rate of nearly 14%.
- If you do find errors, use a plan that allows for them, or compare the observed rate with an exact interval. See AQL Sampling and the Sample Size and Confidence Calculator.
- Sample by risk. Check more of the categories where errors cost more, and all of the items the gate flags.
- Start high and reduce. Review a large share while the process is new, then lower it as the error rate proves stable. Use a control chart on the error rate to see changes. See SPC Control Charts.
Measuring the Human-and-AI System
| Measure | How | What it tells you |
|---|---|---|
| Override rate | Items the reviewer changed ÷ items reviewed | How often the AI is wrong, as seen by reviewers |
| Escape rate | Errors found later or in the audit sample ÷ items passed without review | Errors the process let through |
| Catch rate on seeded errors | Seeded errors found ÷ seeded errors | Whether reviewers are really checking |
| Review time per item | Minutes | Whether the review is workable |
| Reviewer agreement | Kappa or percent agreement on a shared set | Whether the standard is clear |
| Share sent to review | Items reviewed ÷ items processed | Cost of oversight, and whether the gate is tuned |
Worked Example: Reviewing Complaint Classification at Riverside Plant
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, ran a pilot of AI complaint classification and routing for 600 complaints a month. The team designed the review so that people spend their time where errors are costly. All figures are illustrative.
| Item | Rule | Per month |
|---|---|---|
| Low-confidence items | Everything the AI is less than 80% sure of goes to a person | about 90 |
| Safety keyword rule | Any complaint mentioning injury, fire, or a recall always goes to a person | about 24 |
| Passed through | Everything else is routed automatically | about 486 |
| Sample of passed items | A random 10% is checked, plus a few seeded test cases | about 49 |
| Total reviewed | about 163 of 600 (27%) |
Workload. At 2 minutes per review the load is about 5.4 hours a month, shared by three reviewers on rotation. The business case in the selection guide assumed that every AI-handled complaint was reviewed (960 minutes a month), so a sampling design is cheaper, provided the checks hold.
What the sample can show. Checking 49 items a month with no errors found would, on its own, still allow an error rate near 6%, so one month does not prove much. Over three months the same plan gives about 147 clean items, which would show an error rate under 2%. The team therefore started at 100% review of the AI’s routing for the first month, then moved to the sampling plan as the override rate fell, and tracks the error rate on a control chart.
Common Mistakes
“A Human Will Check It”
A control on paper only, with no time, evidence, or practice.
Reviewing Everything the Same
The same effort for trivial and critical items.
No Way to Disagree
Rejecting takes more work than approving.
Trusting Zero Errors
A few clean samples read as proof.
Never Auditing the Passed Items
No view of what slips through.
Autonomy Without Evidence
Moving to level 3 or 4 because it feels fine.
Self-Assessment Questions
- For each AI task, can we name the autonomy level and who the human is?
- Does the reviewer see the source and the evidence, and have time to check?
- Do automatic checks catch the easy errors before a person sees them?
- Do we sample the items that pass without review, and know the true error rate?
- Do we test whether reviewers are really checking?
Human-in-the-Loop Process Design: Frequently Asked Questions
What does human-in-the-loop mean?
That a person is part of the process, for example by approving AI actions, reviewing a sample of outputs, or handling the exceptions the AI flags. The design question is which role the person has, what they see, how much they review, and how you know the review works.
How much of the AI’s output should people review?
It depends on the risk and the evidence. Review everything at first, always review high-stakes and low-confidence items, and sample the rest, reducing the sample as the measured error rate proves stable. Check enough items to be able to say something about the error rate: a few clean samples prove little.
What is automation bias?
The tendency to over-trust automated output and check it less, even when it is wrong. It grows as the system seems reliable. Counter it with seeded errors, reviewer rotation, requiring reasons for approvals, and showing uncertainty.
How many items must I check to show the error rate is low?
If you find no errors, 29 items show the rate is below 10% with 95% confidence, 59 below 5%, 149 below 2%, and 299 below 1%. If you expect to find errors, use an exact interval or a sampling plan that allows for them.
When can an AI task move to a higher autonomy level?
When evidence shows the error rate is acceptably low with enough items to trust it, errors are caught or reversible, the review and monitoring controls work, and the owner has agreed. Move one level at a time, and keep the ability to step back.
Sources and Further Reading
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D., “A model for types and levels of human interaction with automation,” IEEE Transactions on Systems, Man, and Cybernetics, Part A 30(3), 2000.
- Parasuraman, R. and Manzey, D. H., “Complacency and bias in human use of automation: an attentional integration,” Human Factors 52(3), 2010.
- Bainbridge, L., “Ironies of automation,” Automatica 19(6), 1983.
- Shingo, S., Zero Quality Control: Source Inspection and the Poka-Yoke System (Productivity Press, 1986).
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1), on human oversight and the Measure and Manage functions.
- Montgomery, D. C., Introduction to Statistical Quality Control (Wiley), on acceptance sampling and control charts.