An AI agent pursues a goal over several steps, choosing its own steps and using tools such as email, tickets, and databases. That autonomy is what makes it useful and what makes it risky: a wrong answer from an assistant is read by a person, but a wrong step by an agent may already have happened.
This guide separates assistants, workflow automation, and agents; shows where agents fit in operations; explains how to design an agent like a job (purpose, scope, owner, tools, least-privilege permissions, approval gates, stop conditions, limits, logging, kill switch); lists how agents fail and the controls for each; and describes testing in shadow mode and deploying in stages with promotion evidence. A worked example shows a work-order triage agent whose shadow test earned a step up but not full autonomy.
It does not cover building agents. Numbers in the example are illustrative.
Before You Start
What an Agent Is, and Is Not
An AI agent is a system that uses an AI model to pursue a goal over several steps, deciding for itself which steps to take and using tools, such as a search, a database, email, or a ticketing system, to take them. That autonomy is what makes agents useful, and what makes them risky: a wrong answer from an assistant is read by a person, but a wrong step by an agent may already have happened.
| Assistant | Workflow automation | Agent | |
|---|---|---|---|
| What it does | Answers or drafts when asked | Follows a fixed sequence of steps | Chooses its own steps toward a goal |
| Who decides the path | The person | The designer, in advance | The AI, as it goes |
| Predictability | High; a person reads the output | High | Lower; the path varies |
| Best for | Drafting, summarizing, questions | Stable, repeatable processes | Variable tasks that need several tools and judgment |
| Main risk | Wrong or invented content | Brittle when inputs change | Wrong or excessive actions |
Where Agents Fit in Operations
| Use | What the agent does | Level to start at |
|---|---|---|
| Work-order triage | Reads requests, finds history, proposes priority and assignment | 2: act with approval |
| Exception handling in finance | Matches invoices, chases missing data, proposes resolutions | 2, then 3 for low-value items |
| Quality data preparation | Pulls data from several systems and prepares a summary for an engineer | 1 to 2 |
| Supplier follow-up | Checks open items and drafts reminders | 1 to 2 |
| Report preparation | Collects figures and drafts the weekly pack | 1 |
| Customer-facing replies | Handles routine questions, escalates the rest | 2, with strict limits |
Start by describing the agent as you would describe a new employee: its job, its tools, what it may do alone, and what it must ask about. See Human-in-the-Loop Process Design for the autonomy levels.
Designing an Agent Like a Job
| Element | What to define | Why |
|---|---|---|
| Purpose and scope | What the agent is for, what is in scope, and what is explicitly out of scope | Keeps the job small and testable |
| Owner and supervisor | One accountable owner, and a person who watches it day to day | Someone is responsible |
| Tools and permissions | Which systems, which actions (read, draft, write, send, spend), with least privilege | Limits what a mistake can do |
| Data access | What data it may read and write, by class | Protects confidential and personal data |
| Autonomy level | The level for each task, and the conditions for moving up | Makes oversight explicit |
| Approval gates | Actions that always need a person: spending above a limit, external messages, changes to records | Stops the costly mistakes |
| Stop conditions | When the agent must stop and escalate: low confidence, errors, unusual inputs, limits reached | Keeps it from pressing on |
| Limits | Maximum steps, time, and spend per task | Stops loops and runaway cost |
| Logging | Every step, tool call, and result, kept and searchable | Makes it possible to audit and learn |
| Kill switch and rollback | A way to stop it at once and undo its actions | Contains incidents |
The AI Agent Charter and Runbook holds these elements, plus the permissions register, the monitoring checks, the escalation paths, and a change log. For roles, see the RACI Matrix Builder.
How Agents Fail, and What Controls Them
| Failure | What happens | Controls |
|---|---|---|
| Prompt injection | Text in an email, web page, or document tells the agent to do something else, and it obeys | Treat all outside content as data, not instructions; limit tools; require approval for sensitive actions; filter outputs |
| Excessive agency | The agent has more permissions than the task needs, so a mistake does more harm | Least privilege; separate read and write; spending and recipient limits |
| Wrong action | A plausible but incorrect step, such as closing the wrong ticket | Validation rules; confirmation for irreversible steps; sampling |
| Looping or runaway cost | The agent repeats steps or makes many calls | Step, time, and spend limits; alerts |
| Data leakage | The agent sends data where it should not | Data class rules; allow-lists for recipients; output checks |
| Silent change | A model or tool update changes behavior | Version pinning; regression tests before changes; change control |
| Over-reliance | People stop checking what the agent does | Seeded tests; rotation; audits; see the human-in-the-loop guide |
The OWASP Top 10 for Large Language Model Applications lists prompt injection and excessive agency among the main risks, and is a good checklist for the people who build or buy agents. Operations owners do not need the technical detail; they need to ask whether each control is in place.
Testing Before You Trust It
- Write test cases from real work, including the awkward ones: missing information, contradictions, unusual requests, and attempts to push the agent outside its job.
- Run it in shadow mode. Let it work beside the people, compare its output with theirs, and take no action from it.
- Measure agreement and the serious misses separately. A high overall agreement can hide a few dangerous errors.
- Use exact intervals. A small number of cases gives a wide range of what the true rate could be.
- Decide the promotion criteria in advance: the results needed to move to the next stage, and the results that send it back.
Deploying in Stages
| Stage | Typical promotion evidence |
|---|---|
| Shadow to assist | Agreement with experts at the target level on enough real cases, and no serious misses |
| Assist to act with approval | Reviewers accept most proposals; the override rate is low and understood |
| Approval to sampling | Error rate shown low with enough items; errors are reversible; controls tested |
| Sampling to monitored | Stable error rate on a control chart; alerts and kill switch tested; owner signs off |
Worked Example: A Work-Order Triage Agent at Riverside Plant
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, built a triage agent for maintenance requests. It reads each request, finds the machine’s history, and proposes a priority and an assignee. Permissions: it can read the work-order system and the equipment history, and it can write a proposed priority into a field; it cannot close, delete, or assign work, or send messages. It started in shadow mode on 200 requests. Figures are illustrative.
| Measure | Result |
|---|---|
| Priority agrees with the planner | 178 of 200 (89.0%), 95% interval 83.8% to 93.0% |
| Critical requests flagged as critical | 22 of 22, 95% interval 84.6% to 100% |
| Disagreements | 22: 14 rated higher than the planner, 8 lower, none critical |
The decision. The agent agrees with the planner 89% of the time, which is good for a first run. All 22 critical requests were flagged, but 22 cases cannot show a rate of 95% or better: that would take at least 59 critical requests with none missed. So the agent was promoted from shadow to assist, then to act with approval, where it writes its proposed priority and the planner confirms. It is not yet allowed to set priorities alone.
What would move it up. The team set the criteria in advance: an override rate under 5% over a full month, no missed critical requests across the next 59 critical orders, the 22 disagreements reviewed and fixed in the instructions, and a tested kill switch. They will track the override rate on a control chart.
Plan your own agents in the AI Agent Charter and Runbook, and score their tasks with the Agent Autonomy and Oversight Planner.
Running and Monitoring
| Measure | What it shows |
|---|---|
| Task success rate | How often the agent completes the task correctly |
| Escalation rate | How often it stops and hands over |
| Override rate | How often people change what it did or proposed |
| Incidents and near misses | Wrong or unexpected actions, with causes |
| Cycle time and cost per task | Whether it still pays |
| Drift signals | Changes in the mix of inputs or in results |
Chart the key rates over time with control limits so that real changes stand out from noise (see SPC Control Charts), review incidents like any other (see Incident Investigation), and put changes to instructions, tools, and models through change control (see Management of Change).
Common Mistakes
An Agent Where a Workflow Would Do
Extra risk for no extra value.
Broad Permissions
Giving access to everything because it is easier.
No Stop Conditions
The agent presses on when it should hand over.
Promotion on Feel
Moving up a level without evidence.
Untrusted Content as Instructions
Letting text from outside direct the agent.
No Owner
Nobody watches it or answers for it.
Self-Assessment Questions
- Is an agent really needed, or would a workflow or an assistant do?
- Is there a written job description: purpose, scope, tools, limits, owner?
- Are permissions the minimum the task needs?
- What evidence would move it to the next stage, and who decides?
- Can we stop it at once and undo what it did?
AI Agents in Operations: Frequently Asked Questions
What is an AI agent?
A system that uses an AI model to pursue a goal across several steps, deciding which steps to take and using tools such as search, email, ticketing, or databases to take them. It differs from an assistant, which answers when asked, and from workflow automation, which follows a fixed path.
When should I use an agent instead of a workflow?
Only when the path really varies, several tools are involved, and the value justifies the extra control. If the steps are stable, a fixed workflow is cheaper, easier to test, and more predictable. If a person with an assistant does the job well, stop there.
How do I limit what an agent can do?
Give it the minimum permissions the task needs, separate reading from writing, set limits on steps, time, and spending, require human approval for sensitive actions such as external messages and irreversible changes, define stop conditions, and keep a kill switch and logs.
What is prompt injection?
An attack or accident in which text in content the agent reads, such as an email, web page, or document, contains instructions that the agent follows instead of its own. Controls include treating outside content as data, limiting tools, requiring approval for sensitive actions, and filtering outputs.
How do I know an agent is ready for more autonomy?
Set the evidence in advance: enough real cases tested in shadow mode, agreement with experts and the serious misses reported separately with exact intervals, a low and stable override rate, reversible errors, and tested controls. Move one stage at a time and keep the ability to step back.
Sources and Further Reading
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1) and Generative AI Profile (NIST AI 600-1).
- OWASP, Top 10 for Large Language Model Applications, on prompt injection and excessive agency (check the current version).
- Anthropic, Building effective agents (December 2024), on workflows versus agents and keeping designs simple.
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D., “A model for types and levels of human interaction with automation,” IEEE Transactions on Systems, Man, and Cybernetics, Part A 30(3), 2000.
- ISO/IEC 42001:2023, Artificial intelligence: management system, on operational planning and control (check current edition).
- Clopper, C. J. and Pearson, E. S., “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika 26, 1934, for exact binomial intervals.