Organizations usually have more AI ideas than they can staff. A good selection process ranks the ideas on the same criteria, stops the ones with weak data or intolerable error risk, builds a realistic business case for the best few, and tests it in a pilot with a stop rule.
This guide covers six scoring criteria with weights and gates, how to read a portfolio, what a business case must include (baseline, review time, running cost, one-time cost, ramp, and error cost), and the pitfalls in the numbers. A worked example ranks eight ideas at an illustrative plant and builds the case for the first, with payback and a sensitivity chart.
All figures are illustrative. Use your own baseline measurements.
Before You Start
Why Ranking and a Business Case Matter
Most organizations have more AI ideas than they can staff. A weak process picks the idea with the best demo, or the loudest sponsor. A good one ranks the ideas on the same criteria, checks the ones that rank well against a realistic business case, and starts small.
Compare on the Same Terms
A scoring matrix forces the team to say what makes one use better than another.
Surface the Gates
Poor data or intolerable error risk should stop a use case early, however good the savings look.
Count the Whole Cost
Review time, running costs, integration, and the cost of errors often decide the case.
Pilot With a Test
A business case is a hypothesis. The pilot shows whether it holds.
Scoring Candidates: Six Criteria
Score each candidate from 1 to 5 on the criteria below, where a higher score is always better. Agree the meaning of each score with examples before you start, and score in a group so that differences get discussed.
| Criterion | Weight (example) | A score of 5 means | A score of 1 means |
|---|---|---|---|
| Value | 25 | Large, measurable gain in time, quality, safety, or cost | Small or hard to measure |
| Volume | 15 | Thousands of items a month, so benefits repeat | Rare |
| Process readiness | 15 | Standardized, documented, stable | Varies by person; undocumented |
| Data readiness | 15 | Data exists, is accurate, accessible, and allowed for this use | Scattered, poor, or restricted |
| Error tolerance | 15 | Mistakes are small, visible, and reversible | Mistakes are severe or hard to catch |
| Ease of integration | 15 | Fits current systems and people with little change | Needs major system and role changes |
- Score = the weighted sum, scaled to 0 to 100. With weights adding to 100 it is the sum of weight times score, divided by 5.
- Gates. A score of 1 on data readiness or error tolerance blocks a use case until the issue is fixed, whatever its total.
- Tiers (example). 75 or more: pilot now. 60 to 74: prepare first. Below 60: defer. Set your own and apply them the same way each time.
- Treat the score as a conversation tool. The ranking is only as good as the inputs; do not argue over a point or two.
Worked Example: Ranking Eight Ideas at Riverside Plant
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, collected eight AI ideas from its departments and scored them in a half-day workshop. Scores are illustrative.
| # | Use case | Area | Val | Vol | Proc | Data | Err | Integ | Score | Tier |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Complaint classification and routing | Customer service | 4 | 5 | 4 | 4 | 4 | 3 | 80 | Pilot now |
| 2 | Maintenance work-order triage agent | Maintenance | 4 | 4 | 3 | 4 | 3 | 3 | 71 | Prepare first |
| 3 | First drafts of 8D and RCA summaries | Quality | 3 | 3 | 4 | 4 | 4 | 5 | 75 | Pilot now |
| 4 | Invoice matching and exception handling | Finance | 4 | 5 | 4 | 5 | 3 | 2 | 77 | Pilot now |
| 5 | SOP drafting and translation | Operations | 3 | 3 | 3 | 4 | 4 | 5 | 72 | Prepare first |
| 6 | Visual inspection assist at final inspection | Quality | 5 | 5 | 3 | 2 | 2 | 2 | 67 | Prepare first |
| 7 | Demand forecast adjustment suggestions | Planning | 4 | 3 | 2 | 3 | 3 | 3 | 62 | Prepare first |
| 8 | Near-miss report classification | Safety | 3 | 3 | 4 | 4 | 4 | 4 | 72 | Prepare first |
Reading the result. Three candidates reach the pilot tier: complaint classification and routing (80), invoice matching (77), and first drafts of 8D reports (75). Visual inspection has the highest value and volume, but its data readiness and error tolerance are low (2), so it lands in “prepare first”: the plant needs labeled images and a validation plan before any pilot. The team chose complaint classification as the first pilot because it is high volume, easy to check, and, with a safety rule, tolerant of errors.
Use the AI Use-Case Prioritization Matrix to score your own list with editable weights, or the AI Use-Case Register to keep the whole portfolio.
Building the Business Case
Build the case for the top candidates. Keep it simple enough to check, and put every assumption on one page where others can challenge it.
| Item | What to include | Common gap |
|---|---|---|
| Baseline | Items per month, minutes per item, hourly cost, current error rate, cost per error | Guessing instead of measuring |
| Benefit | Share of items the AI can handle, time saved on those items, error reduction, faster response | Assuming it handles everything |
| Human review | Minutes a person spends checking each AI-handled item | Leaving it out |
| Running cost | Licences or usage fees, monitoring, support, and upkeep | Counting only the licence |
| One-time cost | Setup, integration, testing, training, and change management | Ignoring the staff time |
| Ramp | Months to reach full benefit as people adopt and tune it | Assuming full benefit on day one |
| Errors | Cost of AI mistakes compared with today’s mistakes | Assuming no new errors |
The core calculation: time saved per handled item = (minutes today × share of time saved) − review minutes. Monthly labor saving = items handled × time saved ÷ 60 × hourly cost. Add the change in error cost, subtract running cost, and compare the cumulative net with the one-time cost.
Worked Example: The Business Case for Complaint Classification
Inputs (illustrative): 600 complaints a month, with 12 minutes of handling work per complaint that the AI could affect; the AI handles 80% of them and saves 60% of the time on those; a person reviews each in 2 minutes; labor costs $38 an hour; running cost is $450 a month; one-time cost is $18,000; misroutes fall from 4% to 3% at $60 each; benefits ramp up over 3 months.
| Line | Calculation | Per month |
|---|---|---|
| Complaints handled by AI | 600 × 80% | 480 |
| Net minutes saved per complaint | 12 × 60% − 2 | 5.2 |
| Labor saving | 480 × 5.2 ÷ 60 × $38 | $1,581 |
| Error saving | 480 × (4% − 3%) × $60 | $288 |
| Running cost | −$450 | |
| Net benefit at full ramp | $1,419 |
What the case says. The use case pays back in about a year and a quarter, but the margin is not large: the 36-month net is about $31,000 on an $18,000 outlay. The sensitivity chart shows which assumptions to test in the pilot.
Enter your own figures in the AI Business Case and ROI Calculator, which gives the same monthly breakdown, the cumulative chart, and a sensitivity table.
Pitfalls in the Numbers
- Time saved is not cash saved. Ten hours a week saved across ten people is not a saving unless the time is used for something else, or the work is removed. State what happens to the freed time.
- Optimism about coverage. The share of items the AI can handle usually starts lower than the sales pitch. Use pilot data.
- Forgotten review. If every output needs checking, the net saving can be small. Compare review minutes with the time saved.
- Hidden running costs. Usage-based pricing, monitoring, and upkeep add up. Include them.
- Benefits that are not in the baseline. Faster response may be real but have no money value until it changes churn or sales. Keep money and non-money benefits separate.
- The J-curve. New technology often costs productivity before it adds any, because processes and skills must change. Plan for the dip.
From Case to Pilot
- Define success in the numbers that drive the case: share handled, minutes saved, review minutes, error rate, and cost.
- Set a stop rule in advance: what result ends the pilot early, and what result earns a wider rollout.
- Run it small, in one area, with a named owner and reviewer, and measure against the baseline.
- Update the case with real numbers, and decide to expand, change, or stop.
Common Mistakes
Best Demo Wins
Choosing the idea that impresses, not the one that scores best.
Ignoring the Gates
Starting a use case with weak data or intolerable error risk.
A Case on Hope
Assumptions nobody has measured.
Counting Time as Cash
Savings that never reach the budget.
Pilot Without a Stop Rule
A pilot that can never fail and never ends.
Too Many Pilots at Once
Spreading thin so nothing gets measured.
Self-Assessment Questions
- Do we have a ranked list of AI ideas, scored on the same criteria?
- Does each business case include review time, running cost, and the cost of errors?
- Do we know what happens to the time we say we will save?
- Have we set success measures and a stop rule for the pilot?
- Do we know which assumptions matter most to the result?
AI Use-Case Selection and Business Case: Frequently Asked Questions
How do I prioritize AI use cases?
Score each candidate on the same criteria, such as value, volume, process readiness, data readiness, error tolerance, and ease of integration. Weight the criteria, add gates for deal-breakers such as poor data or intolerable error risk, and rank. Then check the top candidates with a realistic business case before piloting.
What should an AI business case include?
A measured baseline, the share of items AI can handle, time saved on those items, the human review time, running costs, one-time costs, a ramp-up period, and the change in error cost. Show payback and a sensitivity analysis, and say what happens to any freed time.
Is time saved the same as money saved?
No. Time saved becomes money only if the time is used for something else of value, or the work or cost is removed. State explicitly what the freed time will do, and track it.
How long should an AI pilot run?
Long enough to collect enough items to measure the success criteria with useful precision, and cover normal variation, often weeks rather than days. Set the success measures and a stop rule in advance, and decide at the end to expand, change, or stop.
Why do AI projects often take longer to pay back than expected?
Review and rework time are underestimated, coverage is lower than promised, processes and skills must change before benefits appear, and running costs accumulate. Research on general-purpose technologies describes a productivity J-curve, where measured gains lag the investment.
Sources and Further Reading
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1), the MAP function on context and benefits.
- Brynjolfsson, E., Rock, D. and Syverson, C., “The Productivity J-Curve: How Intangibles Complement General Purpose Technologies,” American Economic Journal: Macroeconomics 13(1), 2021.
- ISO/IEC 42001:2023, Artificial intelligence management system, on planning and operational controls (check current edition).
- Juran, J. M. and De Feo, J. A., Juran’s Quality Handbook (McGraw-Hill), on the cost of poor quality.
- Montgomery, D. C., Introduction to Statistical Quality Control (Wiley), on sampling and measuring pilot results.