Many AI efforts in operations stall on data, not on the tool. Fields are blank, codes differ by shift, the correct answers were never recorded, or nobody has checked that the data may be used this way. A short readiness check before the pilot is cheap and can save months.
This guide gives ten questions (availability, accuracy, completeness, consistency, timeliness, ground truth, permission, sensitive content, ownership and lineage, representativeness) with three gates, shows how to check accuracy against the source with an interval, explains why to fix data at its source, and walks through a worked example with maintenance work orders.
It does not cover data engineering. Example data are illustrative.
Before You Start
The Data Decides More Than the Tool
Most AI projects in operations that stall do so on data. The tool is ready and the data is not: fields are blank, codes mean different things on different shifts, the correct answers were never recorded, or nobody is sure the data may be used this way. A data readiness check before the pilot costs a few days and can save months. It is the same logic as checking a measurement system before collecting data: if the inputs cannot be trusted, nothing built on them can be.
Check Before You Build
A short assessment against ten questions shows what is missing before money is spent.
Accuracy Is a Gate
Wrong data in means wrong answers out, and a polished tool hides it.
You Need the Right Answers
Without agreed correct answers, you cannot test the tool or monitor it.
Permission Is a Gate
Privacy, contract, and policy limits apply to training, prompts, and logs.
Ten Questions
| Question area | What to ask | Gate? |
|---|---|---|
| Availability | Is the data there, and can the AI use reach it? | |
| Accuracy | Does the data match what really happened? Check a sample against the source. | Yes |
| Completeness | Are the fields the task needs filled in? | |
| Consistency | Are the same things recorded the same way across people, shifts, and sites? | |
| Timeliness | Is the data current enough for the decision? | |
| Ground truth | Are there correct answers you can test against, agreed by experts? | Yes |
| Permission | May this data be used this way, under privacy, contract, and policy? | Yes |
| Sensitive content | Is personal, confidential, or restricted data identified and handled? | |
| Ownership and lineage | Does someone own the data, and do you know where it came from? | |
| Representativeness | Does it cover the awkward, rare, and seasonal cases? |
Score each from 0 (not in place) through 1 (partly) to 2 (in place and working). Three are gates: accuracy, ground truth, and permission. A zero on a gate stops the go decision whatever the total.
Worked Example: Work-Order Data at Riverside Plant
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, wanted an AI tool to suggest failure codes and priorities for maintenance work orders. The data team scored the work-order history on the ten questions. The example is illustrative.
The total is 13 of 20, and the rule gives Prepare first: below 16, but no gate at zero. The findings behind the scores:
- Completeness (1): 31% of the 4,200 work orders from the last year have a blank failure code, mostly from one shift.
- Accuracy (1): a planner re-coded 100 work orders from the technician notes, and 78 agreed with the original code (78%, with a 95% interval of 69% to 85%). Training the tool on the other 22% would teach it the errors.
- Consistency (1): the same fault is coded three ways at two sites.
- Ground truth (1): there is no agreed set of correct answers yet. The planners will build a golden test set of 200 work orders.
- Sensitive content (1): technician names and some customer site details appear in the free-text notes and must be masked or excluded.
Fix It at the Source
Cleaning data downstream is a cost that repeats. The Lean answer is to fix the source: make the right entry the easy entry, remove free-text where a list will do, define each code in plain words with an example, and feed errors back to the people who make them. A data collection plan, as in the Data Collection Plan Builder, is a good start.
Steps
- List the data the use needs and where each item comes from.
- Score the ten questions with the people who create and use the data, using a real sample, not opinions.
- Check accuracy against the source on 50 to 100 records, and report the interval.
- Settle permission with privacy, legal, and the data owner, and write down the limits (see the Data Protection and Privacy Guide).
- Build the golden test set with experts who agree on the answers.
- Decide: go, prepare first (with owners and dates), or stop.
- Re-score before launch and at each review, because data quality drifts too.
Common Mistakes
Assuming Volume Means Quality
Years of data full of the same wrong code.
Skipping the Accuracy Check
Trusting the field names instead of looking at records.
No Ground Truth
Nothing to test the tool against.
Ignoring Permission
Using data for a new purpose without checking the limits.
One-Time Cleaning
The source still produces bad data.
No Data Owner
Nobody is accountable for fixes.
Self-Assessment Questions
- Have we checked a sample of the data against the source and reported the accuracy?
- Do we have agreed correct answers to test against?
- Do we know what we may do with this data?
- Is the source being fixed, not only the extract?
- Does the data have an owner who will fix problems?
Data Readiness for AI in Operations: Frequently Asked Questions
What does data ready for AI mean?
The data the use needs is available, accurate enough, complete enough, consistently recorded, current, and allowed to be used this way, with agreed correct answers to test against and an owner who will fix problems.
How much of my data should I check for accuracy?
Check a random sample of 50 to 100 records against the source, such as the technician notes or the original document, and report the share that matches with an interval. A larger sample gives a tighter interval.
What is ground truth?
A set of cases with correct answers agreed by experts, used to test the AI and to monitor it later. Without it, you cannot tell whether the tool is right. See the golden test set entry.
Can I use data I already have for a new AI purpose?
Not automatically. Privacy rules, contracts, and company policy can limit uses of personal and confidential data. Check with the data owner and privacy or legal leads before using it, and record the answer.
Is it better to clean the data or fix the source?
Fix the source where you can. Cleaning downstream repeats at every extract. Mistake-proof the entry, define the codes, and feed errors back to the people who make them.
Sources and Further Reading
- ISO 8000-8:2015, Data quality: Information and data quality: Concepts and measuring (check the current edition), on dimensions of data quality.
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1, 2023), on data quality and fitness for purpose in mapping and measuring risk.
- Gebru, T. et al., “Datasheets for datasets,” Communications of the ACM 64 (2021), on documenting how a dataset was made and may be used.
- Wilson, E. B., “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association 22 (1927), for the interval on a proportion.