A purchased machine behaves the same way next year. A purchased AI service may not: answers are probabilistic, the vendor can change the model, your data flows to the vendor, and you usually cannot see inside. These differences change what you ask before you buy.
This guide gives a six-step process (requirement, gates, due diligence, test on your data, scoring, negotiation and pilot), a table of questions by area, a weighted scorecard with must-pass gates, and a worked example in which the best-fitting vendor fails a gate on data use. It lists contract topics to raise with legal counsel.
It is not legal advice. Example data are illustrative.
Before You Start
Buying AI Is Not Like Buying a Pump
A purchased machine behaves the same way next year. A purchased AI service may not: its answers are probabilistic, the vendor can change the model behind it, your data flows to the vendor, and you usually cannot see inside. These differences change what you ask before you buy. The discipline is the one you already use for suppliers: define requirements, qualify the source, test the product, agree the terms, and keep checking. See Supplier Quality Management.
Test on Your Data
A demo on the vendor’s examples says little about your cases.
Know Where Your Data Goes
Storage, location, subprocessors, retention, and whether it trains the vendor’s models.
Expect Change
Ask what notice you get before the model changes, and whether you can pin a version.
Plan the Exit
Data export, deletion, and the cost of leaving, agreed before you sign.
A Process in Six Steps
- Write the requirement from the use case: the task, volumes, accuracy needed, data classes, integrations, and the autonomy level.
- Set the must-pass gates before you meet vendors. A vendor that fails one is out, whatever else it offers.
- Send a due-diligence questionnaire and ask for security reports and data terms in writing.
- Test on your own data, with a golden test set and a blind comparison, and report intervals.
- Score and discuss with the people who will use it, who own the risk, and who pay.
- Negotiate and pilot with an exit, then monitor as in the Monitoring Guide.
Questions to Ask
| Area | Ask | Look for |
|---|---|---|
| Performance | How was accuracy measured, on what data, and may we test on ours? | Willingness to be tested; honest limits; no claims of 100% |
| Security | What independent security reports do you have, and how do you control access, log, and respond to incidents? | Current reports (for example SOC 2 or ISO/IEC 27001 certification); encryption; single sign-on; logging; a breach notice commitment |
| Data use | Is our data used to train your models? Who can see it? Where is it kept, for how long, and by which subprocessors? | No training on your data without opt-in; clear retention and deletion; a subprocessor list |
| Transparency and change | Which model and version do we use? How much notice do we get of changes? Can we pin a version? | Version named; advance notice; change log |
| Safety and misuse | How do you handle prompt injection, harmful output, and misuse? | Described controls and testing; a way to report problems |
| Service | What service levels, support hours, and incident handling do you commit to? | Written service levels with remedies |
| Rights and exit | Who owns outputs? How do we export and delete our data? What help is there at exit? | Clear ownership; export in usable formats; deletion confirmed in writing |
| Viability and compliance | Who are your customers? What laws and standards do you follow? | References; a plan for rules such as the EU AI Act, if relevant |
Worked Example: Three Vendors for Complaint Classification
Riverside Plant, an illustrative 140-employee metal fabrication and assembly plant, wanted to buy a complaint classifier. It set eight weighted criteria and three must-pass gates (a written answer to the security questionnaire, no training on its data without opt-in, and export and deletion rights). It tested three vendors on the same 200-case golden set. The scores are illustrative.
| Criterion | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Fit on our data | 25 | 4 | 5 | 3 |
| Security and data handling | 20 | 4 | 3 | 5 |
| Data rights and exit | 15 | 3 | 2 | 5 |
| Transparency and change notice | 10 | 2 | 2 | 4 |
| Integration and fit | 10 | 4 | 3 | 3 |
| Support and service levels | 8 | 4 | 3 | 4 |
| Three-year cost | 7 | 3 | 4 | 3 |
| Vendor viability | 5 | 4 | 3 | 4 |
| Weighted score (of 100) | 100 | 71.6 | 66.4 | 78.6 |
| Must-pass gates | All met | Failed: trains on data by default, no opt-out | All met |
Scores are 1 (poor) to 5 (excellent); the weighted score is the sum of weight times score, divided by 5. Vendor B was best on fit and cheapest, but its standard terms allowed training on customer data with no opt-out, so it failed a gate. Vendor C scored 78.6, ahead of Vendor A at 71.6, mostly on security and on data rights and exit. The team did not simply take the top number: it asked Vendor B whether the term could be changed. The answer was no, so B was out. It also asked the users whether the gap between C and A mattered. A fits the task a little better (4 against 3), but C is stronger on security, data rights and exit, and notice of change, and the users judged those gaps larger than the fit gap.
Terms Worth Asking Your Legal Team About
- Data use and confidentiality: no training on your data without opt-in; limits on who can see it; return and deletion.
- Ownership and rights in outputs and inputs, and who carries the risk if outputs infringe others’ rights.
- Change notice: advance notice of model and terms changes, and the right to stay on a version or leave.
- Service levels, support, and remedies.
- Security obligations, audit or report rights, subprocessors, and breach notification times.
- Liability and indemnity that match the risk of the use.
- Exit: data export, transition help, and deletion.
This is a list of topics to raise with qualified legal counsel, not legal advice.
Common Mistakes
Believing the Demo
No test on your own data.
Terms Read After Signing
Data use and exit surprises.
Choosing by Price Alone
Cheap tools with weak terms or high review cost.
Skipping Security Review
Because the tool is only a pilot.
Lock-In
No export, no exit plan.
No Monitoring After Purchase
Assuming the vendor will tell you when it changes.
Self-Assessment Questions
- Have we written the requirement and the must-pass gates before talking to vendors?
- Have we tested on our data, blind, with intervals?
- Do we know where our data goes, who sees it, and whether it trains the vendor’s models?
- Do we have change notice, version pinning if possible, and an exit plan in the terms?
- Do we monitor the tool after purchase?
Buying and Evaluating AI Tools and Vendors: Frequently Asked Questions
How do I evaluate an AI vendor?
Write the requirement, set must-pass gates, send a due-diligence questionnaire, test on your own data with a golden test set, score against weighted criteria with the users, and negotiate terms including data use, change notice, and exit.
What should I ask about my data?
Whether it is used to train the vendor’s models, who can see it, where and how long it is kept, which subprocessors handle it, how you can export and delete it, and what happens to it at exit.
Is a high score enough to choose a vendor?
No. A must-pass gate, such as no training on your data without opt-in, applies first. Weights are judgments, so test whether the ranking holds when you change them, and let a pilot settle close calls.
What security evidence should a vendor provide?
Independent reports or certifications such as SOC 2 or ISO/IEC 27001 where relevant, plus written answers about access control, encryption, logging, incident response, and breach notice. Read them; do not just collect them.
What if the vendor changes the model?
Ask for advance notice and a change log, try to pin a version, record the version in your monitoring log, and re-test on your golden set after any change.
Sources and Further Reading
- ISO 9001:2015, clause 8.4, Control of externally provided processes, products and services (check the current edition).
- National Institute of Standards and Technology, AI RMF 1.0 (NIST AI 100-1, 2023) and Generative AI Profile (NIST AI 600-1, 2024), on third-party and supply chain risk.
- ISO/IEC 42001:2023, on managing externally provided AI products and services (check the current edition).
- American Institute of CPAs, SOC 2 reporting framework, and ISO/IEC 27001:2022, as examples of independent security assurance.
- OWASP, Top 10 for Large Language Model Applications (check the current version), on supply chain and data risks.