This guide is for buying AI software: a SaaS tool your team logs into, or an API your developers call. It is not about hiring a firm to build something for you. If that is your situation, see our guide to what a custom AI build involves, or, if you want a business process automated end to end, our guide to hiring an AI automation company.
Buying an AI tool is harder than buying most software for two reasons. The output changes with your data, so a demo tells you little. And the vendor often sits on top of another company’s model, so data handling and pricing can change underneath you. The kit below is built around those two problems: a questionnaire to send before any demo, a 30-day pilot plan, and a scoring table for the final decision.
Why a structured process pays off
Adoption is wide, but returns are uneven. In McKinsey’s 2026 State of AI survey of 1,719 respondents, 37% attributed at least some EBIT impact to AI, about the same share as the year before. Data is a common reason for the gap: Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. For a tool buyer, the practical lesson is to test on your own data early and to make the vendor say, in writing, what happens to it.
Step 1: Build a shortlist of two or three tools
Start with the job, not the product. Write one sentence describing the task, who does it today and how you will know it improved, for example: “Support agents draft first replies to billing tickets; we want median handling time down without a rise in reopened tickets.” That sentence decides which category you search in.
You can use AIML Marketplace’s AI tools directory to build the long list. Browse by category, such as writing and content, AI assistants or CRM, and use the listings to note what each tool says it does, how it is priced and what it connects to. Treat a listing as a starting point, not a security review. Our How we review page explains what our listings do and don’t verify, so you know which claims still need checking with the vendor.
Cut the long list to two or three by sending each vendor the questionnaire below. How a vendor answers is evidence in itself: clear, documented answers within a week are a good sign; “let’s discuss on a call” for every item is not.
Step 2: Send the vendor questionnaire
Copy the questions into an email or a shared document. Ask for written answers with links to the supporting documents. Delete the sections that do not apply to you.
Security and compliance
- Do you have a current SOC 2 Type 2 report? Please share it (under NDA if needed) and state the audit period.
- Which sub-processors handle our data, including any third-party model providers? Where is that list published, and how are we notified of changes?
- Do you support single sign-on, role-based access and audit logs of user activity? On which plan?
- Will you sign a HIPAA business associate agreement or a data processing agreement, if we need one?
- If you make regulatory claims (for example, FDA clearance), which specific product and intended use do they cover?
Data use and training
- Are our inputs, outputs or uploaded files used to train or improve your models or any third party’s models? Is that the default, and on which plans?
- If training is opt-out, how do we opt out, and does it apply retroactively?
- Do your staff or contractors ever view our content, for example for abuse monitoring or quality review? Under what conditions?
Retention and residency
- How long do you keep prompts, outputs, files and logs? Can we set a shorter retention period?
- When we delete data, how long until it is removed from backups?
- In which countries is our data stored and processed, including by your model provider? Can we pin it to a region?
Pricing at volume and usage caps
- Based on [our expected monthly volume], what would we pay in a typical month and in a worst-case month?
- Which actions consume credits or tokens, including retries, long documents and agent steps?
- Can we set hard spending caps and alerts per workspace or per user? What happens when a cap is hit?
- What rate limits apply on our plan, and how much notice do you give before pricing or limits change?
Export and exit
- Can we export all our data, prompts, templates, configurations and any fine-tuned models? In what formats?
- What happens to our data and contract if you are acquired, change your model provider or shut down?
- How long do we have to retrieve data after the contract ends, and do you confirm deletion in writing?
For calibration, some providers publish these answers. OpenAI, for example, states on its enterprise privacy page that by default it does not use business data from its enterprise products and API to train its models. A smaller vendor building on such a model should be able to tell you whether those terms pass through to you, and what its own retention terms add.
A note on SOC 2: the AICPA describes SOC 2 as covering controls relevant to security, availability, processing integrity, confidentiality or privacy. It is evidence that controls were tested, but it does not answer the training, retention or residency questions, which is why they are listed separately.
On regulatory claims: an FDA 510(k) clearance applies to a specific device and intended use, so read the clearance itself. The FDA also publishes a list of AI-enabled medical devices authorized for marketing in the US, which you can check a vendor’s claim against. If you sell into the EU, ask how the vendor tracks the EU AI Act: obligations for general-purpose AI models applied from August 2025, and rules for high-risk uses such as employment and education apply from 2 December 2027 following the AI Omnibus amendments.
Step 3: Run a 30-day pilot
A pilot is where demo performance meets your real data. Run the shortlisted tools on the same tasks, with the same testers, over the same period. Agree the success metrics before day one and write them down, so nobody moves the goalposts after seeing the results.
Week-by-week plan
| Week | What happens | Output |
|---|---|---|
| Before day 1 | Pick 30-100 real, representative cases, including awkward ones. Record the current baseline: time per task, error or rework rate, cost per task. | Test set and baseline figures |
| Week 1 | Set up accounts, SSO and any connections to your systems. Two or three testers learn each tool. Note setup effort and support responsiveness. | Setup log, hours spent |
| Week 2 | Run the full test set through each tool. Testers record time taken and grade each output: usable as is, usable with edits, or unusable. | Graded results per tool |
| Week 3 | Use the tools on live work at normal volume. Track usage and spend daily against the vendor’s estimate. | Real usage and cost data |
| Week 4 | Test export: pull your data and configuration out. Collect tester feedback. Fill in the scoring table. | Completed scoring table, export check |
Success metrics to set in advance
- Quality: share of outputs usable as is or with light edits. Set a pass mark, for example a minimum percentage you would accept in production.
- Time saved: time per task compared with your baseline, including the time spent checking and fixing output.
- Cost per task: actual spend during week 3 divided by tasks completed, compared with the vendor’s quote.
- Failure behavior: what the tool does with cases it cannot handle. Saying “I don’t know” is better than a confident wrong answer.
- Adoption: whether testers kept using the tool in week 3 without being reminded.
- Exit check: whether the export in week 4 produced complete, usable files.
Use the pilot to watch for the problems demos hide. Scripted demos run on prepared data. Some products are a thin prompt layer over a third-party model, which is fine as long as you know what you are paying for and what happens if that model’s pricing or behavior changes. Content tools trained on broad data often drift away from your brand voice over longer runs. If the tool needs to read from or write to your CRM, helpdesk or ERP, test that connection during the pilot rather than taking it on trust; our guide to connecting AI to your business tools covers what that work involves.
Step 4: Score the shortlist
Fill in the table at the end of the pilot. Score each criterion from 1 (poor) to 5 (strong), multiply by the weight, and add up. Adjust the weights to your situation before the pilot starts, not after. A regulated business might raise data handling to 25% and lower ease of use.
| Criterion | Weight | Tool A (1-5) | Tool B (1-5) | Tool C (1-5) |
|---|---|---|---|---|
| Output quality on your test set | 25% | |||
| Time saved against baseline | 15% | |||
| Data use, retention and residency terms | 15% | |||
| Security evidence (SOC 2, SSO, audit logs) | 10% | |||
| Cost per task at expected volume, with caps | 15% | |||
| Connection to your existing systems | 10% | |||
| Export and exit terms | 5% | |||
| Ease of use and tester adoption | 5% | |||
| Weighted total | 100% |
Set a few criteria as pass/fail gates as well, so a high total cannot hide a deal-breaker. Typical gates: the vendor will not confirm in writing that your data is excluded from training, there is no way to cap spending, or the export test failed. A tool that fails a gate is out, whatever it scores elsewhere.
If none of the tools clears your bar, that is a useful result too. It may mean the workflow is too specific for an off-the-shelf product, which is when a custom build or an automation project is worth pricing.
Before you sign
- Get the questionnaire answers you relied on into the contract or data processing terms, not just an email.
- Start on a monthly or short annual term if you can, with a price cap for renewal.
- Set spending alerts in the admin console on day one.
- Name an internal owner who reviews usage, cost and output quality each quarter.
- Diary the export test again before renewal, since products and terms change.
Who reviews tools for this site
When you are ready to start a long list, browse the AI tools directory by category and run the shortlist through the questionnaire above.
Frequently Asked Questions
Two, occasionally three. One tool gives you nothing to compare against, so you cannot tell whether a weak result is the tool or the task. More than three spreads your testers thin, and the pilot turns into a feature tour. Shortlist on paper with the questionnaire first, then pilot only the tools that answered it acceptably.
It is a good baseline, not a full answer. A SOC 2 report shows an auditor tested the vendor’s controls, but it says little about whether your inputs train shared models, how long prompts and outputs are kept, or which sub-processors and model providers see your data. Ask those questions separately and get the answers in the contract or data processing terms.
A tool is a finished product you configure and pay for by seat or usage; your leverage is in testing and contract terms. A development firm builds something specific to you, so the questions shift to scope, ownership of code and models, and who maintains it. If no tool covers your workflow after a fair pilot, a custom build may be the better route.
Only if the trial lets you use your own data under terms you would accept in production. Many free tiers have different data-use and retention terms from paid business plans, so check before uploading anything sensitive. A trial run on sample data tells you about the interface, not whether the tool works on your real cases.
Written by: AIML Marketplace Team
