An AI vendor walks into a sales meeting with a 40-slide deck, three buzzwords per slide, and a demo that looks impressive on data they chose. The buyer is a CEO with a budget, a problem, and no technical training. The question on the table is whether to commit six figures and six months to a tool the buyer cannot fully evaluate on its mechanics. The good news is that most of what matters about an AI purchase is not the mechanics. It is the operating questions any executive already knows how to ask.

This piece is the playbook for that meeting. The vendor language to discount. The three questions that separate a real solution from a polished pitch. What good answers look like, and what evasion looks like when you hear it. The diligence is non-technical and takes one conversation to run.

Part 01The technical evaluation is not the bottleneck

Most executives assume the hard part of evaluating an AI vendor is the technology. It is not. The hard part is the operating fit. Whether the tool produces something your business can use. Whether the failure modes are tolerable. Whether the vendor will be around in three years. Whether the integration with your real workflow is plausible or aspirational. None of those questions require a degree.

The technical layer matters, but a working AI tool can be built on any reasonable foundation. A broken integration cannot be fixed by a better model. The most expensive AI mistakes in small and mid-size businesses have not been bad model choices. They have been good models bolted onto workflows nobody verified, supported by vendors who could not name the first thing their tool fails on.

Part 02The vendor language to discount

Certain phrases appear in every AI sales deck and carry almost no decision-relevant information. They are not lies. They are noise. When you hear them, do not penalize the vendor, but do not give them credit either. The signal lives somewhere else in the conversation.

"Powered by [GPT-4 / Claude / Gemini / proprietary AI]"

The base model is rarely the thing that determines whether the product works for you. Two vendors using the same model can ship wildly different products because the integration, prompt engineering, evaluation practice, and operational support all happen above the model layer. A vendor leaning on the model name as a credential is usually trying to borrow trust they have not earned through their own work.

"99% accurate"

Accuracy without a defined task is marketing. Accurate at what. On what data. Measured how. Compared to which baseline. If a vendor cannot answer those four questions in plain language, the number is not real. A good vendor will tell you, unprompted, the categories of input where accuracy drops, because they have measured it.

"Self-learning" / "continuously improving"

Operationally, these phrases usually mean one of two things. Either the vendor retrains the model occasionally on aggregated customer data, or the system has no real learning loop at all and the phrase is decoration. Ask which one. The answer changes your data-sharing posture, your privacy exposure, and your understanding of what you are actually buying.

"AI-driven" / "AI-powered" / "AI-enabled"

Sometimes the AI component is the core of the product. Sometimes it is a single feature in a tool that would function fine without it. Ask which workflows in their product actually use AI, and which are conventional software with an AI label on the box. The answer affects what you are paying for and what breaks if the AI piece fails.

Part 03The three questions that separate real from polished

These three are the entire evaluation, asked in any order. A vendor who can answer all three with specifics is worth a second meeting. A vendor who deflects on any of the three is selling you their pitch, not their product.

Question 1: What does this fail on, and how would I know?

Every AI tool has a failure surface. Categories of input it handles badly. Edge cases it produces confident wrong answers on. Conditions under which performance degrades. A vendor who has shipped real product can describe their failure modes in concrete terms because they have seen them in customer environments. A vendor who claims "it works on everything" has either not deployed at scale or is not paying attention when it breaks. The follow-up is how the failures surface to your team. An AI tool that fails silently is worse than one that fails loudly, because the silent failures travel into your operation undetected.

Question 2: Where does our data go, and what happens to your model if we cancel?

The first half tells you the privacy and compliance picture. Where is our data stored, who has access, is it used to train shared models, can we contractually opt out. The second half tells you the lock-in picture. If we cancel after eighteen months, do we lose only the tool, or do we also lose the institutional knowledge the system accumulated about our operation. Vendors who built their product to make customer exits clean will answer this directly. Vendors who built their product to make exits painful will give you a long answer that does not contain the word "no."

Question 3: Show me the workflow this replaces, end to end, including the human steps.

This is the operating-fit question. The vendor should be able to describe the current state, the proposed state, and every human touchpoint that survives the change. If they cannot describe your workflow back to you in operational terms, they have not done the work to understand whether their tool actually fits. They are pitching features against a problem they have not bothered to map. The cost of that gap shows up three months after implementation, when the tool is technically working but nobody on your team knows when to use it.

A real AI vendor sells you an operating change. A hype-driven one sells you a demo.

Part 04What good answers sound like

The difference between a real answer and an evasion is usually audible to a non-technical listener. Good answers are concrete, qualified, and contain things the vendor wishes were better. Evasions are abstract, universal, and uniformly positive.

A real answer to "what does this fail on" sounds like: "It handles structured invoices well. On handwritten purchase orders it is below 85% extraction accuracy, so we flag those for human review. On documents in languages we have not trained on, we tell you up front the tool will reject them." That answer is specific, names a number, names a failure mode, and names what happens operationally when failure occurs.

An evasion to the same question sounds like: "Our system is designed for high reliability across diverse document types and uses advanced AI to ensure accuracy in production environments." That answer contains zero decision-relevant information. It could be true of any product or no product. Push back. A vendor who keeps producing those answers under pressure does not have the specifics, which means the diligence ends there.

The pattern repeats on the data question. A real answer names the storage region, the access controls, the training-use policy, and a contract clause you can read. An evasive answer talks about "enterprise-grade security" and "industry-leading practices" without pointing at anything specific. The same vendor who hides behind that language during the sale is the vendor whose contract will not let you walk away cleanly.

Part 05The non-technical diligence checklist

Five things an executive can verify before signing, in roughly the time it takes to review the contract. None of them require an engineer.

None of those checks require technical fluency. They require operational seriousness and a willingness to insist on specifics until you get them. The vendor that earns the business is the one who answers all five without flinching. The vendor that loses the business is the one whose polish does not survive the questions.

Buying AI well is not about understanding the model. It is about refusing to skip the operating conversation in favor of the technology conversation. That is the conversation any executive can run.