Five Questions to Ask Before You Approve an AI Project

Sometime in the next year, you are going to be asked to sign off on one of three things: a vendor pitching an AI product, an agency proposing an AI build, or a software purchase where AI features are a big part of the price. The demo will be good (demos are always good).

The question that no one will answer is the one that determines if the project survives: what does this cost at your real volume, not the demo’s volume?

I run technology for a software product that depends heavily on AI, and I have watched this question get skipped at signing and become an emergency later. In one case we modeled, the projected cost of processing a customer’s full document volume came out at more than ten times the annual value of the contract. We caught it because we ran the numbers before the architecture was locked, but most companies run them when the first real bill arrives. By then, the choices that created the bill are expensive to reverse.

If you’re not asking the right questions, you’re setting yourself up to fail. Vendors price the pilot: a small dataset, a handful of users, costs that look like a rounding error. You pay for production: your entire document history, your whole team, every month. Companies with a CTO have someone whose job is to be suspicious in that meeting. If you don’t have a CTO and this job falls on you, these are the five questions I would bring.

1. “When someone on my team uses an AI feature, what does it look at by default: the thing they have open, or everything we have ever stored?”

Most AI systems default to searching or processing everything. That’s the difference between a question costing a few cents and a single message from one employee generating a thousand-dollar bill. It usually happens by accident with your most enthusiastic users.

A good answer sounds like: “It defaults to the current folder or matter, and going wider requires an explicit choice. The user sees how much it will touch before it runs.”

An evasive answer sounds like: “It searches everything so users get the best results.” The best results and the biggest bill are the same setting.

2. “When a user is about to exceed a cost limit, does the system warn them or stop them?”

Why this matters: a warning is a suggestion, and your heaviest users will click through it. If a limit can be bypassed, it’s not a real limit. If the vendor’s cost controls are all warnings, your ceiling on spend is whatever your most active employee decides it is that month.

A good answer sounds like: “There are hard caps enforced on the server. Above the threshold, the action does not run without an administrator raising the limit.”

An evasive answer sounds like: “Users get clear notifications about usage.” Notifications inform users, but they don’t protect you from runaway costs.

3. “This runs on one AI provider. What happens to our price when their pricing changes, and who absorbs it?”

Most AI products are built on a single model provider and pay that provider for every unit of work. Those prices change, and the change flows somewhere: into the vendor’s margin or into your renewal. A vendor locked into one provider has no negotiating position, which means you don’t have one, either.

A good answer sounds like: “We run on multiple models, we route work to the cheapest one that meets our quality bar, and our contract with you does not pass through provider price changes.”

An evasive answer sounds like: “We use the best model on the market.” That is a quality claim standing where a cost answer should be, and it usually means that the risk will be passed onto you.

4. “Show me the cost model at our full volume, not the pilot volume.”

Pilot economics and production economics are completely different businesses. A system that costs $300 a month across ten thousand documents can hit astronomical prices across five hundred thousand, and the relationship is rarely linear. If you’re the first person to ask about this, that tells you something about the maturity of what you are buying.

A good answer sounds like: a spreadsheet, with your numbers in it, walked through line by line.

An evasive answer sounds like: “Costs scale smoothly with usage.” Smoothly is not a number.

5. “You say the cheaper model is good enough for this. Show me the evidence on our kind of work.”

The honest way to control AI costs is to use expensive models only where they are necessary and validate cheaper ones everywhere else. The dishonest way is to quietly swap in cheaper models and hope nobody notices the quality drop. You need documented evidence that the cheaper model meets a defined quality bar on your kind of task instead of a generic benchmark.

A good answer sounds like: “Here are our quality criteria, here are the passing thresholds, and here is how the model scored on documents like yours.”

An evasive answer sounds like: “Our customers have not reported quality issues.” Absence of complaints is not a test.

Not every AI project should move forward

Some projects won’t survive these questions. Sometimes the best outcome is a smaller purchase than the vendor proposed. Sometimes it’s buying an off-the-shelf tool instead of commissioning a build. Or sometimes it even means waiting a year. These questions aren’t designed to kill AI projects: they exist to help you surface irreversible decisions while they’re still cheap to change. They also only cover the vendor-facing half of the risk; the organizational reasons AI initiatives stall, from unclear ownership to no shared strategy, are a separate problem we unpack here.

Thirty minutes of these questions in a vendor meeting is the cheap version of this discovery. The expensive version arrives about eighteen months in, itemized, after the architecture is locked and the data has been migrated. AI systems are easy to start and hard to stop, and the vendors know which half of that sentence they’re selling.

Running this cost model before any commitment is made is the first thing we do in a technology assessment, and if you’re staring at one of these three approvals right now, it’s worth your time (and money) to do it before you sign, not after.

Frequently asked questions

What questions should you ask an AI vendor before signing?

Five questions surface the cost risks that demos hide: What does an AI feature look at by default, the thing a user has open or everything you've ever stored? Does the system warn users approaching a cost limit or actually stop them? What happens to your price when the vendor's AI provider changes theirs, and who absorbs it? What does the cost model look like at your full volume rather than pilot volume? And if the vendor claims a cheaper model is good enough, what documented evidence backs that on your kind of work? Good answers are specific. Evasive answers are quality claims standing where cost answers should be.

What's the difference between a cost warning and a hard cap?

A warning is a suggestion, and your heaviest users will click through it. A hard cap enforced on the server stops the action from running until an administrator raises the limit. When evaluating any AI product, ask specifically whether limits can be bypassed, because a limit that can be bypassed is not a limit.

Should you walk away from an AI project that can't answer these questions?

Not necessarily, but you should change the shape of the commitment. Thirty minutes of questions in a vendor meeting is the cheap version of this discovery. The expensive version arrives eighteen months in, itemized, after the architecture is locked.