How should a CFO evaluate whether an AI vendor pilot delivered ROI?
Compare what the pilot changed against a baseline measured before it started, in the units finance already reports: hours, cost, cycle time, revenue or risk. Then check that the people who own those numbers agree with the measurement, and price the decision both ways: what production costs, and what waiting costs.
Many pilots stall before that test. 83% of enterprises convert fewer than half of their AI pilots to production (Madrona, Aug 2026). Our read is that often the product didn't fail. Nobody measured what it changed, and the person asked to sign off is left choosing between renewing something unproven and cutting something that may have worked.
Step 1: Set the baseline before the pilot starts
Your team picks two or three measures the pilot is meant to move, not the vendor, and records their current value before day one.
- Choose measures finance already trusts. Time per task, cost per case, cycle time, error rate, revenue per rep. Avoid measures that only the vendor can see.
- Write down how each is measured: the system it comes from, the period, and who owns it.
- If the pilot has already started without a baseline: rebuild one from the systems the pilot touches (ticket history, CRM stages, time logs) for the same period before the pilot. Say plainly that it was rebuilt.
Step 2: Agree who signs off, and on what
A pilot is ready to convert when each owner agrees on their own measure. Expect up to four sign-offs:
| Who | What they confirm |
|---|---|
| The business owner (the team that uses it) | The measures moved, and the team will keep using it |
| Finance | The baseline and the measured change are sound, and the payback works |
| IT or security | It can run in production under the company's controls |
| Procurement | The terms, the price and the exit are acceptable |
Name each person before the pilot starts. A pilot that only the person who brought the vendor in has seen is not ready for review.
Step 3: Work out payback and the cost of waiting
Payback period = the one-off cost to move to production ÷ (the monthly value the pilot measured − the monthly cost to run it).
- One-off cost: integration work, training, internal time and any setup fees.
- Monthly running cost: the license or usage fees, plus the internal time to run it.
- Monthly value: the measured change from Step 1 × the volume it applies to × the unit value finance agrees on (for example, hours saved × loaded hourly cost).
- Use a range, not one number. Work out a low case and an expected case, and name who owns each input. A pilot that only pays back in the best case is a different decision from one that pays back in the low case.
Cost of waiting = the net monthly value × the months a decision is delayed. A pilot that is extended "to be sure" has a price, and it belongs in the review.
Compare it with the real alternative, not only with doing nothing: another vendor, building it in-house, or stopping. Each has its own cost and its own risk.
Step 4: Ask the vendor these questions before renewing
- What did you measure, against which baseline, and who on our side agreed to it?
- Which of our systems does the result come from, and can we reproduce it without you?
- What changes in cost and effort between the pilot and production?
- What happens to our data and workflows if we stop?
- Which customers like us moved from pilot to production, and what did they measure?
- What went wrong during the pilot, and what did you do about it?
A vendor that can answer the first two from your own systems is ready for a production decision. A vendor that can only answer from its own dashboard is not there yet. How a vendor handled problems in the pilot is a fair preview of what it will be like in production.
The worksheet
Copy this table into a spreadsheet, one row per measure.
| Measure | Owner | Source system | Baseline (before) | Pilot result | Change | Volume per month | Unit value | Monthly value (low / expected) |
|---|---|---|---|---|---|---|---|---|
Then add: one-off production cost, payback in months (low and expected), cost of a 3-month delay, and the four sign-offs with dates.
Start here: before the next renewal meeting, fill in the Baseline column with each measure's owner. Any cell that stays empty is the first thing to fix.
From Andru Buyer Intelligence. We help technical founders prepare for the reviews their enterprise customers run. This checklist is written to work for you whichever vendor you're reviewing, including the ones we help.