AI Agent Evaluation: An RFP Question Bank Written for the Buyer

Vendor-written checklists ask about features the vendor has, not commitments the buyer can enforce. Of the four checklists ranking for this query, none asks how escalations are billed or for a subprocessor list. Fini’s guide evaluates nine platforms and ranks Fini first. Parloa, Fin and Ada publish neutral-sounding frameworks that close on a demo of their own product.

That gap now has a price. Forrester’s 2027 predictions say AI agents will influence 75% of B2B purchase decisions, and that two-thirds of providers will lose deals because buyers cannot verify their claims. The buyer’s own AI is now the fact-checker.

“These tools rapidly synthesize provider claims and check facts.”

Credit where due: Ada’s guide does tell buyers to get the resolution definition in writing. It just doesn’t ask what happens to the bill, the data or the knowledge base afterwards. Most vendors in this category call their AI agentic. Gartner estimates only about 130 of thousands of agentic AI vendors are real, so the RFP has to do the sorting.

Ask 24 questions across four phases — pre-RFP screening, evaluation, contract, and the first 90 days — and require a written answer to each. Questions 6–13 are the core block to send every vendor. The right-hand column is the part no vendor checklist includes: how you check the answer once the vendor no longer needs to impress you.

The downloadable version adds a “what a good answer looks like” column and blank cells for each vendor’s answer, evidence link and score.

Pull 200 random AI-handled conversations each month and re-score them yourself against the contract’s definition of “resolved.” Aissist’s deflection vs resolution guide describes four checks: count same-issue repeat contacts within seven days, separate confirmed endings from silent ones, report handovers as their own line, and confirm the denominator. The guide cites Aissist’s AI Customer Service Benchmark 2026 for a 20–40 point gap between deflection and resolution on the same deployment.

Silence is the expensive one, and vendors know it.

“If a frustrated customer stops responding, does that count as 'resolved'?”

Sampling 200 conversations is a floor, not a QA program. Aissist’s Agent Insight page, citing Solidroad, puts typical manual QA coverage at 1–2% of conversations; Agent Insight itself scores 100%. Whichever tool you use, the test is the same: every score links to the conversation and states its reason, and the rubric is yours. If the vendor scores itself, the scorecard is marketing.

Aissist answers the billing and helpdesk questions well, and the data-exit and subprocessor questions badly. We ran the bank against Aissist’s own public pages on 26 Oct 2026, accepting only what a buyer could find in writing. Aissist reports 83% average resolution and 4.8/5.0 CSAT on its evaluable AI page — vendor-claimed figures, which is precisely why questions 6–10 exist.

Four gaps out of eight is not a flattering scorecard. It is the honest one, and a buyer would find it in an afternoon. Our pricing models comparison covers how other vendors answer question 8.

A resolution rate is a claim; a written definition plus a monthly audit sample is a commitment. The vendor checklists ranking today help you compare features. This bank helps you hold a vendor to what it said, using the export, the invoice and the helpdesk you already own. If a vendor will not answer question 6 in writing, treat its per-resolution price as a per-conversation price. Send questions 6–13 to every shortlisted vendor, put the answers to 6, 8, 14 and 16 into the order form, and check where your current numbers sit with the industry benchmark tool. Then run Phase 4 every month, including on us.