AI Agent Reliability Benchmark 2026: How AI Agent Error Rate Should Be Measured

An AI agent error rate is the share of AI-handled conversations containing at least one error of a given severity, divided by all AI-handled conversations — reported separately at each severity level rather than blended into one number.

Conversations, not messages. A message-level denominator inflates the volume and flatters the rate: one bad reply inside a forty-message thread reads as a 2.5% failure instead of a 100% one. A conversation can also resolve successfully and still contain an S1, so error rate and resolution rate are independent readings of the same traffic — neither substitutes for the other.

Both factual errors and policy errors count, in replies and in actions. An agent that states the right refund amount but issues it without the verification step your handbook requires has made an error, even though every word it wrote was true. Restricting the count to factual accuracy is how an agent with perfect prose quietly drifts out of policy.

Accuracy is a success metric measured once; reliability is failure measured continuously. Every major vendor publishes the first and none publishes the second, and the binary pass/fail framing hides the only thing a buyer needs — how much a failure costs.

The industry's answer to “is your AI reliable?” is a resolution rate with an accuracy criterion buried inside it. Ada defines an automated resolution as a conversation that is relevant, accurate and safe, where accurate means “the response was correct and consistent with current policies, knowledge, and connected systems” — a binary judgment, pass or fail. Fin reports a 76% average resolution rate across 12,000 customers, “counting only genuine positive resolutions.” Sierra's customers page publishes per-customer results — Airtable at 80%, Chime at 70%+ — with no measurement methodology stated beside them.

Every one of these is a success metric. None is an error metric. And the binary framing hides the thing buyers actually care about: an AI that misspells a product name and an AI that approves a refund it had no authority to approve both register as one failed conversation.

Vendor definitions read from each company's own published pages on 21 September 2026. Aissist.io publishes this table and appears in it.

Worse, single-shot accuracy overstates production reliability. On τ-bench, the customer-service agent benchmark Sierra open-sourced, state-of-the-art function-calling agents succeeded on under 50% of tasks on a single attempt — and under 25% when required to succeed on all eight attempts at the same task. The public τ-bench leaderboard shows the same shape as models improve: the best retail score of 69.2% on one attempt falls to 46.2% across four.

“All the models exhibit considerable performance degradation as k increases, demonstrating their unreliability.”— τ-bench research team, Sierra

Accuracy measured once is not reliability measured continuously. That single fact is why the useful question is not how accurate? It is when it's wrong, how wrong, and who pays?

Grade every AI error on four levels, decided by two questions: does it change a business outcome, and how cheaply can it be undone? S0 is unrecoverable business impact, S1 is recoverable business impact, S2 is process drift with no business impact, and S3 is cosmetic.

The scale is asymmetric by design. S0 and S1 differ by an order of magnitude in cost, not in degree of wrongness. A wrong ETA and an out-of-policy refund are both simply “wrong” in a binary accuracy model; one costs a follow-up message and the other costs money the business cannot get back.

S2 is included because process drift is where S0 incidents come from. An agent that skips identity verification hasn't hurt anyone yet — it has simply removed the control that was standing between a normal conversation and an account takeover. Excluding S2 because “nothing happened” means losing the only leading indicator the scale contains.

S3 is tracked and fed back into Evolve, but excluded from the SLA. Cosmetic errors are a quality signal, not a reliability one, and folding them in is the easiest way to make an error rate look rigorous while measuring nothing.

Detection is two-layer: an LLM audit over every transcript, plus human QA sampling that keeps the auditor honest. An error rate quoted without a stated detection method is a guess, and a rate graded by the model that produced the output is a self-graded exam.

The first layer has to cover everything. Traditional manual QA reviews 1–2% of interactions, which leaves the severity distribution of the other 98% unknown — and severity distributions are exactly what a small sample estimates worst, because S0 events are rare by construction. An LLM audit that reads every transcript is the only way to see a one-in-ten-thousand event at all.

The second layer exists because the auditor is also a model, and it has its own error rate. Human QA sampling over the audit's own labels is what keeps the instrument calibrated. Ada publishes the cleanest example of taking this seriously: its AI scientists validated their labeling methodology to a 0.06% error margin. The measurement instrument needs its own error bar.

At Aissist.io both layers run through Pulse, which reads every conversation rather than a sample, and Agent Insight, which scores AI and human agents on the same scorecard. Grading the AI on a standard your own team isn't held to produces a number that flatters whichever side you chose to measure.

Aissist.io's contractual targets are under 1% for all errors S0–S2 combined, under 0.1% for S1, and under 0.01% for S0 — stated per severity level, over the conversation denominator above, with S3 tracked but excluded from the SLA.

These are thresholds we sign, not averages we observed and rounded. Publishing a target you have to honor is a different act from publishing a result you chose to disclose, and the difference is the whole point of this page.

How the rate is held there

A severity scale measures reliability; it does not produce it. Four mechanisms, stacked, are what keep the number under the threshold — each one described in full on how Aissist.io keeps AI reliable:

• Grounding and prompt guardrails. Answers come only from approved knowledge and connected systems, and a single execution that spins out 12 to 20 tasks applies the same policy guardrails to every one of them. This is the layer that suppresses S1 factual errors.

• Booster — asking more than once. Critical tasks run several times, in an odd number, and the strongest agreement wins. It costs more compute, which is why it is aimed at the decisions where an S0 is possible rather than at every reply.

• Self-inspect before release. A separate pass — often a different model's perspective — checks the output against the request before the customer ever sees it. It is the last chance to catch an error at S3 cost instead of S0 cost.

• Stacked governance. A supervising layer monitors behavior and outputs against policy and escalates anything that violates it. This is the layer that catches S2 process drift, which is the leading indicator the scale exists to expose.

The reason four mechanisms are needed rather than one is that reliability trades against capability. A model constrained tightly enough never to err is also constrained enough to be useless, so each layer buys back a different slice of that tradeoff — the full argument, with the comparison of all four approaches, is on the Reliable AI page.

Ask five questions: the error rate and its denominator, the severity grading and the S0 definition in writing, the S0 rate on its own, who detects the errors, and whether any of it is contractual. A vendor that answers five is measuring reliability.

A vendor that answers with a resolution rate is measuring something else. That is not always evasion — most of them genuinely do not grade severity, so there is no S0 number to give you. The response worth watching is what happens next: whether they offer to start measuring it, or explain why you shouldn't want it. Pair this with the AI Agent Pricing Benchmark 2026, because a vendor that controls both the resolution count and the error count is grading its own invoice.

Reliability cannot be inferred from a success metric, and a blended error rate is only marginally better than none — it lets a thousand cosmetic slips absorb the one refund that shouldn't have been approved.

Three things make an AI agent error rate comparable across vendors: a severity scale that separates unrecoverable harm from cosmetic noise, a denominator stated in conversations rather than messages, and a detection method independent of the model being graded. Ask for all three. Two out of three is a number that cannot be checked.

We publish this scale because we think every vendor in the category should have to, and because the alternative — a market where reliability is asserted rather than measured — is one where the first serious S0 incident is discovered by a customer. The companion reading is the resolution–CSAT tradeoff, which covers the other way a good-looking number hides a bad outcome.

The eight questions we receive most often about AI agent error rate, failure rate and evaluation metrics — each answered directly from the scale above.