ARIA · Auto-Resolve Benchmark

We publish our support AI's false-answer rate.

Anyone can quote a resolution rate. Almost nobody tells you how often their bot confidently answers a question it had no business answering. Both numbers are below — plus the command to reproduce them.

◆ Measured 2026-07-14 · reproducible in one command · no signup

Verification notice · found 2026-10-08 · re-run 2026-10-14 and 2026-10-21, same result

We re-ran our own reproduce-it command on 2026-10-08, 2026-10-14 and 2026-10-21, and it did not reproduce the figures below. We are telling you before you find out yourself.

The numbers on this page were measured on 2026-07-14 against a knowledge base of 31 patterns. Running node tools/measure-kb-selftest.mjs against the knowledge base currently in our repository — 17 patterns — returns 23.6% in-scope deflection, 6.7% out-of-scope false answers and 12% control hijacks, with zero coverage for VPN, webcam, BitLocker recovery and onboarding. The 31-pattern coverage build-out described further down this page is not present in the current tree, so the published figures cannot be reproduced today.

We do not know yet whether that build-out was lost, reverted or never merged, and we will not guess in either direction. Until it is restored and re-measured, treat every figure on this page as a dated historical measurement and not a current one. The restore is logged as our highest-priority open defect; this notice is removed only when the command reproduces the numbers again, and the re-measured figures — whatever they turn out to be — get published here even if they are worse.

Why we left the old numbers visible instead of quietly deleting them: a published figure you can check against a stated date is auditable. A page that silently changes is not. This is the same reason we publish a false-answer rate at all.

One number would lie. A support bot can hit a beautiful "answered 90% of questions" score by answering everything — including the questions it should have handed to a human, and the ones that were never IT support in the first place. That is not coverage. That is a bot guessing with confidence.

So we measure three things and publish all three, including the ones that make us look worse.

1 · In-scope deflection

93.1%

407 / 437 real IT questions

Answered instantly by the offline knowledge base — zero AI calls, zero network, zero human. Higher is better, and this is the number we are actively pushing up.

2 · Out-of-scope false answers

30 non-IT questions

Questions that were never IT support ("bitcoin price", "what's the weather"). If the bot answers one of these, that is a defect, not coverage.

Must be 0 · currently 0

3 · Control hijacks

83 dialogue turns

Conversation turns the dialogue layer owns — "still not working", "get me a human". Firing a printer fix at one of these is a hijack.

Must be 0 · currently 0

Reproduce every number yourself

The knowledge base and the question corpus are both plain files in our repository. The harness reads the exact same knowledge base that ships to your browser — not a copy, not a mock.

node tools/measure-kb-selftest.mjs

The knowledge base: assets/aria-knowledge-base.js — 31 patterns, the same file the browser loads.
The corpus: tests/scenario-corpus.js — 1896 entries, de-duplicated to 631 unique questions. We de-duplicate because the corpus pads itself with ALL-CAPS restatements, and counting those would inflate the sample without adding a single new phrasing.
Case stability: every answered question is re-asked in ALL CAPS and must still be answered. Currently: passing.

Coverage by problem type

This is the honest spread — the strong areas and the weak ones, in the same table. The low rows are our backlog, not a secret.

Problem type	Answered	Rate
printer	21/21	100%
onedrive sync	20/20	100%
performance	20/20	100%
windows / boot	20/20	100%
browser	15/15	100%
microsoft 365 / entra	12/12	100%
macos	8/8	100%
onboarding	8/8	100%
bitlocker	5/5	100%
microsoft teams	29/30	96.7%
vpn / remote access	24/25	96%
mfa / 2fa	28/30	93.3%
out-of-office	14/15	93.3%
bluetooth / audio	14/15	93.3%
usb / external drives	12/13	92.3%
email / outlook	46/50	92%
security / phishing	18/20	90%
webcam	9/10	90%
active directory	9/10	90%
file permissions	9/10	90%
core networking (dns/dhcp)	9/10	90%
password / login	34/40	85%
wi-fi	23/30	76.7%

What we fixed the day we published this

Running this benchmark honestly is only worth something if we act on what it finds. The last run found real defects. Here is what they were and what we did:

The bot answered "bitcoin price" with our pricing sheet. A too-broad pricing pattern matched any sentence containing the word "price". It is exactly the class of defect a single flattering metric hides. Fixed: a price word must now sit next to something we actually sell, and an explicit guard vetoes crypto, stock, flight and shopping phrasings. Out-of-scope false answers went from 6.7% to 0%.
Four whole problem types had zero coverage — VPN / remote access, webcams, BitLocker recovery, and employee onboarding & offboarding. All four are now answered. In-scope deflection rose from 23.6% to 93.1% after the coverage build-out below.
The benchmark itself had a scoring bug — so we fixed the benchmark and said so. It was counting the bot's correct "glad that worked" reply to "thanks, that worked" as a false fire, and it was scoring an unlabelled grab-bag of questions as if every one of them were a dialogue turn. Both were corrected. That grab-bag (80 questions, the bot answers 15) is now reported separately and excluded from the headline entirely, because it mixes real support questions with small talk and neither answering nor abstaining is provably right. We would rather exclude it and tell you than score it in whichever direction flatters us.

What this number is not

Deflection is not resolution. This measures the instant offline answer layer — the part that responds with zero AI cost and zero waiting. Full resolution is that layer plus the AI, plus guided recipes, plus a human when it matters.
Deflection is not remediation. Answering "here is how to fix your VPN" is not the same as changing the machine. That is ARIA Sentinel's job, it runs under an approval gate, and it is measured separately.
This is a self-test, not a customer-validated production rate. It runs against a public corpus of real-phrased IT questions. When we have production numbers from paying clients, we will publish those too — and we will say which is which.

Why any of this matters to you

When you buy IT support with an AI layer in front of it, the risk is not that the AI misses a question — a human picks that up. The risk is that it answers wrongly with total confidence, and your staff act on it. A vendor who cannot tell you their false-answer rate has not measured it. Ask them for it. If they will not publish it, that is your answer.

We would rather ARIA say nothing and hand you to a person than guess. That is why two of the three numbers on this page are required to be zero, and why the one that is allowed to be imperfect is the one we show in full detail.

Try ARIA Free IT health check What managed IT costs in the GTA