ARIA · Auto-Resolve Benchmark
We publish our support AI's false-answer rate.
Anyone can quote a resolution rate. Almost nobody tells you how often their bot confidently answers a question it had no business answering. Both numbers are below — plus the command to reproduce them.
◆ Measured 2026-07-14 · reproducible in one command · no signup
One number would lie. A support bot can hit a beautiful "answered 90% of questions" score by answering everything — including the questions it should have handed to a human, and the ones that were never IT support in the first place. That is not coverage. That is a bot guessing with confidence.
So we measure three things and publish all three, including the ones that make us look worse.
1 · In-scope deflection
93.1%
407 / 437 real IT questions
Answered instantly by the offline knowledge base — zero AI calls, zero network, zero human. Higher is better, and this is the number we are actively pushing up.
2 · Out-of-scope false answers
0%
30 non-IT questions
Questions that were never IT support ("bitcoin price", "what's the weather"). If the bot answers one of these, that is a defect, not coverage.
Must be 0 · currently 0
3 · Control hijacks
0%
83 dialogue turns
Conversation turns the dialogue layer owns — "still not working", "get me a human". Firing a printer fix at one of these is a hijack.
Must be 0 · currently 0
Reproduce every number yourself
The knowledge base and the question corpus are both plain files in our repository. The harness reads the exact same knowledge base that ships to your browser — not a copy, not a mock.
node tools/measure-kb-selftest.mjs
- The knowledge base:
assets/aria-knowledge-base.js — 31 patterns, the same file the browser loads.
- The corpus:
tests/scenario-corpus.js — 1896 entries, de-duplicated to 631 unique questions. We de-duplicate because the corpus pads itself with ALL-CAPS restatements, and counting those would inflate the sample without adding a single new phrasing.
- Case stability: every answered question is re-asked in ALL CAPS and must still be answered. Currently: passing.
Coverage by problem type
This is the honest spread — the strong areas and the weak ones, in the same table. The low rows are our backlog, not a secret.
| Problem type | Answered | Rate | |
|---|
| printer | 21/21 | 100% | |
| onedrive sync | 20/20 | 100% | |
| performance | 20/20 | 100% | |
| windows / boot | 20/20 | 100% | |
| browser | 15/15 | 100% | |
| microsoft 365 / entra | 12/12 | 100% | |
| macos | 8/8 | 100% | |
| onboarding | 8/8 | 100% | |
| bitlocker | 5/5 | 100% | |
| microsoft teams | 29/30 | 96.7% | |
| vpn / remote access | 24/25 | 96% | |
| mfa / 2fa | 28/30 | 93.3% | |
| out-of-office | 14/15 | 93.3% | |
| bluetooth / audio | 14/15 | 93.3% | |
| usb / external drives | 12/13 | 92.3% | |
| email / outlook | 46/50 | 92% | |
| security / phishing | 18/20 | 90% | |
| webcam | 9/10 | 90% | |
| active directory | 9/10 | 90% | |
| file permissions | 9/10 | 90% | |
| core networking (dns/dhcp) | 9/10 | 90% | |
| password / login | 34/40 | 85% | |
| wi-fi | 23/30 | 76.7% | |
What we fixed the day we published this
Running this benchmark honestly is only worth something if we act on what it finds. The last run found real defects. Here is what they were and what we did:
- The bot answered "bitcoin price" with our pricing sheet. A too-broad pricing pattern matched any sentence containing the word "price". It is exactly the class of defect a single flattering metric hides. Fixed: a price word must now sit next to something we actually sell, and an explicit guard vetoes crypto, stock, flight and shopping phrasings. Out-of-scope false answers went from 6.7% to 0%.
- Four whole problem types had zero coverage — VPN / remote access, webcams, BitLocker recovery, and employee onboarding & offboarding. All four are now answered. In-scope deflection rose from 23.6% to 93.1% after the coverage build-out below.
- The benchmark itself had a scoring bug — so we fixed the benchmark and said so. It was counting the bot's correct "glad that worked" reply to "thanks, that worked" as a false fire, and it was scoring an unlabelled grab-bag of questions as if every one of them were a dialogue turn. Both were corrected. That grab-bag (80 questions, the bot answers 15) is now reported separately and excluded from the headline entirely, because it mixes real support questions with small talk and neither answering nor abstaining is provably right. We would rather exclude it and tell you than score it in whichever direction flatters us.
What this number is not
- Deflection is not resolution. This measures the instant offline answer layer — the part that responds with zero AI cost and zero waiting. Full resolution is that layer plus the AI, plus guided recipes, plus a human when it matters.
- Deflection is not remediation. Answering "here is how to fix your VPN" is not the same as changing the machine. That is ARIA Sentinel's job, it runs under an approval gate, and it is measured separately.
- This is a self-test, not a customer-validated production rate. It runs against a public corpus of real-phrased IT questions. When we have production numbers from paying clients, we will publish those too — and we will say which is which.
Why any of this matters to you
When you buy IT support with an AI layer in front of it, the risk is not that the AI misses a question — a human picks that up. The risk is that it answers wrongly with total confidence, and your staff act on it. A vendor who cannot tell you their false-answer rate has not measured it. Ask them for it. If they will not publish it, that is your answer.
We would rather ARIA say nothing and hand you to a person than guess. That is why two of the three numbers on this page are required to be zero, and why the one that is allowed to be imperfect is the one we show in full detail.