We Benchmarked Our Own Routing. Here Is the Corpus, the Code, and the 74%.
Everyone selling an AI gateway quotes a cost saving. Almost nobody publishes the workload they measured it on, which is why most of those numbers are impossible to check.
So we published ours. The corpus, the routing configuration, the pinned price table, the raw per-task results, and a single command to reproduce the whole thing.
What we measured
Seventy-two tasks across the six internal agents most companies already have: HR onboarding, expense policy, IT helpdesk, contract review, CRM lookups, and tier 1 support. Forty-two routine, eighteen moderate, twelve hard. Twelve of them are multi-turn conversations, because single-turn work overstates how well routing performs.
Every answer is scored against facts that are actually written in that agent's own system prompt, so the test measures retrieval and instruction-following rather than trivia the model might happen to know. There are also traps: can you get the expense agent to approve something only a manager can approve, will the support agent escalate a legal threat instead of trying to resolve it.
Then we ran the full corpus through four configurations, three times each, and measured cost and quality together. A saving that costs you accuracy is not a saving.
1,296 conversations. Zero errors.
The result
| Configuration | Cost | Accuracy | Latency | Saving |
|---|---|---|---|---|
| All frontier model | $0.5022 | 99.31% | 8.2s | baseline |
| Routing, hand-rolled script | $0.2368 | 98.30% | 4.9s | 52.8% |
| Routing + context compression | $0.2268 | 98.84% | 4.8s | 54.8% |
| Bonito routing policy | $0.1308 | 98.15% | 3.5s | 74.0% |
Seventy-four percent cheaper than running everything on a frontier model, at about one point of accuracy, and more than twice as fast. Reproducible to plus or minus 0.03 percentage points across three separate runs.
The row worth sitting with
We wrote our own routing script first. It is the obvious thing any competent engineer builds in an afternoon: look at the prompt, guess whether it is hard, send it to the cheap model if it is not. That script got 52.8%.
The gateway policy got 74.0% on the same corpus, because it is not guessing. It prices every model connected to the organization and picks per request against real catalogue pricing, with the cost model shared by the same code that does the billing. There is no keyword list to maintain and no heuristic to tune.
It is not just downgrading everything
The legal agent is the control, and it is the number we would check first if someone else published this.
Hard contract review tasks held 97.2% accuracy, matching the frontier baseline. A router that moves every request to the cheapest model is not a router, it is a downgrade, and the legal column is how you tell the two apart.
What this does not mean
Seventy-four percent is the number on our corpus. It is not a promise about yours. That is exactly why the corpus is published rather than described: run it against your own workload and get your own number.
The strategy is cheapest capable model, not prompt analysis. It does not read your request and decide it looks difficult. Complexity-aware routing is on the roadmap and we are not going to imply it is here.
And there is one honest regression in the data worth naming. Ask the expense agent about a $95 client dinner and the frontier model returns both the receipt rule and the $80 dinner cap. The cheaper model returns the receipt rule and drops the cap, in every single run. One policy detail lost across seventy-two tasks, in exchange for 74%. That is a real trade and you should be able to see it before you make it.
Why we measure this way
Cost tracking that is never checked against the bill is a guess with a dashboard in front of it.
We reconciled our own gateway against an actual AWS invoice: one month, 36,399 requests. Our platform computed $817.88. The invoice said $884.56. We were under-reporting by 7.5%, and twenty-two of twenty-five days reconciled cleanly, which meant the entire gap sat in three days, with a single day hiding $49.65 of spend that never passed through the tool that was supposed to be watching it.
At our scale that is a rounding error. At a million dollars a month it is $75,000 you cannot see, because the instrument you would use to look is the thing that is wrong.
That is the whole thesis. Routing reduces the bill. Reconciliation tells you whether the reduction was real.
Run it yourself
The benchmark lives in the Bonito repository under the benchmark directory. It ships the task corpus, the scoring rules, the pinned price table with its date, the per-task raw results as CSV and JSON, and the reproduce command.
If you disagree with the method, that is the point of publishing it.
