How Not to Light a Pile of Money on Fire

Most teams have no idea what their AI workload actually costs until the invoice arrives, and by then it is a single line item with no way to ask it questions.

So we went and found out on purpose.

In June we were load testing Origami, the part of our platform that turns a sentence like "I need a support agent that reads my docs and escalates angry customers" into the actual working thing.

Testing it properly meant simulating customers at scale. Forty concurrent organisations, then eighty, then a hundred and ten, each provisioning projects and knowledge bases and multi-agent teams, all of it hitting real AWS Bedrock with real model traffic. That is a good approximation of what a single heavy enterprise account looks like sitting on your infrastructure.

Then we measured it, and spent the following weeks engineering the number down.

What enterprise-scale agent traffic actually costs

Measured on AWS Bedrock and expressed as run rate, which is the number that actually matters when you are sizing a workload.

That is the profile of one heavy account. Multiply it across a customer base and you can see why "we will optimise it later" is an expensive sentence.

The useful part is not the total. It is that once you can see the shape of the spend, almost all of it turns out to be addressable.

Where the money actually goes

The assumption is that you pay for answers. You do not. You pay for context.

Our orchestrator sends a system prompt of about 9,200 tokens plus thirteen tool schemas. Byte for byte identical on every single call. Then a user message of a couple of hundred tokens.

Measured ratio: 21.8 to 1, input to output. Around 95% of the bill was the same static document being re-transmitted thousands of times a day at full retail.

Lever one: cache the static prefix

If the first 9,000 tokens of every request are identical, you should be paying for them once, not every time.

Prompt caching does exactly that. On a real twenty person concurrent run, 8,470 of 8,797 prompt tokens were served from cache at roughly a tenth of the price.

That is about 66% off every turn, with identical tokens reaching the model. Same output, same quality, same behaviour. It is a configuration change, not a rewrite.

Lever two: route to the cheapest capable model

The default move is to pick the smartest available model and send everything to it. It feels responsible. It is expensive and usually unnecessary.

We ran eight identical multi-agent builds on a frontier model and on a small fast one. Same builds, same success criteria. $6.84 against $0.50.

Then we did it properly and published the methodology. 72 tasks across the six internal agents basically every company already has: HR onboarding, expense policy, IT helpdesk, contract review, CRM lookups, tier 1 support. Three runs, four configurations, 1,296 conversations, scored on cost and accuracy together, because a saving that costs you accuracy is not a saving.

Cost-optimised routing came in 74% cheaper at 98.15% of the accuracy, and more than halved latency, 8.2 seconds down to 3.5.

Two caveats, because a benchmark without them is marketing:

• 74% is the number on our corpus. It is not a promise about yours. Different workloads route differently, which is why the policy is configurable rather than baked in.
• There is one honest regression in the data. Ask the expense agent about a $95 client dinner and the frontier model returns the receipt rule and the $80 cap. The cheaper model returns the receipt rule and drops the cap. Every single run. One policy detail out of seventy two tasks, for 74% off. That is a real trade, and you should get to see it before you make it rather than discover it in production.

What the two levers add up to

Caching cuts the repetition. Routing cuts the overqualification. They compound, because they act on different parts of the same request.

Same product, same output quality, roughly a tenth of the cost.

This is the product

Everything above is a lever we built for ourselves first, then shipped, because we needed it before anyone else did.

Bonito is one OpenAI-compatible endpoint in front of six providers, AWS Bedrock, Azure, Google Vertex, OpenAI, Anthropic and Groq. Changing models is a string change instead of a migration. On top of that:

• Per-request cost tracking, broken out by model and by key, with cost per 1,000 tokens so you can compare models on the only metric that matters.
• Prompt caching, so your static prefix is billed once instead of every call.
• Routing policies you configure rather than hardcode: cost, latency, balanced, failover, A/B.
• Multi-provider failover that fires on rate limits, timeouts and dead deployments, so one provider's bad day is not your outage.
• Spend caps that stop a runaway loop before it becomes a story.

Your cloud, your credentials, your data. We sit in front of it, we do not take custody of it.

Complexity-aware routing, where the gateway reads a request and picks a model based on how hard it actually is, is next. Today's cost policy picks the cheapest capable model against real catalogue pricing, and that is what produced the 74%.

The 90% was not a clever trick. It was measurement, then two configuration changes. The only hard part is being able to see the spend clearly enough to know which levers to pull, which is exactly what we built the platform to do.