Open Models Deserve a Backup
On August 22, Groq retired its Llama 3.x chat models. There was no long deprecation window and no gentle migration. The model ids simply started returning "model not found." If you were calling those models directly, a feature broke that morning and you heard about it from a customer.
We know because a few things on our own platform quietly stopped working, and we spent the day migrating them.
That is the state of open model hosting right now. The weights are open, but the hosting is not a commodity you can lean on. A provider can drop a model, change a price, or hit a rate limit at the worst possible moment, and if that provider is your only path to that model, you are stuck.
So we added DeepInfra as our eighth provider, and we wired it in with one specific job in mind.
The same weights, in more than one place
DeepInfra serves a broad catalog of open models through an OpenAI compatible API: Llama, Qwen, DeepSeek, Mixtral, and the gpt-oss family, among others. On its own that is a good addition. Cheaper inference for open workloads, and one more provider you can point your own key at so the spend lands on your cloud.
The part that earns its place is overlap. Groq serves gpt-oss. AWS Bedrock serves gpt-oss. DeepInfra serves gpt-oss. These are the same weights producing the same answers. Before this, if you ran gpt-oss on Groq and Groq had a bad minute, the request had nowhere equivalent to go, so Bonito would either wait or fail.
Now those three providers sit in the same failover family. When a gpt-oss request hits a rate limit or a 500 on Groq, Bonito routes it to the identical model on DeepInfra or Bedrock, on your own credentials, and the caller sees a normal answer. The Llama families work the same way. A few models that DeepInfra serves and nothing else in your stack does, like Qwen 2.5 72B and DeepSeek R1, get same family failover for the first time, because there is finally a second place to send them.
This is the reason to route through a gateway instead of calling a provider straight. Your code asks for a model. The platform figures out how to actually get you one.
It shows up everywhere, not only in a dropdown
Adding a provider is the easy half. Making it a real citizen of the platform is the work.
A DeepInfra request logs the same way every other request does. It carries the provider label, the token counts, and the real cost, so it lands in your cost dashboard, your per model breakdown, and the usage ledger with correct attribution. An agent that runs on a DeepInfra model produces the same trace tree as any other agent, with every model call and tool call and retrieval in order and the cost joined per call. Breadcrumbs draws the same picture. Agent to agent handoffs work without knowing which cloud served the tokens. None of it is special cased, which is the whole idea.
We checked this end to end against the live API before shipping. Credentials validate, the catalog lists, a real request returns content, the gateway routes through the DeepInfra path, cost lands as a real number instead of zero, and a gpt-oss request correctly finds its DeepInfra fallback leg.
Turning it on
If you connect providers in the dashboard, DeepInfra is right there in the connect flow. Paste a key and the platform validates it and syncs the catalog.
If you deploy from a file, add it to your providers block:
providers: - name: deepinfra api_key: ${DEEPINFRA_API_KEY}
From the CLI, one line:
bonito providers add deepinfra
All of it is opt in. If you never connect DeepInfra, nothing about your setup moves. If you do, your open model workloads gain a second and third place to land, and your cost tracking sees every token of it.
Open weights were supposed to mean you were never locked in. Hosting quietly put the lock back on. This is us taking it off again.
