In From Cocoa to Chocolate, model output is the cocoa: valuable, sold by every provider, and falling in price. The AI Gateway is where that cocoa enters the factory. This piece walks through it in full: what it checks, what it controls, and what it lets a business see.
One intake door for every model
Every factory has a single intake door. Ingredients arrive from many suppliers, but they all come through the same door, where they are checked, weighed, priced and booked to the line that will use them. Nothing reaches the line without passing it.
Without that door, AI arrives the way it does in most businesses today: team by team, each app and agent reaching out to a model on its own. Provider keys sit in every app. No one can cap what a team, an app or an agent spends until the invoice arrives. And no one can say which models are in use, or what they cost.
The AI Gateway is that door for models: one access layer for every large language model the business uses, across providers and hosts. Apps, agents and people call it, and every request passes four stages in order.
- 1
Access
Every model behind one endpoint and one key.
- 2
Control
Budgets, rate limits and allowed models.
- 3
Cost
The same model, every host and rate, with routing to match.
- 4
Visibility
Every request metered, and attributed to whoever made it.
It is not only for agents built in Chocolate Factory. The AI Gateway serves apps and agents from any vendor. The business’s existing Claude, OpenAI and in-house agents, and any app already calling a model, can route through it and come under the same keys, budgets and metering.
Access: every model behind one endpoint
Commercial models such as Claude and GPT-4o sit beside open-weight models such as Llama, Qwen and Mistral, all behind a single endpoint. Switching from one model to another means changing a model name, not rewriting an integration. The code stays the same.
That matters because the model underneath will change, many times. A business that has wired one provider into every app has to rewrite each of them to move. A business that calls one endpoint changes a name.
Model consumers see a catalog of every model available to them: where each one is hosted, what each endpoint costs, and which ones their team is cleared to use. Choosing a model becomes a decision made with the facts in front of you, not a guess.
Provider keys stay in the gateway
The keys for Claude, OpenAI and every other provider are held centrally and never sit in apps. In their place, the gateway issues virtual keys to teams, apps and members, so access can be granted, scoped and withdrawn without touching a provider account.
A provider key scattered across a dozen apps is a key nobody can safely withdraw. A virtual key belongs to one team, one app or one member, so taking it back affects only them.
Control: budgets, rate limits and allowed models
Spend caps and request limits are set per team, project or app key, and each project chooses which models it has turned on. A team cannot drift onto an expensive model it was never cleared for, and no single app can exhaust the budget for everyone else.
This is control set in advance rather than discovered afterwards. The limits are decided before the first request, not read off the invoice at the end of the month.
Cost: the same model, every host and rate
The same open-weight model is often offered by several hosts, from public clouds to local AI factories to the business’s own servers, each at a different price. The gateway lays them side by side.
A routing rule, set per model or per consumer, decides where each request goes: to the lowest-cost endpoint, to a preferred host, or to one fixed endpoint. If an endpoint is down or rate limited, the request falls back to the next in line, and the app never notices. A rule can be tested with a sample request before it goes live.
Routing turns price and availability into rules the business sets once, instead of choices each app makes on its own.
Visibility: every request metered
Each request is measured in input tokens, output tokens and cost, then attributed to the model, the team and the consumer that made it. Usage can be viewed over the last fifteen minutes, today, this month or a custom range, and spend can be billed back to whoever used it.
This is how a business answers two of the three questions that keep AI in pilots: which models are in use, and what they cost. Finance sees AI spend as it happens, not when the invoice arrives.
Two sides of one catalog
The catalog that consumers browse has a provider side. Data centres and AI factories onboard the models they host, set their own prices, and track usage and revenue per model. Their models and rates then appear in every consumer’s catalog, beside the commercial providers.
For model consumers
Apps, agents and people
- Every model available to them, in one catalog
- Where each model is hosted and what each endpoint costs
- Which models their team is cleared to use
For model providers
Data centres and AI factories
- Onboard the models they host
- Set their own prices
- Track usage and revenue per model
For a business that wants models served close to home, this is where those models appear. For a provider, it is a place in every customer’s catalog.
More in the AI Gateway
- Spend caps with a circuit breakerAn alert at 80% of a budget and, at the cap, a choice to throttle, block or fall back to a cheaper model.
- GuardrailsPersonal data masked and prompt injection stopped before a request reaches a model.
- Request logEvery request recorded: who sent it, with which key, to which model, how long it took and what it cost.
- Load balancingTraffic for one model spread across hosts, regions and keys.
- Routing by difficultySimple requests sent to cheaper models and hard ones to stronger models, without a code change.
- CachingRepeated and near-identical requests answered without paying a model again.
- Keys that expireKeys with a set end date.
What it changes
For the CIO and IT, provider keys live in the gateway instead of in a dozen apps, and every model is in one catalog. For finance, usage is metered per model, team and consumer, budgets and rate limits are set per project and key, and routing sends each request to the cheapest host that fits. For developers, one endpoint reaches every model, and switching models is a name change, not a rewrite.
The intake door does not make the chocolate. It makes sure every ingredient that reaches the line is known, priced and accounted for. The full feature set is on the AI Gateway page.
The short version
- Commercial and open-weight models sit behind one endpoint; switching models is a name change.
- Provider keys stay in the gateway, and budgets, rate limits and allowed models are set per team, project or app key.
- Routing sends each request to the lowest-cost, preferred or fixed endpoint, with fallback, and every request is metered.
- Spend caps, guardrails, a request log, load balancing, routing by difficulty, caching and expiring keys are part of it too.



