FREEAI
All notes

Model infrastructure

One gateway in front of every model

Auth, quotas, retries and per-project metering, and what changed on the bill once every call went through one door.

18 Jul 2026 · 6 min read

One gateway in front of every model

The first few projects call the model provider directly. That is the right thing to do at the start — an SDK, a key in an environment variable, and the feature works. The problem arrives quietly, somewhere around the fourth system.

What goes wrong without one

  • The key exists in four repositories, three deployment environments and at least one developer machine. Rotating it is a scheduling exercise rather than a command.
  • The monthly invoice is a single number. Nobody can say which product spent it, so nobody can decide whether it was worth spending.
  • A provider incident takes down every feature at once, because every feature talks to the same endpoint with no alternative path.
  • Retry behaviour is whatever each SDK defaults to. Under load, four systems retry into the same rate limit and make it worse.

None of these are model problems. They are ordinary infrastructure problems that arrive with the model, and they are solved the way infrastructure problems are always solved: put one door in front of the thing, and make everything go through it.

What the gateway actually does

  • Authentication per project, not per provider. Each service holds a token that identifies it, and it never sees a provider key.
  • Quotas and metering per project, so cost lands on the product that caused it.
  • Retries with backoff in one place, with a shared view of rate limits rather than four independent guesses.
  • Failover to a second provider for the calls where a degraded answer beats no answer. Not every call qualifies, and deciding which do is part of the design.
  • A request log with the model, the parameters and the outcome — the record you need when someone asks why a decision was made three months ago.

What changed

The most useful change was not technical. Once spend was attributable per project, conversations about cost became specific: this feature costs this much per document, and here is what it replaces. Before that, the only available conversation was about the total, which is the least actionable number in the building.

Key rotation became a single operation. Swapping a provider became a configuration change rather than a release across four codebases. And when a provider did have a bad afternoon, the failure was contained to the calls we had decided could not fail over.

What it does not solve

A gateway does not improve a weak prompt, and it does not make a model suitable for a task it is not suitable for. It also adds a hop, so it has to be boring: if the gateway is the least reliable component in the chain, it has made things worse rather than better. We treat it as infrastructure — small, dull, heavily monitored, and rarely changed.

Worth building once you have more than two systems calling models, or the first time someone asks what the model spend is actually for.

Working on something similar?

Describe what happens today in a few sentences. We will tell you which part a model can carry, and which part it cannot.

Send a request