On-premise AI
When the data cannot leave your own infrastructure
Running open-weight models on hardware you control — what it costs, where it tops out, and how to prove the data never left.
5 Aug 2026 · 7 min read

The requirement usually arrives as a sentence in someone else’s contract: this data may not be processed outside our environment. Sometimes it is a regulator, sometimes a client’s security review, sometimes a board that has decided the answer before the question was asked. It is rarely negotiable, and it usually arrives after the rest of the system has been designed.
Three ways to satisfy it
- A commercial provider with a deployment in the region or tenancy you are allowed to use. Least work, and the quality ceiling is the highest available. The question to ask is where inference actually happens, not where the account is registered — those are not always the same place.
- Open-weight models on hardware you control, in your own building or your own rack. Full control over where the data goes, and cost that stops scaling with every call. You now own a GPU, its power bill, its driver stack and its failure modes.
- A split, where only the step that touches restricted data runs locally and everything else goes to a commercial model. Usually the most economical answer, and the hardest to explain in an audit — you have to be able to show exactly which fields crossed which boundary.
The part that surprises people
The gap between a self-hosted open-weight model and a frontier commercial model is not a single number. On constrained extraction — pull these six fields out of this form — a well-prompted open model on local hardware is frequently good enough, and the remaining errors are the same kind the commercial model makes. On open-ended reasoning over long, messy context, the gap is wide and does not close with prompt engineering.
This means the decision is task-by-task, not system-wide. We measure both on the client’s own documents before recommending either. A benchmark score tells you nothing about how a model handles the specific way one organisation photographs its paperwork.
What it costs to run yourself
The hardware is the visible cost and usually the smaller one. The rest is operational: keeping the inference server up, keeping drivers and runtimes on versions that still work together, capacity planning for peaks, and having someone who can diagnose it at four in the afternoon when a batch job is queued behind a stuck request. If nobody is going to own that, a compliant commercial deployment is the honest recommendation even though it looks less impressive.
What changes the arithmetic is volume. Per-call pricing is cheap until a process runs thousands of times a day; owned hardware is expensive until exactly that point. We work out where the crossover sits for the actual workload before anyone buys anything.
Where the gateway fits
A self-managed gateway in front of local models earns its place here more than anywhere else. It gives you one place to enforce that restricted traffic can only reach local endpoints — a rule that is easy to state and easy to violate accidentally when four services each hold their own configuration. It is also the only practical way to produce the log an auditor will ask for.
The claim to be careful with is “our data stays inside”. Be able to point at the boundary, name what crosses it, and show the log.
Next note
Rules and models are not rivalsWorking on something similar?
Describe what happens today in a few sentences. We will tell you which part a model can carry, and which part it cannot.
Send a request