Private AI Strategy: When to Run Models Yourself
Mark aistrategy.guide as a preferred source Our AI strategy guides show up more often, with a Preferred badgePrivate AI (on-premise or self-hosted models) is now a credible option because open-weight models such as gpt-oss-120b fit on a single 80GB GPU. When running models yourself beats an API, how to work out the cost crossover, the governance you take on, and which hardware tier a team actually needs.
Two years ago, “run the model ourselves” was mostly a research-team request. In 2026 it is a mainstream option, for a concrete reason: open-weight models good enough for routine enterprise work now fit on hardware a mid-sized company can buy. OpenAI’s gpt-oss-120b, released under the permissive Apache 2.0 licence, is designed to fit on a single 80GB GPU. The strategic question has shifted from “can we?” to “for which workloads is it worth owning the operations?”
The short answer: run models yourself where the data cannot leave your perimeter, where volume is steady enough to keep hardware busy, or where a predictable unit cost matters more than access to the strongest model. Use APIs for everything else. Most organisations end up hybrid, and the governance work is the part they underestimate.
What counts as private AI?
Private AI covers three deployment patterns with different trade-offs:
- Self-hosted in your data center or colocation: full control, full operational burden.
- Self-hosted in a private cloud tenancy: open-weight models on GPUs you rent, inside your own cloud account and region. The data stays in your environment; the hardware does not belong to you. This is often the pragmatic middle path, and the placement logic is in hyperscaler vs neocloud vs on-prem.
- On the device: models running on laptops and workstations for individual users, covered in the AI PC and workstation buying guide.
All three remove the third-party model provider from the data path. None of them removes the need for governance.
When does running models yourself make sense?
Four triggers justify private AI. One strong trigger is usually enough; none at all means an API is the better choice.
- Data residency and confidentiality. Regulated data, client-confidential material, source code under strict contracts, or data subject to residency rules. Keeping inference inside your perimeter turns a contract-and-transfer problem into an internal-controls problem. The policy context, particularly for European organisations, is in AI sovereignty.
- Steady, high volume. Classification, extraction, summarisation and internal search at a predictable daily load. This is where owned hardware stays busy and unit cost falls below API pricing.
- Latency or disconnection. Plants, stores, vessels or field sites where a round trip to a cloud API is too slow or not always available.
- Supplier independence. A model you host cannot be deprecated, repriced or changed underneath you. That stability has value for workflows that are validated once and must behave the same way for years.
Private AI is the wrong default when the workload needs the strongest available reasoning or coding model, when volume is small or unpredictable, or when nobody in the organisation is ready to run a model-serving platform.
How do you work out the cost crossover?
The comparison is cost per unit of useful work at your expected utilisation. Three inputs decide it:
- Fixed cost of capacity per month: hardware amortised over its useful life plus power, hosting and support, or the rental rate. For scale: Lambda’s public page on 20 October 2026 listed on-demand H100 SXM capacity from $3.99 per GPU-hour, which is roughly $2,900 per GPU per month if run around the clock.
- Throughput you will actually achieve: tokens or documents per hour for your model and prompt mix, measured on your workload, not taken from a benchmark.
- The API alternative: the per-token price of the hosted model you would otherwise use, at the same quality level.
Divide the monthly fixed cost by the work the hardware will really do in a month, then compare with the API price. If utilisation is low, the self-hosted unit cost explodes, because idle hours still cost money. Add the people: someone has to run the serving stack, patch it, evaluate new models and handle incidents. For a single small workload, that staffing cost alone usually decides the question in favour of the API.
What governance do you take on?
Self-hosting moves responsibilities from the vendor to you. Put them on the record before the first model goes live:
- Licence compliance. Open-weight does not always mean unrestricted. Meta’s Llama 4 licence, for example, requires companies whose products had more than 700 million monthly active users to request a separate licence, and incorporates an acceptable-use policy. Apache 2.0 models such as gpt-oss carry fewer conditions. Legal should review each model licence.
- Access control and logging. Who can query which model with which data, and where prompts and outputs are stored. Self-hosting does not make a log of confidential prompts any less sensitive.
- Evaluation and change control. A new model version is a change to a production system. Test it against your own evaluation set before swapping it in.
- Security of the model stack. Model servers, vector stores and orchestration code are software with vulnerabilities. They need the same patching and monitoring as any other production service.
The wider control framework is in the governance hub.
Which hardware tier does a team actually need?
Size the hardware to the model and the number of concurrent users, not to the largest model you have read about.
| Tier | Example hardware | What it supports | Fits |
|---|---|---|---|
| Laptop | 16GB+ memory | 20B-class models for one person (gpt-oss-20b runs within 16GB) | Individual use, offline work |
| Workstation | NVIDIA DGX Spark (128GB unified memory); Mac Studio with M5 Ultra (up to 512GB unified memory) | Models up to about 200B parameters on one DGX Spark, per NVIDIA; a few users | Small team, prototyping, sensitive analysis |
| Single GPU server | One 80GB data center GPU such as NVIDIA H100 | gpt-oss-120b-class models | Departmental service |
| Multi-GPU servers | 8-GPU systems and up | Larger models, many concurrent users, fine-tuning | Enterprise-wide service |
Two practical notes. Workstations are excellent for experimentation and small teams, but they are not built to serve a department with consistent response times; that needs server-class GPUs. And memory capacity determines which models fit, while memory bandwidth and compute determine how fast they answer, so a machine that can load a model is not necessarily fast enough to serve it.
For the detailed sizing (memory per parameter, quantisation, concurrency), see ctaio.dev’s local AI hardware guide.
What should you decide this quarter?
- List the workloads blocked by data concerns. These are your strongest private-AI candidates, and often the ones with the clearest business case.
- Pilot on rented GPUs inside your own cloud account before buying hardware. It tests the model, the serving stack and real throughput without capital commitment.
- Write the model-licence and evaluation policy before the first production deployment.
- Decide the hybrid rule explicitly: which data classes may go to external APIs, which must stay private. A one-page rule ends most of the internal debate.
The verdict
Private AI is now a real option, not a research project. Use it where data, volume or independence justify owning the operations, keep APIs for frontier capability, and budget for the governance you inherit from the vendor.
Sources & methodology
- Hugging Face: openai/gpt-oss-120b model card and openai/gpt-oss-20b: licence, parameter counts, memory fit
- Meta: Llama 4 Community License Agreement
- NVIDIA: DGX Spark: 128GB unified memory, models up to 200B parameters
- Apple: Mac Studio technical specifications: M5 Ultra up to 512GB unified memory
- NVIDIA: H100 Tensor Core GPU
- Lambda: pricing (checked 20 Oct 2026)
- Methodology: specifications from vendor pages and model cards, checked 20 October 2026. The monthly GPU figure is our arithmetic (rate × 730 hours), not a vendor quote. No vendor paid for placement.
