Private LLM or OpenAI API in 2026: How We Run the Math

A finance director at a 600-person manufacturer called us in March 2025. "We crossed $48,000 a month on OpenAI API. Procurement is asking why."

That's the conversation in 2026. Not GDPR. Not data sovereignty. Not any of the 2024-era talking points. The 2026 question is much simpler: when does the math break?

That manufacturer became our first private LLM client. We have shipped 10 private LLM deployments since March 2025. Everything that follows is what we learned across those projects.

What changed in 2026 that didn't matter before

Two things flipped this year.

First, Qwen 3 32B and DeepSeek V3 are now within 4 to 7 percentage points of GPT-5 on most enterprise benchmarks. We finally crossed the line where "the open model is good enough" is a defensible answer, not just a thrifty one. In 2024 you had to apologize for using an open-weight model. In 2026 you do not.

Second, a single H100 dropped from $35,000 to $24,000 on the secondary market by Q1 2026. NVIDIA's H200 is shipping in volume now. Inference cost on owned hardware is closer to $0.0008 per 1,000 tokens. OpenAI's GPT-5 is at $4 per million input tokens. Run the math and the breakeven is around 2.5 billion tokens per month.

When does private actually win?

We use three filters in this order.

Filter 1. Volume. If you're below 200 million tokens per month, do not build. The API economics will beat you and the operational tax of running your own GPUs is real. A single A100 idle 14 hours a day still costs $400 in power and amortization.

Filter 2. Variance. If your token consumption is bursty, the API wins because it scales to zero. Private LLM hardware does not. We've seen healthcare and finance use cases where the entire monthly volume happens in the 3 days around month-end close. Private would mean 27 days of expensive idle GPUs.

Filter 3. Data class. This is where 2024 thinking was right but for the wrong reason. The risk is not OpenAI looking at your data. It's that your security team cannot audit what they cannot see. We've seen one client get a SOC 2 finding specifically because the auditor couldn't independently verify how the API call's payload was logged on OpenAI's side. The finding wasn't "data left the perimeter." It was "we cannot prove what happened to it."

The number people forget to count

People run the math on inference cost and skip the engineering cost. Our internal benchmark: a serious private LLM stack needs 1.5 senior engineers to keep alive. Vector store, eval harness, drift monitoring, prompt versioning, rate-limit and queueing. At a $220k loaded cost per engineer, that's $330k a year before you ship a single token.

If your inference savings vs. OpenAI API are less than $400k a year, private is a money-losing proposition once you count the people. We tell most of our mid-market clients the honest answer: stay on the API and revisit in 12 months.

What we got wrong on the manufacturer's first deployment

Back to the manufacturer. We sized their GPU cluster off their existing OpenAI API consumption. Rookie mistake.

API consumption was artificially low because their developers had been rate-limiting themselves to control costs. The moment we removed the cost ceiling, internal usage climbed about 25% within six weeks. That sounds modest until you realize we had sized for the existing volume with no headroom. Latency on the single H100 we had stood up started missing SLA on weekend batch runs. We had to add a second H100 on credit while the procurement team caught up.

The lesson: 25% is the floor, not the ceiling. Once a private LLM goes live and developers stop self-policing, demand keeps climbing as new use cases get unlocked. We now bake a 50% capacity headroom into every sizing exercise and tell clients up front to expect it.

The contrarian take

We don't deploy private LLMs to "save money." That's a backward framing. We deploy them when the API has become the bottleneck on creativity. When developers stop building features because they're scared of the bill. When product managers reject ideas in planning because "the AI cost would kill margins." That's the real moment private wins. Not when you cross the cost-curve breakeven on a spreadsheet.

The 2026 stack we ship

Most of our private LLM deployments share a common spine.

• Model. Qwen 3 32B Instruct. We benchmark every new client against this baseline first. For RAG, structured extraction, code review, and long-context summarization, it lands within a few percentage points of GPT-5 at zero per-token cost. We swap in Qwen 3 235B MoE for the rare client whose accuracy floor is non-negotiable.
• Runtime. Ollama. Simpler operationally than vLLM or TGI for teams that don't want to staff a full model-serving SRE function. Trade-off is some throughput at very high concurrency, but for the 200M to 2B tokens-per-month band that most of our clients sit in, Ollama is the right answer.
• GPU. Single H100 baseline. We scale to 2x H100 once SLA latency tightens or concurrency climbs past about 30 simultaneous users.
• Retrieval, eval, observability. Depends on the client's existing stack. pgvector if they're already on Postgres, dedicated vector DB if they want one. Eval is where the most variance lives. Some clients invest heavily, others run on vibes for the first six months and we have to retrofit it.
• Fallback. OpenAI or Anthropic API kept on for the top 5% of "this needs frontier-model quality" queries. Hybrid is almost always cheaper than going pure-private or pure-API.

FAQ

Is OpenAI API safe for healthcare data in 2026?

It depends on your BAA. OpenAI signs HIPAA BAAs with enterprise customers. The bigger problem is auditability. If your SOC 2 or HIPAA auditor needs to independently verify what happens to the payload after it leaves your network, the API is hard to defend. Private LLM gives you that audit trail.

What's the breakeven token volume for private LLM in 2026?

Roughly 2.5 billion tokens per month at current GPU prices and OpenAI's GPT-5 list pricing. Below that, the API wins. Above that, run the engineering-cost numbers before you commit.

Can we run Qwen 3 32B on a single H100?

Yes, comfortably. At FP16 the model lands around 64GB which fits inside an H100's 80GB. We do this for most of our small-to-mid deployments. You only need a second H100 once concurrency climbs past about 30 simultaneous users or you start chaining long-context calls back to back.

How long does a private LLM project take from kickoff to production?

10 to 14 weeks for a clean greenfield deployment. Add 4 to 6 weeks if you need to integrate with an existing identity provider, data lake, and eval pipeline. The longest pole is almost always procurement on the GPU hardware.

Should we wait for the next generation of models before deciding?

No. Both Anthropic and Meta are on roughly 6-month release cycles now. If you wait for the next model, you'll wait forever. Pick a deployment pattern that lets you swap the underlying model in 2 weeks. We use a thin abstraction layer specifically for this.

If you're working through this decision and want a sanity check, we do a free 90-minute architecture review for serious enterprise teams. Email [email protected] with your monthly token volume and we'll either tell you to stay on the API or send you our private-LLM sizing worksheet.