Someone on your team has already priced out an RTX 6000 Ada workstation. Maybe it came from a vendor pitch, maybe from a forum thread about running Llama locally, but the number that stuck was somewhere between CHF 15,000 and 30,000, spent once, with no monthly invoice after. That’s the pitch for a local inference cluster. It’s also not quite the arithmetic.
This article is for the team that has already decided, for reasons of data residency, contract language, or professional secrecy obligations, that inference needs to happen on hardware they control. If you’re still weighing whether to go on-prem at all, our companion piece on who actually builds on-prem AI covers that decision: the skills gap, the data-location question, and when the API still wins. This one assumes you’ve made that call and walks through what a local inference cluster actually is once you buy it. And once a workflow is running on it, whether that workflow itself is well designed is a separate question, one we cover in how we re-architect processes around agents rather than bolting AI onto whatever process already exists.
A CHF 15,000 to 30,000 local inference cluster costs roughly CHF 671 a month once you amortise the hardware over three years and add realistic Swiss electricity. Set against a blended API rate of about CHF 4.60 per million tokens for real agent traffic, that fixed cost only pays for itself around 146 million tokens a month mathematically, and closer to 250 to 300 million once you account for how sub-linear concurrency scaling actually works rather than the clean multiplication a naive projection assumes. A 20-person team making 50 agent calls a day, an illustrative anchor rather than a market statistic, reaches only about 66 million tokens a month. The honest read is that local inference at this tier is fundamentally a data residency decision. Don’t buy it expecting to save money, because for most SMBs the arithmetic below doesn’t support that.
What CHF 15,000 to 30,000 Actually Buys
Three hardware paths sit inside this budget band in mid-2026, and a fourth is worth knowing about precisely because it doesn’t.
RTX 6000 Ada or the newer RTX Pro 6000 Blackwell, built into a workstation chassis (Lenovo ThinkStation, Dell Precision), is the closest thing to a default. The Ada card carries 48GB of VRAM at 300W and lands a complete workstation around CHF 15,000 to 18,000. Step up to the Blackwell card’s 96GB at 600W and the same build pushes toward CHF 25,000 to 30,000, the top of this tier.
A single RTX 5090 is the budget consumer-grade option: 32GB of VRAM, a complete build around CHF 8,000 to 12,000. It lacks ECC memory and Quadro-class driver support, but plenty of practitioners run it anyway once they’ve tuned their own stack.
A refurbished RTX A6000 lands in the same CHF 8,000 to 12,000 band as enterprises cycle out Ampere-generation hardware in 2026, and runs 10 to 15 percent slower than the Ada equivalent on inference while offering the same 48GB.
The Mac Studio M4 Max is the outlier, and not in the direction you’d expect. A 64GB unified-memory configuration costs roughly CHF 3,000 to 4,500, dramatically less than any of the NVIDIA-based paths above and not a pricier, quieter option at the top of the budget band. It needs no separate GPU to source, runs inference through Apple’s MLX framework instead of CUDA, and its low cost matters more than its silence.
The number that matters more than price here is power: a Mac Studio draws around 60 watts total, against 600 watts to a full kilowatt for a complete NVIDIA-based workstation, consistent with independent power-consumption benchmarking of inference hardware. That decides where the box can live. The Mac Studio sits on a desk and stays silent; the NVIDIA builds need a dedicated 16-amp circuit and real ventilation. They also run at 50 to 65 dBA under load, loud enough to notice over a phone call. A handful of teams pay CHF 2,000 to 5,000 more for a liquid-cooling loop to avoid that.
None of these need a server room, just somewhere with power and airflow that isn’t someone’s desk in the NVIDIA cases, plus someone technical enough to rack the hardware and install the drivers.
Which Open-Weight Models Fit This VRAM Envelope
The available VRAM decides which model fits. At 48GB, the ceiling for the Ada and A6000 cards, a 32-billion-parameter dense model runs comfortably at Q5 or Q6 quantisation with room for a decent context window. Qwen 3.5’s 27B variant and similar Mistral dense models sit here. Pushing a 70B model into 48GB at Q4 is technically possible but leaves little headroom for concurrent users or long context, so treat it as an edge case rather than a default.
Step up to 96GB on the Blackwell card or 64GB of unified memory on the Mac Studio, and 70B-class models at Q4 quantisation become the comfortable default. Llama 3.3 70B and Qwen 2.5 72B both fit with room for a real context window and a handful of concurrent requests. Meta’s Llama 4 Scout, a mixture-of-experts model with 17 billion active parameters but 109 billion total, also fits on the larger cards once quantised and holds up well on long-context reasoning tasks.
What doesn’t fit matters as much as what does. Mistral Large 3, one of the strongest fully open-weight reasoners in mid-2026, carries 675 billion total parameters and needs two to four H100-class GPUs even at aggressive quantisation. That is firmly enterprise-cluster territory, and no amount of quantisation trickery brings it down to a single workstation card.
On tool calling specifically, Qwen 3.5’s smaller variants lead independent evaluations of open-weight tool-calling performance. Benchmark leaderboards, though, do not reliably predict how a model behaves against your own toolset at the quantisation level you intend to run, using your own chat template. Test your agent’s tool definitions against that exact stack before committing a model to production.
The Honest Throughput Ceiling
A single workstation GPU does not behave like an API endpoint with unlimited concurrency. On a 70B model at Q4 quantisation, the RTX 6000 Ada and A6000 deliver roughly 40 to 50 tokens per second to one user, and hold up for 4 to 10 concurrent users before response times start climbing past a few seconds. The closest published concurrency benchmark runs a 27B model on the pricier Blackwell card rather than this piece’s Ada/A6000 build. It measured 46 tokens per second for one user at 1K context, dropping to roughly 26 to 30 tokens per second per stream at three concurrent users. Aggregate throughput was still climbing rather than collapsing.
Past 10 to 15 concurrent requests the KV cache, which is the GPU memory holding active conversation state, fills up. The server then swaps that state out to system RAM to keep going. Every swap costs latency, and past a certain point the whole system slows down for everyone connected to it rather than failing gracefully for just the newest request.
The Mac Studio trades speed for silence: 20 to 28 tokens per second for a single user on a 70B model. That ceiling likely caps it at two to five concurrent users, since nobody has published rigorous concurrency benchmarks for it yet. The RTX 5090 runs a touch faster than the Ada card for a single user, in the 60 to 70 tokens per second range, but shares the same concurrency limits.
This is capacity for one team’s internal tool. It falls well short of what a customer-facing support widget or a call center needs, and treating it as such is the fastest way to disappoint whoever asked for it.
The Serving Layer: vLLM, SGLang, or Ollama
Three inference servers cover essentially every workstation deployment in 2026, and picking between them matters more than picking between GPUs.
vLLM is the default for anything with more than one user. Its core trick, PagedAttention, pools GPU memory for active conversations instead of reserving a fixed block per request, cutting memory waste from roughly 60 to 80 percent down to about 4 percent and roughly doubling to quadrupling throughput over naive batching on the same card. It is the production standard at Meta, Mistral AI, Cohere and IBM, a reasonable signal for infrastructure nobody wants to babysit.
SGLang earns its place when your agents emit structured output, JSON tool calls or formatted extraction fields, repeatedly in a tight loop. It is purpose-built for constrained generation and measures roughly 29 percent higher throughput than vLLM on workloads where requests share context, the classic pattern in a tool-calling agent chaining several calls per turn.
Ollama deserves its popularity for exactly one job: getting a model running in under five minutes to see if it’s worth pursuing. It has no fine-grained resource controls and doesn’t scale past single-user use, and deploying it as the serving layer for a shared team tool is the single most common way people leave throughput on the table at this hardware tier.
Start with vLLM unless a structured-output-heavy workload points you toward SGLang. Both are open source with active communities, and switching between them later is a configuration change rather than a rebuild.
Quantisation: What Q4 Actually Costs You
At this hardware tier, VRAM is almost always the binding constraint. That makes Q4 quantisation, specifically Q4_K_M, the default rather than a compromise. It cuts memory use by roughly 75 percent and speeds up inference 3 to 4 times, the only reason a 70B model fits on a single card at all.
That saving is not free. Q4 costs roughly 1 to 2 percent of measurable quality on models 30B and larger, a loss most agent workloads won’t notice in practice. On models under 7B, the same quantisation costs 5 to 10 percent. That shows up as a genuinely worse JSON parse rate or a subtly wrong tool argument. Those failures are easy to miss in a demo and expensive to miss in production.
FP8 is the gentler alternative where the hardware supports it natively, chiefly on Blackwell-class cards. Quality loss sits closer to 0.3 to 0.5 points on standard benchmarks. It doesn’t free up the VRAM that Q4 does, so most workstation deployments run Q4 out of necessity rather than choice.
Whichever quantisation you land on, test it against your own chat template and tool schema before deploying it. A model that parses JSON perfectly at full precision can start dropping fields at Q4. The only way to know is to run your own evaluation set through the exact stack you plan to ship.
The Economics: Fixed Cost Against Variable Usage
Here is the arithmetic behind that summary.
Take the RTX 6000 Ada workstation as the reference build, since it sits mid-band. CHF 18,000 amortised over 36 months costs CHF 500 a month in hardware alone. A roughly 1kW system run continuously draws about 720 kWh a month; at the ElCom commercial rate of CHF 0.237 per kWh, that’s CHF 171 a month in electricity. Total fixed cost lands around CHF 671 a month, in a band of roughly CHF 650 to 700 depending on your local tariff, whether the box sits idle overnight or runs flat out.
Set that against a blended API rate of roughly CHF 4.60 per million tokens for real agent traffic, based on current frontier-model API pricing blended across input and output tokens in a realistic four-to-one ratio, the pattern a typical retrieval-backed agent call actually produces. That sits deliberately above the cheapest rate on the market, since it reflects a realistic mix for agent workloads instead of a best-case input-only figure.
Divide the fixed cost by that rate and you get the mathematical break-even point: roughly 146 million tokens a month. That assumes you can actually sustain it, and the concurrency data below shows that’s harder than it looks: throughput scales well below linearly as more users share one GPU, pushing the realistic threshold higher than the bare math suggests.
Utilisation decides your real cost per token far more than the hardware you buy. The multi-user rows below use a range, explained underneath the table:
| Workload (illustrative, RTX 6000 Ada reference build) | Monthly tokens | Fixed cost | Cost per million tokens |
|---|---|---|---|
| 1 user, continuous at 45 tokens/second | ~117 million | CHF 671 | CHF 5.74 |
| 10 concurrent users, continuous (sub-linear scaling) | ~207 to 518 million | CHF 671 | CHF 1.30 to 3.25 |
| 1 user, realistic 15% utilisation | ~17.5 million | CHF 671 | CHF 38 |
| 10 users, realistic 30% utilisation (sub-linear scaling) | ~62 to 155 million | CHF 671 | CHF 4.30 to 10.80 |
Ten concurrent users do not deliver ten times one user’s throughput: they compete for the same GPU memory and compute rather than each getting a dedicated slice. A published concurrency benchmark on a comparable card put the realistic aggregate multiplier at roughly 1.8 to 4.4 times a single user’s rate rather than 10 times, the basis for the ranges above.
That correction reshapes each row. The continuous single-user row still falls short of the API running flat out, since 117 million tokens a month sits under the 146 million break-even. The continuous ten-user row is the genuinely strong case for local hardware, at CHF 1.30 to 3.25 per million tokens even at the conservative end. The ten-user, 30 percent utilisation row is the one to read carefully. Its band, CHF 4.30 to 10.80 per million tokens, straddles the API rate rather than beating it outright. A moderately used shared tool is closer to a coin flip against the API than a clear win.
That is also where the realistic 250 to 300 million threshold in the summary above comes from: beating the API with real margin means running close to the continuous ten-user pattern rather than the lighter thirty-percent-utilisation pattern most shared tools actually see. A 20-person team making 50 agent calls a day at roughly 3,000 tokens a call, illustrative rather than a market survey, reaches about 66 million tokens a month; a 50-person team at the same pace reaches about 165 million. Both sit well short of a comfortable margin over break-even, which is the point: token volume alone rarely gets a typical SMB there. This is the same trade-off our TCO model for custom versus platform AI agent infrastructure walks through in more general terms.
The Mac Studio’s economics look different once its price is corrected. CHF 4,000 amortised over 36 months is CHF 111 a month. Add its roughly 60-watt draw and total fixed cost lands near CHF 120 a month. That puts its mathematical break-even at about 26 million tokens, a volume a single continuous user running its 20 to 28 tokens per second clears alone. The pricier NVIDIA build cannot manage that even at full saturation. The trade-off is real: less VRAM headroom for concurrent users and a throughput ceiling that won’t serve more than a handful of people. For a single analyst or a two-person team with a genuine data-residency requirement, it’s the more defensible starting point of the four.
None of this includes the cost of actually building the agent workflow that calls the model, which follows the same tiers regardless of which infrastructure it talks to.
What Breaks After the Install, and Who Owns It
Buying the hardware is the easy part. What determines whether the cluster is still useful in month six is a set of unglamorous, ongoing tasks that nobody budgets for at purchase time.
Setup takes one to two weeks for someone who has done this before: assembling the hardware, installing drivers, standing up vLLM or Ollama, and pulling down a 70GB-plus model file, which alone can take hours over a typical office connection. Simpler than standing up a Kubernetes cluster, but not plug-and-play.
New model versions arrive roughly every quarter. Each one needs downloading, evaluating against a test set of real questions and known-good answers, and a deliberate decision on whether to swap. Skip that step and models drift silently: the same prompts produce subtly worse answers, and nobody notices until a user complains.
There is no redundancy at this tier. A single workstation is a single point of failure: if the GPU dies, you’re down until you source a replacement or fail over to an API, typically a day or two, unless you’ve built a fallback path in advance.
Monitoring is ad hoc by default. Enterprise clusters get proper observability dashboards; a workstation in a cupboard usually gets nothing beyond a ping check, which tells you about uptime and nothing about whether the answers are still good, the metric that actually matters.
Put together, this runs roughly 20 to 30 percent of one person’s ongoing time, every month, for as long as the cluster stays in production. It doesn’t need a dedicated hire, but it does need someone named, with time carved out before the hardware arrives rather than after the first embarrassing wrong answer. The operational risk here echoes our broader look at AI agent security risks: an unmonitored, under-owned deployment often becomes a bigger exposure than the infrastructure choice meant to reduce risk in the first place.
When This Is the Right Call
A few worked examples show the pattern. A 15-person Swiss accounting firm at 100 million tokens a month, with no client-confidentiality constraint beyond ordinary discretion: the API is cheaper, and nobody can spare 20 to 30 percent of an engineer’s time. A 25-person legal practice at 150 million tokens a month clears the mathematical break-even but falls short of the realistic 250 to 300 million threshold, so the numbers alone lean toward the API even here. A 40-person financial services firm at 300 million-plus tokens a month, with real regulatory constraints and someone already on staff to own the infrastructure, is the clean yes.
That middle case is where most genuine local-inference decisions in Switzerland actually happen. Volume alone rarely clears the break-even bar. The pull toward local hardware comes instead from the revised Swiss Federal Act on Data Protection, GDPR where EU data is in scope, professional secrecy under Article 321 of the Swiss Criminal Code, FINMA expectations for regulated financial services, or a data-location clause a client’s legal team already wrote into the contract.
Buy the workstation when: data residency is a hard requirement a cloud API genuinely cannot satisfy, your internal use case can tolerate 4 to 10 concurrent users and a few seconds of latency, and someone on the team can own model selection and evaluation as an ongoing job rather than a one-time task.
Skip it when: nothing regulatory or contractual is forcing the decision, your monthly volume sits in the tens of millions of tokens rather than the hundreds, or you need customer-facing uptime and concurrency that a single card was never built to deliver. In every one of those cases, renting GPU capacity from a Swiss-hosted provider or staying on a public API costs less and asks less of your team, at least until volume and regulatory pressure both point the same direction.
Frequently asked questions
How much does a local inference cluster cost to run each month?
A workstation-class local inference cluster in the CHF 15,000 to 30,000 range costs roughly CHF 671 a month once you amortise the hardware over three years and add Swiss commercial electricity rates. That figure barely changes whether the box sits idle or runs continuously, which is why the deciding factor for beating an API subscription is utilisation rather than purchase price.
Is a local inference cluster cheaper than using an API like Claude or GPT?
Only at high, sustained volume. Mathematical break-even against a blended API rate of roughly CHF 4.60 per million tokens sits around 146 million tokens a month, and the realistic threshold once you account for how throughput actually scales with concurrent users is closer to 250 to 300 million. A 20-person team making 50 agent calls a day reaches only about 66 million tokens a month, well under either figure, so for them the API remains the cheaper option.
How many users can a CHF 15,000 to 30,000 GPU workstation actually support?
Realistically 4 to 10 concurrent users on a 70 billion parameter model at 4-bit quantisation, with a single user getting 40 to 50 tokens per second. Past 10 to 15 concurrent requests, the GPU runs out of memory for tracking active conversations and response times climb sharply. This tier suits an internal tool for one team rather than a customer-facing chatbot handling many simultaneous conversations.
Which open-weight models actually fit on this hardware?
At 48 to 96GB of VRAM, dense and mixture-of-experts models in the 30 to 70 billion parameter class fit at 4-bit quantisation, including Llama 3.3 70B, Qwen 2.5 72B, and the smaller Qwen 3.5 variants built for tool-heavy workflows. Frontier open-weight models such as Mistral Large 3, designed for multi-GPU enterprise clusters, do not fit on a single workstation card at any quantisation level.
When does local inference make sense for a small or midsize business?
Mainly when data residency is a hard requirement driven by Swiss data protection law, professional secrecy rules, financial sector oversight, or a contractual data-location clause, and a public API genuinely cannot satisfy it. Outside that case, the cost and maintenance burden rarely beat renting GPU capacity or paying per token, since most businesses never reach the sustained volume where ownership pays for itself.