Enterprise vs Consumer Inference: Local LLMs Have Split Into Two Segments, and Only One Has a Name

Every few months someone shows me a benchmark and asks whether local models are “ready yet.” The question is unanswerable as stated, because the person asking is usually thinking about one of two completely different problems and doesn’t know which.

Problem one: I want a model running on the laptop in front of me, answering my prompts, with nothing leaving the machine. Batch size one. Audience of one.

Problem two: I want a model running on hardware my company controls, serving a few hundred concurrent users, with an SLA attached.

Both get filed under “local AI.” They share a motive and almost nothing else. The engineering constraints are not similar, they are inverted. And the advice written for one of them is actively harmful when applied to the other, which is most of what I see on LinkedIn.

The second problem has a name. We call it on-prem inference, private inference, sovereign AI, and enterprise vendors have built a whole vocabulary around it. The first one doesn’t. It gets called “running LLMs locally,” which describes a hobby rather than a category.

So let me propose one.

Consumer inference

#

Consumer inference (n.) — LLM inference run to completion on hardware an individual already owns, for an audience of one.

Three properties define it, and each one flips an assumption that holds in the datacenter.

Batch size is one. There is no second request to amortize a weight read against. In a serving cluster you pull a tensor from memory once and use it for sixty concurrent sequences. On a laptop you pull it once and use it once. Every technique that depends on batching — continuous batching, PagedAttention’s page sharing, most of what makes vLLM good — has nothing to work with.

It is memory-bound, not compute-bound. Decode reads the model’s active weights for every single token generated. The ceiling is bandwidth, not arithmetic. This is why the TOPS numbers on chip datasheets are close to irrelevant for the thing people actually want, and why a phone with a 100-TOPS NPU and 60 GB/s of memory bandwidth generates tokens roughly as fast as its memory bus allows and no faster.

Capital cost is sunk. The machine was bought anyway. The marginal cost of a token is electricity, which for a single user is somewhere between negligible and unmeasurable. This is the property that makes the whole segment strategically interesting, and it is the one enterprise buyers consistently underweight because their mental model is a per-token invoice.

Enterprise inference is the mirror image on all three: batch N, compute- and interconnect-bound at scale, and dominated by capex and utilization economics. Same motive — control — opposite problem.

Once you separate them, the last eighteen months make much more sense.

What actually changed on the consumer side

#

The unlock was not better quantization. It was sparsity, exploited properly.

Mixture-of-experts models activate a small fraction of their parameters per token. A 35B model might light up 3B of weights on any given forward pass. The obvious inefficiency of holding the other 32B in RAM has been obvious for two years. What changed is that people built runtimes that stop doing it.

The current best expression of this on Apple Silicon is a small lineage of Swift projects: TurboFieldfare by Andrey Mikhaylov, which pioneered the approach and documented over a hundred experiments behind it, and its derivative Mference, which extended it to more architectures. The mechanism is straightforward once stated: keep the shared core and the KV cache resident in memory, lay out each layer’s routed experts on disk as fixed-stride, page-aligned blobs, and pread only the eight experts the router selects for the current token.

The numbers are the point. Gemma 4 26B in roughly 2 GB of memory. Qwen 3.6 35B in roughly 1.45 GB, because its gated-DeltaNet layers keep a fixed-size recurrent state instead of a growing KV cache. Measured decode of 18.8 to 23.1 tok/s on an M5, and 5.1 to 6.3 tok/s on an 8 GB M2 Air — a machine Apple sold as the entry-level laptop. The model never has to fit in RAM. Only its working set does.

The same idea has landed in the mainstream runtimes. llama.cpp merged SSD streaming of routed experts with a per-layer slab cache, on top of the older CPU-offload flags. KTransformers out of Tsinghua’s MADSys Lab keeps attention and KV on the GPU while pushing experts to system RAM, and runs 671B-class models on a single 24 GB card; its June 2026 release covers DeepSeek-V3/R1, Kimi K2.5, GLM-5.2, Qwen3-Next and MiniMax. The research literature around it is now thick — MoE-Infinity’s sparsity-aware expert cache, HOBBIT’s mixed-precision offload, half a dozen speculative-prefetch papers.

Apple’s own MLX is, oddly, behind here. The open issue asking for SSD expert offload so a 395 GB GLM-5.2 checkpoint can run on a 128 GB Mac is still open. There is a real gap between what the first-party framework does and what three people in a Swift repo have already shipped.

The honest limit, which the Mference docs state plainly: decode is currently around 53% expert-read wait, serialized with GPU work. Half the wall-clock time is the SSD. That is not a defeat, it is a queue — overlapping I/O with compute is engineering that will get done.

The hardware caught up at the same time

#

The other half of the story is that a specific hardware category appeared and then got crowded fast: single-chip machines with a large pool of unified memory addressable by both CPU and GPU, with no copy step between host and device.

Three vendors converged on the same architecture within about a year. That is not a coincidence, it is the market pricing in the fact that bandwidth-per-dollar to a large memory pool is the binding constraint for single-user inference, and that discrete VRAM is a bad way to buy it.

The DGX Spark price move from $3,999 to $4,699 is worth noticing for a different reason. It went up because of memory supply. Consumer inference is now competing for the same DRAM the hyperscalers are buying, which tells you the segment is no longer a rounding error.

What you get for it: something in the range of 70–85% of frontier-model quality on a broad set of tasks, at zero marginal cost per request. Qwen 2.5 32B lands 83.2% MMLU running entirely on a Mac Studio. For summarization, extraction, classification, structured output, and any workflow where you control the prompt tightly and the output is bounded, that gap has stopped mattering.

The phone tier is a different animal again

#

Consumer inference on a phone is the same category under harsher physics, and 2026 was the year the platforms opened up.

At WWDC 2026 Apple opened the Foundation Models framework to third-party providers through a public LanguageModel protocol, shipping in iOS 27 and macOS 27. Any model vendor can now implement it and plug in. Apple simultaneously shipped Gemini-backed models inside its own framework via Firebase, which is a genuinely strange sentence to write and tells you how much the on-device layer has become plumbing rather than differentiation. Android’s equivalent path is AICore and Gemini Nano through the ML Kit GenAI APIs.

On silicon, Qualcomm’s Snapdragon 8 Elite Gen 5 puts roughly 100 TOPS in the Hexagon NPU and claims 220 tok/s decode at 32K context, with 38% more memory bandwidth than its predecessor. MediaTek’s Dimensity 9500 matches the TOPS number with NPU 990 and takes a more interesting swing: compute-in-memory, native 1.58-bit ternary weight support for BitNet-style models, a claimed 33% power reduction, and 128K on-device context.

MediaTek is aiming at the right target. The bottleneck on a phone is that mobile memory delivers 50–90 GB/s against 2–3 TB/s for a datacenter accelerator, a 30–50x gap that decides decode speed because weights get read every token. You do not close that by adding TOPS. You close it by moving fewer bits — ternary weights, table-lookup kernels, aggressive quantization — or by moving them a shorter distance, which is what compute-in-memory means.

The reflex to put everything on the NPU is also being questioned by people measuring carefully. Several 2026 papers find NPUs win on prefill and frequently lose to the GPU on decode, and that sustained-load thermal behaviour reorders the answer again. If you are shipping mobile AI, benchmark the stage, not the chip.

The tooling settled at the same time: ExecuTorch hit 1.0 GA in October 2025 and 1.1 in January 2026, with a 50 KB runtime and backends for Core ML, Qualcomm QNN/Hexagon, XNNPACK and MediaTek. There is now a boring, portable way to ship a model to a phone, which is what adoption actually requires.

Enterprise: the quiet repatriation

#

The enterprise segment moved less visibly and, in aggregate, more consequentially.

The share of enterprises using public cloud as the primary environment for production AI inference fell from 56% to 41% in a year. 56% now run or plan production inference on private cloud. And 77% of organizations report factoring an AI vendor’s country of origin into selection decisions, which is the sovereignty argument arriving as procurement policy rather than as a conference panel.

The economics stopped being a debate somewhere in the last twelve months. Self-hosted inference on a 70B model runs around $0.013 per 1,000 tokens against $0.15–0.60 through comparable APIs. Hardware breaks even in under four months at 30 million tokens per day, and above that threshold self-hosting saves 60–85% on inference cost alone. Those numbers assume you have the operational competence to run it, which is the real gate — but the arithmetic no longer requires a strategic justification.

In regulated verticals the cost argument is usually not even the one that closes the decision. Data residency, auditability of the inference path, and the ability to freeze a model version for the lifetime of a compliance regime tend to matter more than the invoice. What changed is that those requirements used to force a painful trade against capability, and now they mostly don’t.

The stack standardized too. vLLM for open flexibility or NVIDIA NIM for a supported package, sitting on a GPU tier, with an offline model registry, Kubernetes orchestration, and a gateway doing mTLS, RBAC and audit logging. Five layers, well understood, boring in the good way.

Here is the part that gets missed, and it is why the two-segment split is not pedantry: the consumer techniques do not transfer. SSD expert streaming is excellent for one user and destructive under concurrency. With batch size 1 the router picks eight experts and your LFU cache stays warm. With sixty concurrent sequences the routers collectively demand a large, shifting fraction of the expert pool every step, the cache thrashes, and you have converted a compute problem into a random-read storm. Enterprise inference goes the other way for exactly this reason: put everything in VRAM, batch hard, amortize.

Anyone telling an enterprise team that they can run a 400B model on a 24 GB card because they saw it work on a MacBook is giving advice from the wrong segment.

Where I actually stand

#

I think this is one of the more important structural shifts in how AI gets deployed, and I think the constraints that make people dismiss it — memory, bandwidth, quantization loss — are the kind of constraints engineering resolves rather than the kind it lives with. Eighteen months ago the claim that you could get useful tokens out of a 35B model on an 8 GB laptop would have been a joke. It is now a reproducible benchmark with a frozen protocol.

The consumer side is moving faster than enterprise right now, and it will keep doing so, because it has no procurement cycle, no compliance review and no vendor relationship to defend. Three people in a Swift repo shipped something Apple’s own framework hasn’t. That asymmetry is the whole story of the segment.

But my reason for caring is not the benchmarks.

I see both halves of this from different chairs. At A17 I oversee large machine learning implementations in regulated verticals, which is the enterprise problem at full weight — audit trails, data residency, model governance, the works.

The consumer half comes from my own ventures, from the agentic loops I keep building and breaking on my own hardware, and from helping smaller companies take their first steps. Those companies cannot afford a private cloud and will not be able to for years. They start small, on machines they already own, and the question of whether that path leads anywhere is not academic for them.

In both chairs the thing I am accountable for is not model quality. It is whether the capability is there tomorrow. Cloud inference providers are a dependency risk in the ordinary sense: pricing changes, models get deprecated on ninety days’ notice, rate limits appear, terms shift, capacity gets allocated to someone with a bigger contract. The infrastructure providers underneath them are a second, correlated layer of the same risk, and renting GPU capacity does not make it yours.

I like the big private clouds — for the comfort they provide. The issue is that the entire stack is more brittle than its marketing suggests, the concentration is historically unusual, and every technical leader I know has quietly accepted a single point of failure they would never accept in any other part of their architecture. We have collectively decided that a capability our AI products now depend on is fine to rent from four companies.

But at the end of the day — resilience and business continuity is my job. Not the vendor’s. If a workflow matters enough to build on, some version of it should survive its provider having a bad quarter — and increasingly, it can, because the fallback is no longer a toy. That is what consumer inference and its enterprise counterpart really buy: not cost savings, not privacy compliance, though you get both. Optionality. The ability to keep working when someone else’s roadmap changes.

That is worth naming, and worth building toward.

Sources and further reading: Mference · TurboFieldfare · llama.cpp MoE disk streaming PR · mlx-lm expert offload issue · KTransformers overview · MoE-Infinity · HOBBIT · WWDC 2026 Foundation Models · On-prem AI statistics 2026 · On-device LLM inference guide · Is Your NPU Ready for LLMs? · State of local AI 2026