Local Inference Runtimes

"Local inference" is three different jobs wearing one label

The phrase gets used for everything from a chatbot on a laptop to a forty-GPU rack serving an agent product, and the conflation ruins most tool discussions before they start. The field is really three jobs with different success metrics. Job one is personal and interactive: run a competent model on the machine in front of you, load in seconds, answer in seconds, cost nothing per token. Job two is workgroup and product serving: one or a few boxes, real concurrent load, throughput and latency percentiles under batch, an OpenAI-shaped API that agent frameworks can point at. Job three is fleet-scale serving: tensor and expert parallelism, prefill-decode disaggregation, day-zero support for each new frontier open-weights release, and cost per million tokens as the only number that matters.

The tools that dominate each job are different, and the most common mistake in this field is carrying a recommendation across job boundaries. Ollama's one-line install is a triumph of job-one design and the wrong answer for job three. TensorRT-LLM's kernel library is arguably the fastest thing on NVIDIA silicon and an absurd way to chat with a seven-billion-parameter model on a laptop. What gets miscategorised most often: GGUF quantization formats get treated as a llama.cpp exclusive (they are now a cross-runtime interchange), MLX gets treated as a llama.cpp alternative on Mac rather than a parallel universe with its own weights format and community, and "local" gets conflated with "small" — a two-node GB300 NVL72 deployment running SGLang is as local as it gets, and it is nobody's desktop.

This dossier covers the runtimes I have read closely enough to hold positions on: llama.cpp and the convenience layers built on it (Ollama, LM Studio), the serving tier (vLLM, SGLang, TensorRT-LLM, and Text-Generation-Inference on its way out), and Apple's MLX as the platform-specific parallel track. Every claim below comes from the project READMEs, licence files, and documentation pages listed in Resources, fetched for this piece.

Everything on a consumer machine eventually flows through llama.cpp

llama.cpp is the substrate of local inference, and its README makes the design thesis explicit: LLM and vision-model inference in plain C/C++ with no dependencies, state-of-the-art performance across an extraordinary range of hardware. The backend table is the widest in the field — CUDA for NVIDIA, HIP for AMD, Metal as a first-class citizen on Apple silicon, Vulkan and SYCL for everything else, plus OpenCL, OpenVINO, CANN for Ascend NPUs, IBM zDNN, MUSA for Moore Threads, RISC-V vector extensions, and WebGPU. It implements integer quantization from 1.5-bit to 8-bit, and — the feature that quietly rescues more projects than any benchmark — CPU-plus-GPU hybrid inference, which runs models whose total size exceeds VRAM by spilling some layers to system memory.

That hardware breadth is the strategic fact. NVIDIA and AMD GPU support is table stakes; the reason llama.cpp is in every phone app, every Raspberry Pi project, and every "I ran an LLM on my fridge" post is that it compiles almost anywhere and degrades gracefully. The GGUF format it anchors has become the interchange standard for quantized open weights: convert once, run on the CPU stack, in LM Studio, in Ollama, and (per vLLM's own feature list) now loadable directly in vLLM too. The MIT licence is verified in the README badge, and there is no commercial entity taking a cut — ggml-org maintains it as a dependency-free community project.

Recent development has softened its rough edges. The project now ships llama cli and llama serve front-ends that pull a GGUF straight from Hugging Face (llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF) and expose an OpenAI-compatible API server with a built-in web UI — functionality that used to require one of the convenience layers below. GBNF grammars give constrained decoding at the engine level. My position: llama.cpp is the single correct answer for job one if you are willing to touch a command line, and the engineering foundation everyone else in this dossier stands on. Its weakness is not capability but surface area — server tuning, model availability curation, and multi-model management are yours to own.

Convenience layers win desktops because they hide the parts that hurt

Two products wrap the engine for people who do not want to own it, and they make opposite trade-offs.

Ollama is MIT-licensed (licence file verified) and states plainly in its README that its supported backend is the llama.cpp project. The one-line install scripts for macOS, Windows, and Linux, the model registry with ollama run gemma4 ergonomics, the REST API on port 11434, and first-party Python and JavaScript libraries made it the default way non-infrastructure people run open models. The recent direction is interesting: ollama launch claude and ollama launch openclaw wire a local model into coding agents and personal-assistant harnesses, which is a recognition that job-one users increasingly want a local backend for agents, not a chat toy. The client ecosystem documented in its README — dozens of chat UIs, editor integrations, and language SDKs from LiteLLM to Spring AI — is the widest in the field. The trade: Ollama wraps the engine, so engine-level capabilities arrive on Ollama's schedule, and server-grade serving behaviour (sophisticated batching, fine-grained KV-cache control) is not what its abstraction is for. My position: the right default for personal machines and for giving coding agents a local backend; the wrong tool for anything with concurrent production load, where you want the engine's own server or a serving runtime.

LM Studio is the polished desktop application: proprietary and closed-source (the one non-open entry in this dossier — free to download, with no licence file to audit, which is exactly the difference from everything else here), available for macOS, Windows, and Linux. Its docs describe the two-engine architecture honestly: it runs llama.cpp with GGUF weights everywhere, and Apple's MLX on Apple Silicon, with engine runtimes installable and updatable from inside the app. Around the chat interface it ships an OpenAI-compatible API plus a beta REST API, Python and TypeScript SDKs, an lms CLI, MCP client support, headless operation via "llmster" for servers and CI, and an LM Link feature for routing workloads across machines. My position: the best GUI experience in local inference and the cleanest way to hand a non-engineer a private model with an API endpoint attached. The closed-source core is the price; the fact that its engines are open (llama.cpp, MLX) means the inference behaviour is auditable even when the application is not.

The serving tier is a two-horse race with a very fast third horse

Job two and job three belong to GPU serving runtimes, and here the field has consolidated hard.

vLLM is Apache-2.0 (licence file verified), born at UC Berkeley's Sky Computing Lab, and now claims over 2,000 contributors. Its PagedAttention paper — published at SOSP 2023 and cited in the README — is the founding insight of the modern serving tier: manage attention key-value memory like an operating system manages pages, and throughput stops being an accident. The current feature list reads as a serving-tier checklist: continuous batching, chunked prefill, prefix caching, FP8 to INT4 quantization including GGUF loading, speculative decoding (n-gram, EAGLE, DFlash), tensor/pipeline/data/expert/context parallelism, structured outputs via xgrammar, multi-LoRA, an OpenAI-compatible API server plus Anthropic Messages API and gRPC, support for 200-plus Hugging Face model architectures, and hardware beyond NVIDIA — AMD, Intel, TPUs, Gaudi, Ascend, Apple Silicon. My position: vLLM is the default answer for job-two and most of job-three — the broadest model coverage, the deepest ecosystem, and the least surprise. Its sheer feature velocity means you should pin versions and read release notes like a dependency, because you are effectively tracking a fast-moving distribution.

SGLang is the performance challenger that has stopped being a challenger. Apache-2.0 (licence file verified), hosted under LMSYS, its README claims deployments on more than 400,000 GPUs serving trillions of tokens daily, with adopters from xAI to Cursor to the major clouds. The technical signature is RadixAttention — prefix caching built around a radix tree so shared prompt prefixes are computed once — alongside a zero-overhead batch scheduler, prefill-decode disaggregation, and large-scale expert parallelism; its news list documents frontier-scale results on GB200 and GB300 NVL72 racks and a day-zero support cadence that has covered DeepSeek-V4 and Kimi K3 within days of release. Two structural facts set it apart: it is the proven rollout backend for reinforcement-learning post-training (AReaL, Miles, slime, verl, Tunix per the README), and it won an a16z Open Source AI Grant. My position: for frontier-model serving — big MoE models, RL rollout loops, racks — SGLang is frequently the first engine to work and the fastest measured; for heterogeneous model gardens, vLLM's breadth still wins. Tracking both is not indecision; the serious serving tier genuinely runs both.

TensorRT-LLM is NVIDIA's answer, Apache-2.0 with the licence badge in the README, and per its news log it became "fully open-source, with developments moved to GitHub" in March of this year. It is architected on PyTorch now, with a high-level Python LLM API, and its optimization list is the aggressive end of the field: specialized kernels for attention, GEMM, and MoE; prefill-decode disaggregation; wide expert parallelism over NVLink; speculative decoding; sparse and skip-softmax attention for long context; weight-stripped engines. The performance claims are Blackwell-specific and large — Llama 4 above 40,000 tokens per second on B200s, past 1,000 tokens per second per user. It integrates with NVIDIA Dynamo and Triton Inference Server, and — notable for a vendor project — it publishes a formal deprecation policy with a three-month migration window. Where it stops: NVIDIA hardware only, and the engineering posture assumes you want the fastest possible inference more than you want portability or a short stack. My position: the right call at real scale on Blackwell and Hopper when GPU cost per million tokens is the KPI, and the wrong call for anything heterogeneous; every week you spend on it is a week inside NVIDIA's roadmap, which is a fine place to be until it is not.

Text-Generation-Inference is the cautionary tale, and its README says so itself. The caution box at the top of the Hugging Face project's README states that TGI is now in maintenance mode — bug fixes and docs only — and credits TGI with initiating the migration of optimized inference engines to transformers model architectures, a direction "now adopted by downstream inference engines, which we contribute to and recommend using going forward," naming vLLM, SGLang, llama.cpp, and MLX. TGI was the Rust-and-gRPC server that powered Hugging Chat and the Inference API, with continuous batching, tensor parallelism, OpenTelemetry, and early quantization support. My position: do not deploy TGI new; its historical importance is that it proved production serving shapes (streaming, batching, messages APIs) before vLLM industrialized them. Two maintenance-mode exits in one field — TGI here, AutoGen in my frameworks dossier — mark the end of the first generation.

The comparison, on the axes that decide adoption:

Runtime	Licence	Primary target	Hardware focus	Serving API	Best at	Watch out for
llama.cpp	MIT	Personal, embedded, edge	CPU, Metal, CUDA, HIP, Vulkan, more	llama-server, OpenAI-compatible	Max hardware reach; hybrid CPU+GPU	You own tuning and model curation
Ollama	MIT	Personal, agent backends	Via llama.cpp	REST on 11434, first-party SDKs	Zero-friction install and model management	Engine features arrive on its schedule
LM Studio	Proprietary, free	Personal desktop + small API host	GGUF plus MLX on Apple silicon	OpenAI-compatible REST, Python/TS SDKs	GUI experience; MCP client; headless mode	Closed core; licence audit is impossible
vLLM	Apache-2.0	Product and fleet serving	NVIDIA, AMD, Intel, TPU, more	OpenAI-compatible, Anthropic Messages, gRPC	Breadth: 200+ architectures, quantization zoo	Feature velocity demands version pinning
SGLang	Apache-2.0	Frontier and RL serving	NVIDIA racks, AMD, TPU, Ascend	OpenAI-compatible	RadixAttention caching, day-zero MoE support	Smaller model-garden coverage than vLLM
TensorRT-LLM	Apache-2.0	NVIDIA-only scale	Blackwell, Hopper	LLM API, Dynamo, Triton	Raw tokens-per-second per GPU	NVIDIA-only; deepest vendor coupling
MLX / mlx-lm	MIT (Apple)	Apple silicon dev and fine-tuning	Apple unified memory only	mlx_lm.server, Python API	Fine-tuning plus inference on Macs	Apple-only; separate weights ecosystem
TGI	Apache-2.0	Maintenance mode	NVIDIA, AMD, Inferentia, TPU	Messages API, SSE streaming	Historical reference	Explicitly in maintenance mode

Apple silicon earned its own runtime, and the two stacks do not mix

MLX is Apple's machine-learning array framework, and mlx-lm is its LLM package — MIT-licensed, copyright Apple, per the licence file. The README covers text generation and, critically, fine-tuning: low-rank and full-model fine-tuning with quantized models, distributed inference and training via mx.distributed, one-command quantization and upload to the MLX Community Hugging Face organization (thousands of pre-converted weights), a rotating key-value cache for long prompts, file-backed prompt caching for multi-turn reuse, and macOS 15+ memory-wiring behaviour with the iogpu.wired_limit_mb sysctl for models that approach the unified-memory ceiling.

The practical point is that MLX is a parallel universe, not a dialect. GGUF weights do not run in mlx-lm and MLX weights do not run in llama.cpp; the communities maintain separate conversion pipelines (the mlx-community organization on Hugging Face is the MLX-side equivalent of the GGUF ecosystem). What you get for the split is real on Apple hardware: fine-tuning in the same runtime you serve from, memory wiring tuned for unified memory, and generation quality unmediated by GGUF quantization choices. My position: on an Apple Silicon Mac doing serious local work, MLX deserves the default slot and llama.cpp stays installed for GGUF-only models and edge cases. LM Studio's decision to ship both engines is the pragmatic acknowledgment that neither wins alone.

TGI stepped aside and TensorRT-LLM opened up: the serving tier consolidated this cycle

Four developments in the current cycle alter choices, not just news feeds. First, TGI entering maintenance mode with an explicit handoff recommendation removed a formerly default option from the serving tier and confirmed the transformers-architecture-first direction that vLLM and SGLang had already taken. Second, TensorRT-LLM completing its move to fully open development on GitHub — with a published deprecation policy, a rarity for vendor frameworks — turned what was effectively a NVIDIA-internal artefact into a real option for platform teams that require source access. Third, the day-zero support race at the frontier — SGLang shipping DeepSeek-V4 and Kimi K3 support in days — has become the serving tier's competitive axis, because every week an engine cannot serve the new open-weights release is a week of product decisions made without it. Fourth, speculative decoding has crossed from research to default: vLLM's list (n-gram, suffix, EAGLE, DFlash) and SGLang's Spec V2 read as settled infrastructure now, which materially changes the tokens-per-second-per-GPU math that justifies local serving against API prices.

Licences are uniformly permissive; the costs are silicon, electricity, and your time

The licence audit is the shortest in any dossier I have written: llama.cpp MIT, Ollama MIT, mlx-lm MIT (Apple copyright), vLLM Apache-2.0, SGLang Apache-2.0, TensorRT-LLM Apache-2.0, TGI Apache-2.0 — each verified against the repository's licence file or README badge. LM Studio is the sole proprietary entry: free to use, closed source, no licence file published, which places it outside any audit regime that requires provenance. Nothing in the field is AGPL, nothing is research-only, and no runtime restricts commercial deployment.

The real cost shape is therefore hardware and operations, and the honest arithmetic has three lines. Capital: the model must fit — a 70-billion-parameter model at 4-bit wants roughly 40-plus gigabytes before context, so the machine precedes the runtime choice, and llama.cpp's CPU-plus-GPU hybrid is the escape hatch that trades speed for feasibility. Electricity and duty cycle: local serving is a fixed cost that amortizes only above a usage threshold; a workstation idling between prompts is more expensive per token than an API for sporadic use and dramatically cheaper for sustained load — the crossover depends entirely on your duty cycle, which is why job-one and job-three recommendations diverge so hard. Engineer time: the serving tier's performance is tunable (quantization choice, KV-cache policy, batching parameters, speculative decoding), and tuning is hours of benchmarking — SGLang and NVIDIA publish tuning guides because the default configuration is not the fast configuration. The one-line summary of the field's economics: the runtime is free, the weights are free, the tokens are free — the GPU, the power bill, and the engineer are what you pay for.

Choose by hardware and workload, not by community enthusiasm

The decision compresses to two questions. What hardware is physically in the room? Which of the three jobs is this?

On a laptop or desktop without meaningful GPU headroom, the question is which convenience layer fits the operator: Ollama if the consumer is a CLI user or an agent that speaks its API, LM Studio if the consumer is a human who wants a GUI and an endpoint, raw llama.cpp if the operator wants full control and accepts full responsibility. On Apple Silicon doing real work, add MLX to the shortlist and prefer it for fine-tuning workflows. On a single NVIDIA box serving a product, vLLM is the default and SGLang the performance alternative — benchmark both on your model and your traffic, because the gap between them flips with workload shape. On multi-GPU racks serving frontier open weights, SGLang and TensorRT-LLM are the contenders, decided by your tolerance for vendor coupling and your expert-parallelism requirements.

Two cross-cutting rules. First, prefer the runtime that names your exact hardware in its backend matrix — llama.cpp's table and vLLM's hardware plugins list are the honest compatibility contracts. Second, treat model availability as part of the runtime: GGUF breadth for the llama.cpp ecosystem, MLX Community conversions for Apple, and the 200-plus-architecture list for vLLM are each load-bearing parts of the decision.

The lifecycle of a request through the serving tier — where the batching and cache wins actually come from:

sequenceDiagram
    participant C as Client
    participant S as Serving runtime
    participant B as Continuous batch
    participant K as KV cache and prefix store
    participant G as GPU

    C->>S: OpenAI-compatible request
    S->>K: Check prefix cache for shared prompt
    alt prefix present
        K-->>S: Reuse cached prefix states
    else miss
        S->>G: Prefill full prompt
    end
    S->>B: Admit request into running batch
    loop token by token, batch-wide
        B->>G: One decode step for every active sequence
        G-->>B: Next-token distributions
    end
    B-->>S: Sequence reaches stop condition
    S-->>C: Streamed completion
    Note over B,K: New sequences join and finish mid-flight - finished slots are reused within the same batch

Where local inference breaks

Each failure mode below is one I have hit, diagnosed, or watched teams hit, with the trigger stated.

Quantization quality cliffs. Trigger: choosing aggressive quantization (2-bit, 3-bit) to fit a model in memory and discovering the degradation only in production behaviour — degraded instruction-following, broken structured output. The cliff is model- and family-specific, and the fix is evaluating the specific quantized build, not the architecture. The 1.5-to-8-bit range llama.cpp advertises is a range of trade-offs, not a range of free lunches.

Context length versus VRAM arithmetic. Trigger: serving a long-context workload on a GPU sized for short prompts. The KV cache grows with sequence length and batch size, allocation fails mid-traffic, and the error surfaces to users as a 500, not a capacity-planning memo. vLLM's PagedAttention softened the fragmentation half of this; the capacity half is arithmetic you must do — sequence length times batch times per-token cache cost against real VRAM after weights.

The unified-memory wiring limit on Apple silicon. Trigger: loading a model that occupies most of unified memory on a Mac. mlx-lm's README documents the symptom — a warning that generation will be slow — and the fix: raising iogpu.wired_limit_mb via sysctl on macOS 15 or higher so the model and cache can be wired. Machines below the OS requirement simply run slow, and teams misdiagnose this as a broken runtime.

Day-zero gaps on new open-weights releases. Trigger: a frontier open-weights model ships and your serving engine does not support its architecture yet — a new attention variant, a new MoE routing scheme. vLLM and SGLang compete on closing this gap in days; llama.cpp's GGUF conversion ecosystem usually follows close behind; anything pinned to older versions simply cannot serve the model. If your product depends on running the new thing immediately, engine day-zero cadence is a selection criterion, not a nicety.

Driver and toolchain matrix rot. Trigger: upgrading CUDA, ROCm, or macOS without re-validating the runtime. The serving tier compiles against specific toolchains — TensorRT-LLM's README badges specific CUDA and PyTorch versions — and a framework upgrade can silently change kernels or break a build. The fix is the same as any compiled dependency: pin, test on upgrade, and read the release notes that document breaking changes.

Maintenance-mode dependency risk. Trigger: building on a runtime that has just been deprecated by its own maintainers. TGI's caution box is this year's example: the project still runs, security patches continue, but the recommendation from its own README is to move to vLLM, SGLang, llama.cpp, or MLX. The general rule the field keeps teaching — AutoGen last year, TGI this year — is that first-generation infrastructure gets sunset once the second generation proves itself, and the sunset notice is the cheapest possible time to migrate.

Local-model agent failures masquerading as framework bugs. Trigger: pointing coding agents at a local model through Ollama or LM Studio and watching the agent loop behave erratically. The local model's instruction-following, tool-call syntax discipline, and context discipline are simply weaker than the frontier cloud models the agents were designed around; the failure looks like an agent bug and is a capability mismatch. Size the model and the task honestly — a local backend is for privacy, cost, and offline work, not for parity with the frontier.

The choice as a decision flow, with the failure-prone branches marked:

flowchart TD
    START["What hardware is in the room"] --> Q1{"Apple silicon Mac"}
    Q1 -->|yes| MLX["mlx-lm default<br/>llama.cpp for GGUF-only models"]
    Q1 -->|no| Q2{"Personal machine<br/>one user at a time"}
    Q2 -->|yes| Q3{"Want a GUI"}
    Q3 -->|yes| LMS["LM Studio<br/>proprietary core, open engines"]
    Q3 -->|no| OLL["Ollama or raw llama.cpp<br/>CLI and API ownership"]
    Q2 -->|no| Q4{"NVIDIA GPUs serving real load"}
    Q4 -->|single box, mixed models| VLLM["vLLM<br/>breadth and ecosystem default"]
    Q4 -->|frontier MoE or RL rollout| SGL["SGLang<br/>RadixAttention and day-zero cadence"]
    Q4 -->|NVIDIA-only at maximum scale| TRT["TensorRT-LLM<br/>deepest vendor coupling"]
    MLX --> VERIFY["Benchmark on your model and traffic<br/>before committing"]
    LMS --> VERIFY
    OLL --> VERIFY
    VLLM --> VERIFY
    SGL --> VERIFY
    TRT --> VERIFY

Weight-format standards, serving-tier consolidation, and the duty-cycle crossover stay unresolved

Three questions will reshuffle this field. First, does the GGUF-versus-native-weights split persist? vLLM loading GGUF and MLX maintaining a parallel conversion universe suggest the interchange-format question is unsettled; a single well-specified quantized-weights standard would erase one of the last real boundaries between the desktop and serving tiers. Second, where does the serving tier's consolidation end? SGLang's RL-rollout position and vLLM's breadth are converging on each other's territory, and TensorRT-LLM's full open-sourcing raises the possibility that the performance frontier and the ecosystem frontier land on one codebase. Third, what is the real duty-cycle crossover for local versus API at the model qualities people actually want — and does speculative decoding plus memory-bandwidth growth move it fast enough to matter for job-one users, for whom the answer today is usually "the API is cheaper unless you value privacy for its own sake"? The hardware roadmap — wider unified memory, faster LPDDR, inference-optimized consumer NPUs — is aimed exactly at that crossover, and it is the number to watch even though no runtime README prints it.

Resources

Engine and runtime repositories

• llama.cpp README — backend matrix, quantization range, hybrid CPU+GPU inference, cli and serve front-ends, MIT badge.
• Ollama README — install paths, REST API, agent launch integrations, llama.cpp attribution, client ecosystem.
• Ollama licence — MIT, verified from the file.
• vLLM README — PagedAttention, feature list, quantization and parallelism coverage, model-architecture breadth, hardware plugins.
• vLLM licence — Apache-2.0, verified from the file.
• SGLang README — RadixAttention, deployment scale claims, day-zero support history, RL-rollout integrations, LMSYS hosting.
• SGLang licence — Apache-2.0, verified from the file.
• mlx-lm README — generation and fine-tuning on Apple silicon, conversion and prompt caching, wired-memory behaviour.
• mlx-lm licence — MIT, Apple copyright, verified from the file.
• TensorRT-LLM README — Apache-2.0 badge, PyTorch-native architecture, optimization list, deprecation policy, Blackwell performance news.
• Text-Generation-Inference README — maintenance-mode caution box and handoff recommendation, historical feature list.

Applications and documentation

• LM Studio documentation — dual-engine architecture (llama.cpp and MLX), APIs, SDKs, headless mode, system requirements.