Living dossier

Local AI Hardware

Back to topics•Blog
Mehran Mozaffari·
14 resources0 related posts

Local AI hardware is a memory problem wearing a compute costume

The honest definition of local AI hardware is narrower than the marketing: it is the set of machines where the decisive constraint is how many model bytes sit addressable next to the compute, and how fast the compute can stream those bytes. Everything else is downstream. A chip with enormous FLOPS and small memory runs small models quickly and large models not at all. A chip with modest FLOPS and enormous memory runs large models slowly but runs them. Local inference is a memory problem wearing a compute costume.

Three things get miscategorised into this field, and the distinctions matter commercially. First, "AI PCs" rated in NPU TOPS: AMD's Ryzen AI Max+ 395 lists up to 126 total TOPS with 50 from its NPU, and Intel NPUs appear in OpenVINO's stack — but the LLM work in practice runs on CPU and GPU, because llama.cpp, the de facto reference runtime, lists its Intel NPU backend as "In Progress" and has no AMD XDNA NPU backend at all. TOPS sold the machines; the software rails did not follow. Second, gaming GPUs repurposed as inference engines: genuinely excellent at tokens per second, capped by VRAM capacity that the vendor controls. Third, robot-and-camera accelerators like Hailo or the Coral Edge TPU line: real neural inference, but a different job — fixed-vision models, not 4,000-token conversations — and treating them as "local AI" in the LLM sense produces bad purchases.

The field, properly scoped, is six machine families (NVIDIA DGX Spark, Jetson Orin Nano Super and its successor, Jetson Thor, Apple Mac Studio, the RTX 5090 class of desktop GPU, and AMD's Strix Halo mini-PCs), one single-board option (Raspberry Pi 5 plus a Hailo HAT), and the runtimes that make them usable — llama.cpp and its GGUF quantization format above all, with MLX on Apple silicon and OpenVINO on Intel.

The Only Number That Decides Token Speed: Memory Bandwidth

Token generation in an LLM is largely sequential: every generated token requires reading essentially all activated weights once more. That makes steady-state decode speed track memory bandwidth almost linearly, and it makes the spec sheets' favourite number — peak INT8 or FP4 TOPS — close to irrelevant for the single-stream chat case. The bandwidth table across the current field, taken from vendor spec pages, spans a factor of twelve:

Machine Memory Stated bandwidth Compute claim Power envelope
Jetson Orin Nano Super 8 GB LPDDR5 (128-bit) 102 GB/s 67 INT8 TOPS 7–25 W
Raspberry Pi 5 + AI HAT+ (Hailo-8L) Pi's own RAM + 13 TOPS module Pi-class (PCIe 2.0 link) 13 TOPS (vision INT8) ~10 W class
DGX Spark (GB10) 128 GB LPDDR5x (256-bit) 273 GB/s 1 PFLOP FP4 (sparse) 140 W chip TDP, 240 W PSU
Jetson Thor T5000 128 GB LPDDR5X (256-bit) 273 GB/s 2070 FP4 TFLOPS (sparse) 40–130 W
Ryzen AI Max+ 395 up to 128 GB LPDDR5X-8000 (256-bit) 256 GB/s (arithmetic from spec) 50 NPU TOPS of 126 total 45–120 W cTDP
Mac Studio M5 Max 36–128 GB unified 460 GB/s (614 GB/s at 40-core) — desktop class, 480 W max
Mac Studio M5 Ultra 96 GB–512 GB unified 1.2 TB/s — 480 W max continuous
RTX 5090 (desktop) 32 GB GDDR7 (512-bit) ~1.8 TB/s at rated GDDR7 speed (arithmetic from the 512-bit bus) Blackwell, 5th-gen Tensor Cores, FP4 575 W board

Read that table as a scatter plot with two axes: capacity and bandwidth. The unified-memory machines (Spark, Thor, Strix Halo, Mac Studio) cluster at 128–512 GB but 256 GB/s to 1.2 TB/s. The discrete-GPU machines (RTX 5090) cluster at 32 GB and roughly 1.8 TB/s. Nobody ships 512 GB and 1.8 TB/s in one box at consumer prices — that combination remains a multi-GPU data-center rack. Every purchasing decision in this field is choosing which axis to fail on.

The Memory-Ceiling Arithmetic Every Buyer Should Run First

The runtime layer makes the decision mechanical. llama.cpp quantizes models from 8-bit down to 1.5-bit integers, so a model's resident size is roughly parameters times bytes-per-weight. Working the numbers at common quantization points: an 8-billion-parameter model at 4-bit is around 4.5 GB; a 70-billion-parameter model at 4-bit is around 40 GB; NVIDIA's own DGX Spark copy claims inference for models up to 200 billion parameters and fine-tuning up to 70 billion, both consequences of 128 GB of coherent unified memory rather than of the chip's petaflop figure.

Two corrections to that naive arithmetic decide whether a machine actually works. First, quantization is lossy and unevenly so — the 8B model that "fits" an 8 GB Jetson Orin Nano Super at 4-bit leaves only about 3.5 GB for the operating system, the KV cache, and activation buffers, and the KV cache grows with every token of context. Long-context agentic workloads, the exact thing people buy these machines for, push the cache from megabytes to gigabytes. Second, weights are not the only resident memory: vision-language models carry a second encoder, and any serving stack holds more than one model. NVIDIA's own Jetson team published memory-reclamation work reclaiming "up to 12GB" on Jetson modules — vendor acknowledgment that the ceiling, not the FLOPS, is the binding constraint at the edge.

The practical tiers that fall out of the arithmetic: 8 GB runs 4-bit models to roughly 8B with short contexts; 16–32 GB runs 14–32B comfortably or MoE models with sparse activation; 128 GB runs 4-bit 70B dense models with real context, or fine-tunes up to 70B per NVIDIA's Spark documentation; 256–512 GB runs 4-bit models past 200B parameters, the territory NVIDIA invokes with two Sparks bridged over ConnectX-7 for 405B-class models.

One file format and one runtime feed every backend in the field

The field's architecture is quieter than the hardware wars suggest. A GGUF file holds quantized weights; a runtime — llama.cpp overwhelmingly, MLX on Apple, OpenVINO on Intel — maps the model's operations onto whatever backend the machine offers; the backend is where vendor lock-in actually lives. llama.cpp's supported-backend table is the most honest compatibility matrix in the field: Metal for Apple Silicon, CUDA for NVIDIA, HIP for AMD GPUs, Vulkan as the everyone-else floor, SYCL and the in-progress OpenVINO backend for Intel, CANN for Ascend NPUs, Hexagon for Snapdragon (in progress), MUSA for Moore Threads, ZenDNN for AMD CPUs. Every hardware claim in every vendor deck must survive contact with that table, and several do not yet.

flowchart TD
    M[GGUF model file<br/>quantized 1.5 to 8 bit] --> R[Runtime<br/>llama.cpp or MLX or OpenVINO]
    R --> B{backend dispatch}
    B --> Metal[Apple Metal<br/>unified memory]
    B --> CUDA[NVIDIA CUDA<br/>RTX 5090 or Spark or Jetson]
    B --> HIP[AMD HIP or Vulkan<br/>Radeon 8060S]
    B --> OV[OpenVINO in progress<br/>Intel CPU GPU NPU]
    B --> CPU[plain CPU<br/>the universal floor]
    Metal --> UC[unified memory pool<br/>OS shares the pool]
    CUDA --> VRAM[dedicated VRAM<br/>capacity fixed by card]
    HIP --> UC
    OV --> UC
    CPU --> RAM[system RAM<br/>slowest decode]

The sequence of a token — what actually happens per generated word — explains why the bandwidth table above dominates every benchmark:

sequenceDiagram
    participant U as User
    participant R as Runtime llama.cpp
    participant M as Memory LPDDR or GDDR7
    U->>R: submit prompt
    R->>M: load quantized weights once
    Note over R,M: prefill is compute-heavy, decode is bandwidth-bound
    loop every generated token
        R->>M: read activated weights plus KV cache
        M-->>R: stream bytes at bus bandwidth
        R-->>U: emit one token
    end
    Note over U,M: context growth keeps adding KV bytes on top of weights

Eight machines, two bets: capacity or bandwidth, never both

NVIDIA DGX Spark is the most consequential product of the cycle and the easiest to misbuy. The spec page is candid: 128 GB coherent unified memory, 273 GB/s, 1 PFLOP FP4, ConnectX-7 at 200 Gbps for pairing two units, a 140 W chip in a 240 W chassis with declared 35 dB operating noise. My position: buy it for the CUDA software stack and the 70B fine-tuning claim, not for inference speed — at 273 GB/s a 70B 4-bit model decodes at single-digit tokens per second. It is a development bench for the stack your servers run, priced accordingly, and the January 2026 "2x performance" DGX OS update shows the software is still maturing under it.

Jetson Orin Nano Super is the field's best-value education: 67 INT8 TOPS, 8 GB at 102 GB/s, 7–25 W — and the quietly brilliant part, NVIDIA shipped the "Super" uplift as a software update to every existing Orin Nano kit. My position: the right first edge machine, with the 8 GB ceiling respected; it runs vision transformers and small LLMs, not your 70B ambitions. It is also a dying platform in the good sense — Jetson Orin Nano 2 was announced August 25, 2026 with 78 TOPS, double the inference performance, 40% lower power at matched performance in 15 W mode, arriving first half of 2027.

Jetson Thor (T5000) is the robotics flagship: 2070 sparse FP4 TFLOPS, 128 GB at 273 GB/s, 40–130 W, MIG partitioning, four 25 GbE ports for sensor fusion. Position: for humanoid and physical-AI teams this is the target platform — Isaac and GR00T assume it — but note that NVIDIA's own page disagrees with itself on module naming (the blog blurb says T3000/T2000; the specification tables say T5000/T4000), so verify the exact SKU before ordering anything.

Mac Studio (M5 Max / M5 Ultra) is, on the fetched spec sheet, the only consumer machine combining 512 GB of unified memory with 1.2 TB/s of bandwidth. Position: the best large-model local inference per watt and per liter, now backed by Apple's own MIT-licensed MLX framework — whose unified-memory model is its stated differentiator — plus first-class llama.cpp Metal support. The costs that get left out: macOS runs only on Apple hardware, memory is configure-to-order at purchase and fixed forever, and the price climbs steeply through the 256 GB and 512 GB tiers.

RTX 5090 desktop rigs are the tokens-per-second kings of the table: 32 GB GDDR7 on a 512-bit bus, fifth-generation Tensor Cores with FP4 support, and the fastest decode this side of a rack. Position: buy one if your models fit in 32 GB and you want speed; document the risks instead of discovering them. The Wikipedia record here is unusually useful: MSRP $1,999 with street prices above it, a 575 W board drawing 125 W more than its predecessor's 450 W, the 12V-2×6 connector melting controversy inherited and revised, early cards shipping with missing ROPs, dropped 32-bit CUDA support (partially restored in the 591.44 driver of December 2025), and a late-2025 memory supply shortage pushing prices up again.

AMD Ryzen AI Max+ 395 (Strix Halo) mini-PCs are the x86-Linux answer to Spark: 128 GB of 256-bit LPDDR5X-8000, 16 Zen 5 cores, Radeon 8060S graphics with 40 compute units, 45–120 W configurable TDP. Position: the honest framing is "128 GB at Mac-adjacent bandwidth on an open OS." The 50 NPU TOPS are real silicon with no llama.cpp backend — your LLM path is the Radeon iGPU via Vulkan or HIP — and PCIe stays at 4.0. For a Linux-first homelab wanting 70B-class 4-bit inference without NVIDIA, this is currently the credible road.

Raspberry Pi 5 + AI HAT+ (Hailo-8L or Hailo-8, 13 or 26 TOPS) is vision acceleration, not LLM hardware, and Raspberry Pi positions it exactly that way: integrated into the camera stack for detection, segmentation, and pose estimation, from $70, in production until at least January 2030. The original AI Kit is discontinued in favour of this HAT. Position: unmatched for camera-pipeline inference per dollar; wrong tool for chat models. Note the lineage: Google's Coral — the previous generation's Edge TPU darling — has itself been repositioned into an open-source, RISC-V-and-MLIR-based platform play, a warning about how fast accelerator ecosystems churn when the toolchain is proprietary.

The runtimes deserve their own verdicts. llama.cpp (MIT licence, 126,000 stars) is the field's true platform — its backend table is the compatibility layer, its quantization ladder is the memory arithmetic, and its CPU+GPU hybrid mode quietly rescues models that overshoot VRAM. MLX (MIT) is Apple's research-grade framework with the unified-memory model as its stated advantage, now shipping a CUDA backend on Linux. OpenVINO (Apache-2.0) is the only stack taking Intel NPUs seriously for inference, with a GenAI serving path and a vLLM backend — still marked in-progress inside llama.cpp. The licence point generalizes: the permissive MIT and Apache runtimes port to whatever you buy next; the accelerator vendors' own toolchains mostly do not, and that asymmetry — not the sticker price — is the real cost shape of the field. Hardware prices are one-time; a locked toolchain bills you again every hardware generation.

Four dates from the last year that changed what to buy

Four dated events altered real decisions. October 2025: DGX Spark began shipping, moving the "can I run 70B at my desk" question from theory to a purchasing decision (NVIDIA lists pricing only through its marketplace, itself a statement about the channel — check the current number before planning around it). December 2025: NVIDIA's DGX OS update claimed a further 2x inference uplift on Spark — early-adopter machines materially improved after purchase, which is rare and worth pricing in. Early 2026: Apple replaced the prior Studio generation with M5 Max (460–614 GB/s) and M5 Ultra (1.2 TB/s, configurable to 512 GB), doubling the memory ceiling available under a desk-sized power envelope. August 25, 2026: Jetson Orin Nano 2 announced — 78 TOPS, double the inference performance, 40% less power at matched performance, first-half-2027 availability — which should freeze non-urgent Orin Nano Super purchases that can wait two quarters. Running underneath all of it: the late-2025 memory supply shortage documented on the RTX 50 series record, which raises the real cost of the discrete-GPU path specifically and has no announced end date.

Your constraint picks the machine before your preferences get a vote

Data sensitivity and latency decide more purchases than benchmarks. If the requirement is "never leaves my network, answers in under a second," the field collapses to: an 8–32 GB machine (Jetson, RTX 5090, Strix Halo) running quantized small models — pick the cheapest tier whose memory arithmetic closes with your longest real context. If the requirement is "largest model that fits, regardless of speed," climb the memory ladder: 128 GB (Spark, Strix Halo, Thor) for 4-bit 70B-class; 256–512 GB (M5 Ultra) for 200B-class. If the requirement is tokens per second per dollar for models that already fit in 32 GB, nothing on the table beats the RTX 5090 — accept the power draw, the connector history, and the driver drama as the fee. If the requirement is watts — battery, drone, wall-less deployment — the answer is Jetson at 7–25 W or Thor at 40–130 W, and the model shrinks to meet the memory. If the job is camera inference rather than language, stop reading LLM comparisons entirely: a $70 Hailo HAT on a Pi 5 is the whole answer.

Software rails are the tiebreaker because they outlive hardware. Everything MIT or Apache-2.0 (llama.cpp, MLX, OpenVINO) ports to the next machine you buy; vendor-locked toolchains do not. A purchase that only works through one company's proprietary stack is a rental whatever the weights' licence says, because the weights are the part you can replace and the toolchain is the part you cannot.

Where it breaks

The 8 GB cliff. Trigger: 4-bit 8B model plus long context on Jetson Orin Nano Super. The weights leave roughly 3.5 GB for OS, KV cache, and buffers; a long agentic session walks the cache up until allocation fails or the box swaps. Mitigation: cap context, drop to 3-bit, or move up a memory tier.

Bandwidth disappointment at the top of the memory ladder. Trigger: buying Spark or Strix Halo expecting chat speed on 70B. At 273 GB/s and 256 GB/s, a 4-bit 70B model decodes at single-digit tokens per second; the memory fits, the experience feels broken. This is not a defect — it is the bandwidth table above — but it generates most of the buyer's remorse in this category.

The connector. Trigger: RTX 5090 Founders Edition and some partner cards, 575 W through the 12V-2×6 connector, incompletely seated. The melting-connector record from the RTX 4090 carried into the 50 series before the PCI-SIG revision tightened the sense-pin behaviour; verify seating and cable rating on install day rather than after the scorch mark.

Silicon lottery, compute edition. Trigger: early RTX 5090/5080/5070 Ti cards shipped missing eight ROPs. NVIDIA put it under 0.5% of cards and claims rectification; graphics output was affected while pure compute was not, so an LLM buyer may never notice — but check a used card's ROP count against spec before paying flagship money.

Driver archaeology. Trigger: RTX 50 launch drivers — black screens after install, BSODs through the 572.XX and 576.XX branches, removal of 32-bit CUDA/OpenCL breaking older acceleration paths until driver 591.44 partially restored them in December 2025. Pin driver versions on inference boxes; never auto-update a production rig.

NPU seduction. Trigger: buying an "AI PC" for its NPU TOPS, then finding the LLM runtime has no backend for it. The practical path on those machines is CPU and iGPU; the NPU idles. Verify against llama.cpp's backend table before believing any TOPS number on a box.

Accelerator-ecosystem churn. Trigger: Coral's Edge TPU line — 8-bit-only models, then the product line effectively frozen while the brand repositioned as a compiler-and-RISC-V platform. Hardware bought against a proprietary toolchain can outlive its own toolchain. Hailo's HAT integration on the Pi is the current counterexample done right, precisely because Raspberry Pi ships OS-level auto-detection and camera-stack integration in the standard image.

Fixed memory, forever. Trigger: every unified-memory machine here (Spark, Jetson, Mac Studio, Strix Halo) solders its RAM. The 36 GB Mac Studio or 8 GB Jetson bought small cannot grow; a discrete-GPU box can swap cards. If your model sizes will grow, this asymmetry is the single most expensive thing to get wrong.

Open questions

Whether NVIDIA sustains DGX Spark as a first-party line or folds GB10-class silicon into generic OEM mini-PCs is unresolved, and it determines whether the CUDA-on-desktop bet is durable for buyers who cannot tolerate dead platforms. Whether NPUs ever earn an LLM runtime — OpenVINO's progress on Intel versus the absence of any AMD NPU backend in llama.cpp are the two live datapoints — decides if the next "AI PC" generation deserves a premium over its GPU. What the memory shortage does to 2027 pricing across GDDR7 and LPDDR5X is unknown, and it dominates every cost model written in 2026. Whether Apple pushes MLX beyond a CUDA backend into a genuinely cross-vendor framework, or keeps it Apple-first, determines whether the Mac becomes the centre of local inference or stays its most capable niche. And Jetson Orin Nano 2's eventual pricing, promised for the first half of 2027, will re-anchor the entry tier of the entire edge market either way.

Resources

Hardware spec pages, all fetched:

Runtimes, licences, and the RTX 50 record: