Lily: How Perplexity's Appliance-Style Inference Engine Squeezes Apple Silicon Dry

What Lily actually does: a fixed-function inference appliance

When I look at Lily, I don't see a local inference framework. I see an appliance — the local-inference equivalent of a dedicated hardware coprocessor that happens to be expressed as software. Perplexity built it for one job: making sure on-device compute doesn't become the bottleneck in the hybrid compute loop that powers Perplexity Computer.

That hybrid split is the context that makes Lily make sense. In this architecture, workflows are partitioned by nature rather than by convenience. Latency-sensitive operations — parsing sensitive files, reading local environment context, fast tool invocation — run locally, where raw inputs never leave the user's machine. The cloud handles the heavy lifting: multi-step reasoning, large-scale context aggregation, and computation that would be absurd to attempt on-device. Lily is the local half of that partnership, and it's built to disappear into the workflow — a sub-100ms response to a tool call shouldn't be the thing that makes an agentic loop feel sluggish.

Three design decisions define Lily's architecture. First, it's a single-process Rust runtime. No Python layer, no interpreter overhead, no garbage collection pauses — just deterministic memory management and minimal process-startup latency. For a runtime that's invoked repeatedly during agentic interactions, that matters far more than it sounds on paper.

Second, it uses hand-tuned Metal Shading Language kernels directly on Apple Silicon's GPU and unified memory, bypassing general graph compilation layers entirely. The kernel shapes, threadgroup allocations, memory tiling, and quantization layouts are hard-optimized for one specific model architecture.

Third — and this is the part that makes Lily fundamentally different from what most people expect from "local inference" — it's specialized for Qwen3.6-35B-A3B, an active-parameter MoE/sparse variant. Not "Qwen-family models." Not "MoE models." This specific architecture, on this specific compute platform. Kernel shapes are compiled to match its exact dimensions. Threadgroup allocations are tuned to its expert activation patterns. Memory tiling is designed around its hidden size, intermediate size, attention head count, and router top-k selection.

The tradeoff is absolute, and I want to be blunt about it: Lily is not a drop-in replacement for GGUF or Safetensors files. It's not a general-purpose runtime you can point at any model. If you want to run Llama-4 or DeepSeek on it, you're rewriting kernels. What Lily gives you is something different — a fixed-function engine that saturates Apple Silicon's memory bandwidth for one model on one hardware family, in exchange for throwing away the architectural portability that llama.cpp and MLX-LM treat as their core value prop.

That's the appliance mindset, and I think it's genuinely interesting for one specific use case: hybrid agent platforms where the local model is a deliberate, fixed design decision, not a moving target.

Under the hood: how the hand-tuned Metal kernels map to Qwen3.6-35B-A3B

Here's where Lily's "narrow is better" philosophy gets concrete.

The baseline contrast is MLX-LM. Apple's framework generates or dispatches Metal operations through a generic array computation graph — NumPy/PyTorch-like, dynamically traced. That's elegant and flexible, but every operation goes through dispatch overhead, and the graph compiler has to make generic choices about threadgroup sizes, register allocation, and memory tiling that work across hundreds of potential architectures. The research suggests this generic dispatch leaves roughly 10–25% of theoretical memory bandwidth on the table for sparse MoE models, which is actually a substantial number when you're in a bandwidth-bound decode loop.

Lily bypasses the graph compiler entirely. The MSL kernels are hand-written, with threadgroup dimensions and register tile sizes hardcoded to match Qwen3.6-35B-A3B's specific dimensions — hidden size, intermediate size, attention heads, MoE router top-k, and the A3B active expert count. No dynamic dispatch, no generic tensor slicing, no format conversions. The kernel shapes are what they are.

The A3B part is where the architectural insight lives. This is a Mixture-of-Experts model with a small active parameter count relative to its total size — sparse expert routing means only a subset of experts activate per token. In a generic engine, this creates a real problem: when active experts are dispatched across threadgroups, memory divergence can occur — the GPU's execution units trying to read from scattered memory locations that don't coalesce cleanly. Hand-tuned kernels can structure the memory layout so that expert loads match Apple Silicon GPU threadgroup tiling, keeping memory access coalesced and avoiding the serialization that kills decode throughput.

For unified memory bandwidth utilization, the specificity matters in two ways. During prefill — the compute/bandwidth-bound prompt processing phase — hand-tuned kernel fusion reduces the number of memory transactions. During decode, where you're purely memory-bandwidth-bound (loading weights for every token generated), knowing the exact weight shapes and expert routing patterns lets Lily structure memory accesses to saturate the unified memory bus more effectively than generic kernels would. The result, per the research, is lower time-to-first-token and higher token-per-second throughput than generic inference engines on this model.

Now, the practical constraint: 32GB+ unified memory. That's not a suggestion — Qwen3.6-35B-A3B at this scale requires roughly 18–24GB for quantized weights and KV-cache at moderate context lengths, and that's before you account for the OS and background processes sharing the same physical RAM. On an M-series Mac with 8 or 16GB, Lily simply won't initialize. This makes it a real constraint on who can run it — and a reminder that the line between local inference and cloud fallback is a hardware threshold, not just a software preference.

Here's the flow, visualized:

graph TD
    U[User request] --> H[Hybrid orchestrator]
    H --> D{Local or cloud}
    
    D -->|Local: latency-sensitive / private| L[Lily Rust runtime]
    L --> L1[Tokenize and preprocess prompt]
    L1 --> L2[Dispatch Metal kernels with fixed threadgroup / tile sizes]
    L2 --> L3[MoE router: top-k expert selection]
    L3 --> L4[Memory-bound decode loop<br/>reading weights from unified memory]
    L4 --> L5[Output token stream]
    
    D -->|Cloud: heavy reasoning / large context| C[Cloud inference]
    C --> C1[Multi-step reasoning<br/>large-scale context aggregation]
    
    L5 --> O[Return to orchestrator / user]
    C1 --> O

The comparison: Lily vs llama.cpp, MLX-LM, and ExLlamaV2

Lily sits on a very specific corner of the local inference landscape, and the comparison table makes that corner clear. It's not a "better or worse" question — it's a question of what you're optimizing for.

Engine	Primary Goal	Target Hardware	Core Abstraction	Model Breadth	Runtime Overhead
Perplexity Lily	Low-latency local agent execution for a fixed hybrid pipeline	Apple Silicon (M-series) exclusively	Bare-metal Rust + hand-tuned Metal Shading Language (MSL) kernels	Hardcoded / specialized: Qwen3.6-35B-A3B only	Minimal — single-process Rust runtime, zero abstraction layers
Apple MLX / MLX-LM	First-party research & development framework for Apple Silicon	Apple Silicon (M-series)	Python/C++ array framework with dynamic tracing (NumPy/PyTorch-like)	Broad — any supported PyTorch/Hugging Face conversion	Medium — Python runtime bindings over C++ core
llama.cpp / Ollama	Universal cross-platform accessibility & broad quantization support	CPU, CUDA, Metal, ROCm, Vulkan, SYCL	C/C++ tensor graphs (GGML) with hardware backends	Universal — GGUF: Llama, Mistral, Qwen, Gemma, DeepSeek, etc.	Low-to-medium — C/C++ runtime; Ollama adds Go/container wrapping
ExLlamaV2	Extreme decoding throughput & VRAM compression	NVIDIA CUDA	C++/CUDA hand-coded GEMM/GEMV kernels (EXL2)	Broad transformer support (EXL2 / GPTQ)	Low — Python/C++ driver

Let me be concrete about the tradeoffs on each.

MLX-LM is the first-party generalist, and it's genuinely good at what it does. It runs hundreds of architectures, it's maintained by Apple, and it's the path of least resistance for anyone doing research or prototyping on Apple Silicon. But its generic array computation graph pays a real cost: the 10–25% bandwidth shortfall I mentioned earlier. For a MoE model where expert activation patterns are sparse and irregular, generic dispatch just doesn't structure memory accesses as well as kernels that know exactly what they're loading.

llama.cpp / Ollama is the universal standard, and I don't say that lightly. C/C++ tensor graphs with GGML backends, cross-platform to nearly anything with a silicon process, and quantization formats (Q4_K_M, Q8_0) that are wisely standardized. But it pays for that universality in generic kernel templates — it has to handle any tensor dimension, any quantization scheme, any platform. Dynamic dispatch, generic tensor slicing, and format conversions add up. It works everywhere; it just doesn't saturate a specific GPU's bandwidth the way Lily does.

ExLlamaV2 is the most interesting analogy, because it took Lily's philosophy in a different direction — NVIDIA's ecosystem. Hand-crafted CUDA kernels for specific quantization layouts (EXL2), rejecting standard cuBLAS or PyTorch dispatch. Same principle: narrow the scope to maximize throughput. But its target is NVIDIA VRAM and sub-4-bit quantization, not Apple Silicon unified memory or MoE routing dynamics. It's proof that "hand-tune for one hardware+model combo" is a valid strategy, and a reminder that Lily's approach isn't novel in spirit — it's just been applied to a different platform and a specific model architecture that has particular sparse-routing characteristics.

The takeaway I'd land on: Lily is not a general-purpose substitute for any of these. If you want model flexibility and community tooling, use llama.cpp or MLX-LM. If you're on NVIDIA, ExLlamaV2 is an interesting model to follow. But if you're building a hybrid agent platform where the local model is a fixed design decision — and you're targeting Apple Silicon — Lily's vertical specialization is the correct trade, and it's one that general-purpose runtimes can't match.

Where it breaks: the failure modes I'd watch for in production

Lily's specialization is its superpower and its vulnerability. The same hardcoding that lets it saturate unified memory bandwidth creates sharp edges that will bite in production. Here's what I'd watch for.

Kernel shape rigidity. When prefill lengths don't align with compiled tile and chunk boundaries, Lily pays an exorbitant penalty in threadgroup divergence or has to pad batch buffers that waste scarce local RAM. In practice, you'd see this as erratic time-to-first-token (TTFT) on certain prompt lengths — not a smooth curve, but spikes at specific token counts. The telemetry signature is a bimodal TTFT distribution: most requests fast, some mysteriously slow. Mitigation: pad prompts to aligned boundaries in the orchestration layer before they reach Lily, sacrificing a small amount of compute for predictable latency.

Unified memory pressure and macOS jetsam. A 35B model at this scale needs 18–24GB for quantized weights plus KV-cache, all sharing physical RAM with the OS, browser, Docker, and whatever else the user runs. macOS doesn't give GPU-mapped allocations standard swap grace — when memory pressure elevates, the kernel aggressively pages or terminates without ceremony. You'll see this as kern.memorystatus pressure warnings followed by abrupt Metal command buffer failures. The practical mitigation is a pre-flight probe at app startup checking sysctl hw.memsize and Metal device capability flags, enabling Lily only on 32GB+ configurations. But that's not enough — you also need runtime memory pressure monitoring, because a user can start with headroom and then open 40 browser tabs mid-session.

MoE expert routing imbalance. Qwen3.6-35B-A3B's sparse routing is dynamic. When consecutive tokens route to the same expert, it creates threadgroup serialization — the GPU's execution units can't parallelize when everyone's waiting on memory reads for the same expert weights. The read is sudden decode throughput collapse: tokens/sec drops off a cliff mid-generation, not gradually. This is the hardest failure mode to predict, since it depends entirely on token content. Your mitigation is a runtime heuristic — if generation speed falls below a threshold for sustained bursts, trigger a rebalance or accept the hit and move on.

Thermal throttling. Thin MacBooks aggressively throttle GPU clock and memory bus frequencies under sustained load. After 2–3 minutes of continuous agentic loops, TTFT and decode speed can drop 30–50%. Telemetry: a slow, monotonic degradation in P50 decode speed across a session, not a sharp spike. This one's mitigated by design — structure your agent loop so local inference is bursty, not sustained, or re-route to cloud after N consecutive local turns.

stateDiagram-v2
    [*] --> Idle
    Idle --> AllocatingWeights: User request
    AllocatingWeights --> Prefill: Allocation complete
    Prefill --> Decoding: Prefill complete
    Decoding --> Decoding: Generate tokens
    Decoding --> RebalancingMoE: Routing skew detected
    RebalancingMoE --> Decoding: Rebalance complete
    Decoding --> Throttled: TTFT > 1.5s OR memory pressure high
    Throttled --> Idle: Pressure drops
    Decoding --> Failed: Kernel error / OOM
    Throttled --> Failed: Sustained pressure
    Failed --> CloudInference: Fallback to cloud
    CloudInference --> [*]: Request complete

The maintenance trap: why model upgrades will p

The trap is structural, not incidental. Lily's optimizations live in Metal shader code, not in an intermediate representation. MLX and llama.cpp have graph compilers and tensor abstractions — when a new model comes out, you convert weights and the existing kernel templates handle the new shapes. Lily doesn't have that. The kernels are the implementation. A model upgrade means everything is on the table: threadgroup sizes, register tile dimensions, memory tiling, quantization layouts, MoE routing patterns.

Here's the real economics. If you're Perplexity, shipping a fixed model and fixed hardware, this is fine. You build once, maintain continuously, and the specialization pays for itself in user experience. But if you're a developer building a general local AI tool — a runtime your users point at whatever model they want — this model is a liability, not an asset. Every time someone asks for Llama-4 support, you're not running a conversion script. You're doing kernel development.

The maintenance trap: why model upgrades will p

Let me put a realistic timeline on what a kernel re-tune actually involves.

You're profiling on multiple M-series tiers because M1, M2, M3, and M4 have different memory bus widths, SIMDgroup execution characteristics, and dynamic caching behavior — a kernel tuned for M1 Max may regress on M3 Pro. That's dozens of profiling runs across hardware, each with careful measurement of TTFT and decode throughput at various context lengths. Then you're adjusting threadgroup allocations to match the new model's hidden size, intermediate size, and attention head count — not a one-line change, but a re-derivation of the tiling strategy. Then you're re-validating quantization layouts, because the new model may not respond to the same quant scheme. Then you're testing MoE routing behavior, because the active expert distribution might be different enough to break your static assumptions.

This is weeks of work, not days. And it doesn't account for the ongoing hardware maintenance — as Apple updates Metal architectures and introduces new GPU features, hand-tuned kernels require manual profiling and adjustment to avoid regression. The research flags this explicitly: M2-to-M3-to-M4 shifts in dynamic caching and SIMDgroup execution require re-validation even if you're not changing the model.

So the question becomes: what's your update cadence? If your product can pin a model for a year, Lily's trade is strong. If you expect to ship model updates quarterly, you're signing up for perpetual kernel tuning — every release is a re-tune, every hardware generation is a re-tune, and the accumulated cost dwarfs the latency benefit. ll

Three projects you can build with Lily (or its philosophy)

Lily's architecture is specific enough that you won't accidentally build something useful with it — you have to deliberately aim. But that specificity makes it a natural fit for three concrete projects that are worth trying.

A hybrid local/cloud agent harness with circuit-breaking. This is the project that most directly mirrors what Lily was built for. You'd build a Rust or Python orchestrator that sends private file parsing, tool-calling, and context prep to Lily, and routes multi-step reasoning to a cloud API. The orchestrator includes a pre-flight probe reading sysctl hw.memsize and Metal device flags to enable local mode only on ≥32GB machines — otherwise you're setting up users for a startup crash. The part that matters is the circuit-breaker: if local TTFT exceeds ~1.5s under load, you fall back to cloud without failing the user workflow. You also need a thermal watch — if decode speed drops >30% for >30s, shift more load to cloud. What connects: Lily runtime, the hybrid orchestrator, a cloud API client, and memory/telemetry hooks. What to watch for: the failure isn't a hard crash, it's a slow degradation that users perceive as "the app got sluggish." The circuit-breaker threshold matters less than the telemetry that triggers it.

A Lily production telemetry dashboard. The Rust-side structured error logging is the foundation — you capture Metal command buffer errors, memory pressure state from kern.memorystatus, and token latency percentiles (P50/P95/P99). Feed those into a time-series dashboard for monitoring across a fleet of Macs. The interesting part: when paging happens, it manifests as a spike in TTFT rather than an outright crash. The dashboard's real value is clustering by M-series chip and memory size to identify hardware-specific regressions after macOS updates — I'd expect M3 and M4 to show different thermal profiles, and the dashboard makes that visible before users report it.

The benchmark you should actually run: can a custom MLX kernel match Lily's bandwidth saturation? For a MoE model like Qwen3.6-35B-A3B, you'd write a custom Metal kernel in MLX that mimics Lily's fixed threadgroup and tiling strategy for that specific architecture, and measure TTFT and decode throughput across 32GB and 64GB Macs. What connects: MLX-LM, custom MSL kernels, the model, and a benchmark harness. What to watch for: you might get within a few percent of Lily's speed, but you'll still face generic array dispatch overhead. The bigger problem is MoE routing skew — you may need to precompute expert routing patterns to cache threadgroup assignments. This experiment tells you whether Lily's advantage is fundamental or just a matter of engineering effort.

Is Lily the right model for your stack? A decision guide

The decision isn't subtle, and I'll make it blunt: if you can't commit to the Qwen3.6-35B-A3B + 32GB+ M-series combo, don't adopt Lily. That's not a hedging recommendation — it's the only honest answer, because everything about Lily's value proposition depends on that fixed pairing.

Here's the rule of thumb I'd put on a whiteboard. If you're building a single-model agent appliance on Apple Silicon with fixed hardware — a product where the local model is a deliberate design decision, not a moving target — Lily's approach is unbeatable. You're paying a one-time engineering cost to saturate a known hardware+model combination, and that pays off in every user interaction. If you need model portability, cross-platform support, or frequent fine-tuning, you're better off with MLX or llama.cpp. They'll leave some bandwidth on the table, but they'll let you ship a new model in days instead of months.

The maintenance cost is the tax on Lily's specialization. Every model upgrade is a kernel re-tune. Every Apple silicon generation is a re-validation. If your product roadmap has quarterly model changes, that tax will eat the latency benefit whole.

I want to end with a genuinely opinionated take: I'd rather see more appliance engines like Lily than another general runtime. The trend in agent products is fixed-function coprocessing — dedicated hardware/software pairs that do one thing optimally, not universal runtimes that do everything acceptably. Lily is a bet that the consumer AI market will reward that specialization, and I think it's a bet worth making. If you're building an agent platform, the question isn't "which inference framework is best?" — it's "what is my product's fixed function, and am I willing to build an appliance around it?"

Resources

• Perplexity · GitHub