Inside Lily: Why Perplexity Ditched General-Purpose Frameworks for Hand-Crafted Metal Kernels

The Architecture: A Single Rust Process Driving Custom Metal Kernels

The first thing that struck me about Lily is not what it does, but what it removed. Most local inference stacks I've worked with are accretions — you start with a tensor framework, add a model loading library, bolt on a server, and hope the layers don't fight each other. Lily throws that away. It's a single Rust process that owns the entire execution path, from checkpoint bytes on disk to tokens streaming out over an HTTP socket.

The Rust runtime loads model checkpoints directly into Apple's unified memory. There's no PyTorch, no MLX, no intermediate representation. When I say "no MLX in the execution path," I mean it literally — the token generation loop, session state management, memory layout, and kernel scheduling are all orchestrated by Rust code that knows exactly what it's running. This isn't a framework that happens to be written in Rust; it's a purpose-built runtime where the control flow and the compute kernels were designed together.

The contrast with MLX-LM is instructive. MLX gives you NumPy-like composable array semantics with lazy evaluation. You build a model from reusable primitives, and the framework figures out how to schedule them on the GPU. It's elegant, it's general, and it pays a cost for that generality — every operation dispatches through an abstraction layer that adds latency and prevents the kind of aggressive specialization you'd want for a specific model.

llama.cpp takes a different route: a C++ compute graph executor with GGUF-format weights, dispatching chunks to Metal shaders. It's portable, it's ubiquitous, and its Metal kernels are generalized across transformer variants. For dense models that's fine. For the irregular workloads of sparse hybrid architectures, those generalized kernels leave performance on the table.

Lily's approach is the vertical specialization play. Instead of dispatching through a generic tensor library, the Rust runtime couples directly to hand-crafted Metal compute kernels written specifically for Qwen3.6-35B-A3B's operations. Kernel dispatch, memory movement, and GPU threadgroup layouts are tailored to the hardware and the model together. The runtime knows the exact memory layout the checkpoint expects. It knows which kernels to invoke for MoE routing. It knows how to sequence recurrent state transitions. Nothing is generic.

flowchart TD
    A[Checkpoint file] --> B[Rust runtime loads weights into unified memory]
    B --> C[Session state allocated: KV cache + recurrent state]
    C --> D[Prefill phase: Metal kernels optimized for parallel compute across prompt tokens]
    D --> E[Decode phase: Metal kernels optimized for memory-bandwidth-bound autoregressive generation]
    E --> F[Token streamed over OpenAI-compatible chat completions SSE endpoint]
    F --> C

Another design decision worth noting: the OpenAI-compatible API is embedded in the same process. There's no separate server hop, no IPC boundary between the inference engine and the API layer. When the Mac app calls Lily, it's talking directly to the runtime that owns the model weights and the token loop. For an embedded desktop backend, that's the right call — you eliminate serialization overhead and network latency that would otherwise add tens of milliseconds to every request.

The tradeoff is obvious and worth stating plainly: this is not a general-purpose framework. Drop in a dense Llama or a different MoE topology, and you're looking at rewriting Metal kernels and Rust execution graphs. The specialization is the point, and the cost is rigidity. If you need broad architecture support, use MLX or llama.cpp. If you need maximum throughput on a specific hybrid model on Apple Silicon, Lily is the answer.

Prefill vs. Decode: A Phase-Split That Actually Matters

The prefill/decode split is one of those architectural decisions that sounds like obvious engineering until you realize how few local inference engines actually do it properly. Generic frameworks use the same kernels for both phases, treating prompt processing and autoregressive generation as variations of the same tensor operations. Lily doesn't. It maintains separate execution plans and separate kernel choices for each phase, and that separation is the source of much of its throughput advantage.

The reason is that the two phases are bottlenecked by fundamentally different hardware resources. Prefill is compute-bound. You're processing hundreds or thousands of prompt tokens in parallel, hammering the GPU's arithmetic units with dense matrix multiplications. The goal is maximum parallel compute saturation — keep every SIMD lane busy, keep the tensor cores churning.

Decode is memory-bandwidth-bound. You're generating one token at a time, and each token requires reading the entire model's weights from memory. The arithmetic work per token is trivial relative to the memory traffic. The bottleneck is how fast you can move the weights from unified memory into the compute units. Dispatch latency and scheduling overhead become the enemy — every abstraction layer between the kernel and the hardware adds microseconds to a loop that's already memory-stalled.

On Apple Silicon's unified memory architecture this divergence is especially pronounced. CPU and GPU share the same physical memory, so there's no PCIe transfer or host-to-device copy to worry about. But that also means the bandwidth is shared, and the phase-dependent bottleneck is purely about how the kernels are scheduled. Lily's Rust runtime knows which phase it's in and selects kernels accordingly. No dynamic dispatch, no generic operator fusion — just a direct path to the tuned Metal kernel for that specific workload shape.

stateDiagram-v2
    [*] --> Idle
    Idle --> LoadingCheckpoint: Request received
    LoadingCheckpoint --> Ready: Weights loaded
    Ready --> Prefill: New prompt tokens
    Prefill --> Decode: Prompt processed
    Decode --> Streaming: Token generated
    Streaming --> Decode: Next token
    Streaming --> Ready: Turn complete (session state persists)
    Ready --> Idle: Session evicted
    Prefill --> Fallback: Error or OOM
    Decode --> Fallback: Error or OOM

The multi-turn behavior matters too. After streaming completes, the session state persists — both the KV cache for attention and the recurrent state for the hybrid layers. Next turn, prefill only processes the new tokens. This is standard for KV caches, but for recurrent states it's more delicate: they're strictly sequential, so you cannot selectively recompute or slice them like attention positions. The runtime has to manage state transitions with precision, and the state diagram above shows the error paths that would trigger a fallback to cloud — if you're running on a lower-end Mac, or memory pressure kills the process mid-decode, you need that escape hatch to preserve the user experience.

Sparse Hybrid Models: Where MoE Routing Meets Recurrent States

The reason Lily exists — the reason you'd build a custom inference engine rather than use MLX-LM — is the specific architecture of Qwen3.6-35B-A3B. It's a sparse hybrid model: 35 billion parameters total, but only about 3 billion active per token. That sparsity is what makes local inference on a desktop feasible, but it also creates the hardest optimization problems I've seen in local serving.

The Mixture-of-Experts layer is the first challenge. Each token dynamically routes to a subset of expert weights, and that routing is input-dependent. From the GPU's perspective, this means irregular memory access patterns. One batch of tokens might activate expert 3 and expert 7 heavily while leaving expert 12 idle; the next batch inverts that entirely. Generic matrix multiplication kernels — the kind that make up MLX's composable operator set — don't handle this well. They're designed for dense, regular memory access. When tokens route unevenly to experts, you get warp divergence and threadgroup under-occupancy: SIMD lanes sit idle while other lanes hammer the same expert weights.

Lily handles this by making expert dispatch a first-class concern in the Metal kernels. The runtime coordinates token-to-expert routing directly on GPU threadgroups, meaning the kernel knows which experts each threadgroup should process and can schedule memory fetches accordingly. It's not a generic tensor operation that happens to support MoE; it's an execution plan built around the routing pattern.

The recurrent state is the second challenge, and it's arguably more interesting. Fixed-size recurrent states impose sequential dependencies — each token's recurrent state depends on the previous token's state, which means you can't parallelize across the sequence the way you can with attention. You need low-latency sequential state transitions, not massive parallel tensor contractions.

MLX-LM and llama.cpp treat recurrent states as modular extensions of a standard transformer loop. They add the state mechanism on top of the generic execution framework, which incurs overhead — the state update becomes another operation in the graph, another dispatch point, another abstraction to traverse. Lily treats recurrent state transitions as primitive operations in the kernel execution schedule, coupled directly to the Metal kernels that perform them.

The result is that a 35B-parameter model with only 3B active per token becomes tractable on a 128GB unified memory Mac like the M5 Max. You're not loading the full 35B into active compute — you're streaming expert weights on demand while the recurrent state handles the sequential aspect of generation. It's a fundamentally different execution model than dense transformer inference, and it deserves a runtime built for it rather than adapted to it.

Why Not Just Use MLX or llama.cpp? A Trade-Off Analysis

The question every engineer asks when they see a custom inference engine is: why not just use what already exists? It's a fair question, and the answer reveals the fundamental tension in local inference. Here's how the landscape actually stacks up:

Runtime	Language	Compute Backend	Model Generality	Performance Focus	Integration Effort
Lily	Rust	Hand-crafted Metal kernels	Very low (Qwen3.6-35B-A3B only)	Maximum throughput on sparse hybrid models	High per-model, low per-deployment
MLX-LM	Python / C++	Composable Metal kernels	High (dozens of architectures)	General-purpose Apple Silicon serving	Low for new models
llama.cpp	C / C++	ggml-metal	Very high (broadest ecosystem)	Portability and quantization breadth	Low for existing formats
Candle / Mistral.rs	Rust	Metal backend	Moderate to high	Safe, lightweight, no Python	Medium

The key insight is that these aren't competing on the same axis. MLX-LM and llama.cpp compete on breadth — how many models you can run with minimal effort. Lily competes on depth — how fast you can run one specific model on one specific hardware profile. Those are different engineering problems with different optimal solutions.

MLX-LM is the natural default for most Mac developers. It's first-party from Apple, supports dozens of architectures out of the box, and the composable array semantics make adding a new model a matter of writing a Python model definition. The cost is that every operation dispatches through a generalized kernel. For dense models, that overhead is negligible. For sparse hybrid architectures with MoE routing and recurrent states, it's the difference between a usable local experience and a frustrating one. The abstraction layer that gives you flexibility is the same layer that caps your decode throughput.

llama.cpp wins on portability and ecosystem. GGUF quantization formats are mature, the community is massive, and you can run the same binary on a Mac, a Linux server, or a Windows box. But its Metal kernels are generalized across transformer variants. When you need recurrent state transitions sequenced with precision or MoE routing dispatched to maximize threadgroup utilization, those generalized kernels simply don't have the granularity. They're optimized for the common case, and sparse hybrid architectures are anything but common.

Candle and Mistral.rs are the Rust-native option, and they're close cousins to Lily's philosophy — eliminate the Python runtime, get memory safety, ship a lightweight binary. But they still implement generalized tensor graph abstractions over their Metal backend. They're a safer, faster version of MLX's approach, not a departure from it. Lily goes further: the Rust runtime couples directly to specific Metal execution buffers and phase-split kernels, with no tensor graph in between.

The trade-off is stark. Lily is production-ready for exactly one model on exactly one hardware platform. It's not a framework you prototype with; it's a runtime you deploy. If you need to swap in a different architecture, you're not adjusting a config file — you're writing Metal Shading Language. That's a real engineering cost, and teams should budget for it honestly. But the payoff is an inference engine that actually keeps up with the irregular memory access patterns and sequential dependencies of modern sparse models, rather than one that limps along on abstractions designed for dense transformers.

Where Lily Breaks in Practice: Model Lock-In and Rigid Execution Plans

The trade-off I discussed above isn't abstract — it manifests as a very concrete set of failure modes that anyone considering Lily for production should understand before committing.

The first and most severe constraint is model lock-in. Lily's performance edge isn't achieved through clever runtime optimizations that could be reused across architectures. It comes from Metal kernels written explicitly around Qwen3.6-35B-A3B's specific operations: its MoE routing scheme, its fixed-size recurrent state transitions, its memory layout. The Rust runtime has hardcoded execution plans that know exactly which kernels to invoke for each operation, in what order, with what memory buffer allocations. There's no dispatch table, no plugin system, no model-agnostic path through the engine.

This means that if you try to load a dense Llama model, you're not hitting an unsupported path — you're hitting a broken one. The runtime expects expert routing. It expects recurrent states. A standard transformer doesn't have those, and the execution plan doesn't know how to handle their absence. You'd be looking at rewriting Rust execution graphs and Metal kernels, not tweaking configuration.

Even subtle architectural revisions of Qwen itself will break things. Change the number of experts, alter the routing mechanism, adjust the recurrent state size — each of those touches the kernel-level assumptions that Lily was built around. The specialization is so tightly coupled that version upgrades of the target model are a major engineering undertaking, not a routine update.

Weight format inflexibility compounds this. Because Lily bypasses PyTorch, MLX, and Hugging Face's transformers stack entirely, it doesn't inherit any automatic weight format conversion. There's no dynamic dequantization on load, no support for GGUF, AWQ, or GPTQ variants, no tensor reshuffling to adapt a checkpoint to the engine's expectations. Checkpoints must arrive in exactly the memory layout and quantization schema that Lily's kernels expect. That's a significant burden for teams that want to experiment with different quantization levels or fine-tuned variants.

The second failure mode is hardware specificity. Lily's kernels are tuned for high-end Apple Silicon — the M5 Max with its 40-core GPU, the Ultra chips with their massive compute and memory bandwidth. Those kernels assume threadgroup sizes and dispatch patterns that work well on big chips. On a base M-series chip with fewer cores, or a Pro model with less unified memory, those same kernels can experience dispatch stalls or sub-optimal occupancy. The engine isn't just model-specific; it's hardware-tier-specific. A MacBook Air with 16GB of RAM isn't a degraded experience — it's potentially a broken one.

The third constraint is less obvious but equally important: Lily is not a developer SDK. It doesn't support training, fine-tuning, or even prototyping with alternative model definitions. It's an execution runtime for production deployment. If your team's workflow involves iterating on model architectures or custom fine-tunes, Lily won't accommodate that. You'd need to develop and validate the model on a separate stack, then port it into Lily's kernel expectations. That's a substantial process overhead.

Recurrent State Drift and MoE Load Imbalance: The Subtle Killers

Two failure modes in Lily's target architecture are particularly insidious because they don't manifest as crashes or obvious errors. They degrade silently, and by the time you notice, you're debugging user-facing quality issues that are hard to trace back to their root cause.

The first is recurrent state drift. Qwen3.6-35B-A3B combines standard attention with fixed-size recurrent states. Attention has a KV cache, and that KV cache is positional — you can index into it, slice it, recompute from any token position without disturbing the rest of the sequence. Recurrent states don't work that way. They're strictly sequential. Each state depends on the previous state, and the entire chain is a linear progression that can't be arbitrarily recomputed from a partial position.

This creates a problem in multi-turn chat scenarios. When a user edits a message in a conversation, or when the system wants to run speculative rollouts to compare alternative continuations, you need to branch the state. With attention KV caches, you can save a snapshot and rewind — it's just a matter of storing position indices. With recurrent states, branching requires a checkpoint-based rollback. If Lily's Rust state manager doesn't handle this precisely, you get state corruption: the recurrent hidden state no longer matches the conversation history, and generated tokens degrade in ways that are hard to diagnose.

The practical implication is that session management has to be over-engineered. Every potentially branching operation must checkpoint the recurrent state. Every session teardown must flush those states cleanly. And if you're running long conversations with frequent edits, you're accumulating redundant checkpoints that consume memory. A production deployment needs explicit policies for session eviction and state lifecycle management, otherwise you'll leak memory or corrupt context across idle periods.

The second failure mode is MoE load imbalance. The sparse activation that makes local inference feasible also creates irregular memory access patterns. When tokens route disproportionately to a few experts, the Metal kernels that map expert dispatch onto GPU threadgroups face a challenge. Some threadgroups are processing heavily-requested experts while others sit idle. You get SIMD lane divergence — expensive on any GPU, but particularly costly on Apple Silicon where the threadgroup architecture assumes predictable work distribution.

This isn't just a theoretical concern. The model's routing behavior is input-dependent, so some prompts will naturally produce balanced routing while others will concentrate on a small subset of experts. If you're serving a workload with consistently skewed token distributions, the throughput degradation is real and consistent. The kernels that Lily has tuned for Qwen's routing patterns assume a certain balance; when that balance shifts, the threadgroup scheduling becomes suboptimal, and you lose the performance advantage that motivated building the engine in the first place.

For production teams, this means you need to measure routing distribution on your actual workloads, not just on benchmark prompts. If your application's query patterns produce heavy concentration on a few experts, you may need to couple Lily with specialized scheduling logic or accept degraded performance on those specific inputs.

Client-Side Realities: Memory Pressure, Thermal Throttling, and macOS Gotchas

The reference benchmark for Lily is an M5 Max with 128GB of unified memory. That's a gorgeous machine, but it's not what most people own. Serving a 35B-parameter model — even one with only 3B active per token — on a typical 32GB or 64GB Mac requires real memory budgeting. You're not just loading weights; you're allocating KV cache, recurrent state buffers, and Metal working memory alongside the operating system, the Mac app, and whatever else the user has open. On a 64GB machine, you're probably fine with some headroom. On 32GB, you're squeezing, and every additional context window makes it tighter.

macOS makes this more dangerous than a headless Linux server would be. Linux has swap and predictable OOM behavior; macOS has Jetsam, which aggressively throttles or kills background processes under memory pressure. If Lily allocates large Metal buffers without cooperating with macOS memory warnings, it doesn't degrade gracefully — it just dies. Mid-prefill, mid-decode, gone. For an engine that's meant to be embedded in a desktop app, that's a user experience disaster.

Thermal throttling is the quieter problem. Sustained local inference on a 40-core GPU generates serious heat. On a MacBook Pro, running extended prefill or long generation sessions will push the thermals up, and clock speeds drop. Decode rates degrade over time, sometimes significantly. The first minute of generation might be fast; the tenth consecutive minute won't be. Laptop users also pay a battery cost that's non-trivial. This isn't a bug in Lily — it's physics — but it's a constraint you must design around if your app does long-form generation.

Metal API compatibility adds another layer of variance. Kernels tuned for high-end Max and Ultra chips — with their specific threadgroup sizes and dispatch patterns — can experience dispatch stalls or sub-optimal occupancy on base and Pro chips. The M1, M2, and M3 have different GPU architectures than the M4 and M5. Lily's specialization is hardware-tier-specific, not just model-specific. A kernel that saturates an M5 Max might underutilize an M3 Pro, and there's no configuration switch to fix that — you'd need to tune the Metal kernels themselves.

Finally, remember what Lily is not: it's not a serving engine. There's no continuous batching, no paged attention, no multi-tenant request queue. It's single-process, single-user, optimized for interactive latency on one stream. If you try to run it as a shared inference server for a team or a multi-user app, you'll hit limits fast. And its SSE streaming, while functional, doesn't have robust mid-stream error handling — an OOM during decode can truncate the response without a structured error payload, leaving the client hanging on a half-finished stream.

Project Applications: What You Can Build with Lily

Lily's specialization makes it a poor general-purpose inference engine but an excellent embedded backend for specific on-device workloads. Concretely, I see three categories of projects where its constraints become strengths.

A local document summarizer is the most natural fit. Build an app that reads local PDFs or documents, feeds them to Lily through its OpenAI-compatible API, and streams summaries back — all without the content ever leaving the machine. The prefill phase is your workload here, since processing long documents is compute-bound and benefits from Lily's parallel throughput across prompt tokens. Watch memory closely: a large PDF with a long context can push the engine's memory usage up sharply. If the user's machine is tight on unified memory, your app should detect the pressure and fall back to a cloud model with a clear indication of where the processing happened. Don't let Jetsam kill the process while the user is watching a progress bar.

A private code assistant embedded as an IDE plugin is a different set of trade-offs. Connect Lily's endpoint to editor events — file saves, selection changes, cursor position — and use its low-latency decode path for interactive suggestions while code stays local. The MoE routing and recurrent state handling matter less than raw generation latency here; you want fast token streaming for autocomplete-style responses. The gotcha is recurrent state drift. If the user edits code mid-session, that's a state branch, and a strictly sequential recurrent state doesn't handle branches well. You need to reset state on significant edits, not just append to the conversation. And code generation is exactly the kind of workload where Lily's model might not be fully optimized — some languages and patterns may produce mediocre output, so a hybrid approach with a cloud model for complex reasoning is worth building in from the start.

A private research agent is the most ambitious and maps most directly onto Perplexity's hybrid compute philosophy. Build a pipeline where Lily handles local file parsing, retrieval, and structured reasoning over notes, emails, and personal documents, while cloud models handle broad web research that requires massive context or frontier reasoning. You're not choosing one or the other — you're routing queries based on sensitivity and complexity. Lily's streaming API gives you structured outputs for local tasks, and its prefill specialization shines when you're processing large local corpora. Watch for MoE load imbalance: queries that concentrate routing on a few experts will degrade throughput unpredictably. And if you reuse sessions, monitor for state corruption — the recurrent state is sequential, so old sessions must be flushed cleanly. Build robust fallback logic. Lily being model-locked means if you want to swap to a different local architecture, you're rewriting kernels, not changing config.

Resources

• Optimizing On-Device Inference for Apple Silicon
• GitHub - perplexityai/pplx-garden: Perplexity open source garden for inference technology · GitHub
• GitHub - ml-explore/mlx: MLX: An array framework for Apple silicon · GitHub