Rethinking the Routing Layer: A Deep Dive into Decision 2.0 and Single-Pass Foundation Classifiers

The False Dilemma Between 2ms Vectors and 500ms Generative Routers

Every time I design an ingress layer for an agentic mesh or multi-model gateway, I run straight into an architectural compromise that feels archaic. We routinely force our routing, safety validation, and intent classification into one of two extremes: sub-symbolic vector lookups or heavy autoregressive decoders.

The vector route is seductive because of its raw speed. Dropping a tool like Aurelio AI's semantic-router or an embedding centroid matcher into an API gateway gives you sub-5ms route decisions on commodity CPU cores. But vector proximity is purely topical; it does not reason. The moment an incoming query relies on boolean negation ("I want to cancel my subscription, do not renew it under any circumstance"), an embedding matcher locks onto the lexical neighborhood of "renew" and misroutes the payload. The same breakdown happens with conditional routing rules ("Route to enterprise tier only if the user query contains a technical deployment error and billing dispute"). Embedding routers cannot resolve compositional logic or evaluate scalar risk thresholds. Fine-tuned cross-encoders like DeBERTa-v3 handle classification nuances better, but their label spaces are statically baked at training time; updating a routing policy or adding a transient intent requires assembling new datasets and retraining the head.

To bypass this rigidity, production architectures often swing to the opposite extreme: throwing small autoregressive models like Llama-3.1-8B, Qwen-2.5-7B, or GPT-4o-mini at the problem using Outlines or JSON schema constraints. While this unlocks zero-shot steerability—allowing you to update dispatch rules via natural language prompts—it extracts a punitive tax on latency and compute. Even for a tiny classification payload, an autoregressive model requires an initial prefill phase followed by sequential decoding loops across multiple generated tokens, consuming tens to hundreds of megabytes of KV-cache memory and taking anywhere from 200ms to over a second. Placed on the ingress critical path, that overhead destroys the overall time-to-first-token budget before downstream generation even starts.

quadrantChart
    title Routing Architecture Latency vs Zero-Shot Reasoning
    x-axis Low Latency --> High Latency
    y-axis Low Reasoning Depth --> High Reasoning Depth
    quadrant-1 High Latency High Reasoning
    quadrant-2 Low Latency High Reasoning
    quadrant-3 Low Latency Low Reasoning
    quadrant-4 High Latency Low Reasoning
    Aurelio semantic-router: [0.08, 0.22]
    DeBERTa-v3: [0.18, 0.35]
    Decision 2.0 Kai to Vega: [0.28, 0.82]
    Outlines Llama-3-8B: [0.85, 0.88]

This is why I view Decision 2.0 (vllm-sr) as an essential architectural correction. By providing an open suite of foundation discriminative models spanning from Kai-0.6B up to Vega-27B, the project occupies the unpopulated space between static embeddings and sequential generation. You define arbitrary criteria dynamically in natural language, yet execution terminates within the single prefill pass. It delivers promptable zero-shot evaluation in 15 to 45 milliseconds without allocating a KV-cache or spinning up sequential decoding loops.

Inside the Forward Pass: How System One Bypasses Sequential Decoding

The fundamental mechanical bottleneck of generative routers is their reliance on the causal decoding loop. When you prompt a standard causal language model with a guided JSON schema, the engine computes the full attention matrix across the prompt during the prefill phase, writes key-value projections into VRAM, and then repeatedly samples the token distribution one step at a time until the closing brace of the JSON payload is emitted. Every generated token requires an individual memory-bandwidth-bound kernel launch.

Decision 2.0 completely bypasses this loop by running purely as a discriminative "System One" engine. In my analysis of the vllm-sr execution flow, the model is initialized via AutoModel.from_pretrained(..., trust_remote_code=True) and invoked through a specialized execution path: model.system_one(state=..., questions=...). Rather than instantiating an autoregressive text-generation loop, the forward pass packages the contextual state alongside the question criteria, routes them through the transformer's bidirectional or causal attention blocks, and routes the final hidden representations directly into custom classification and regression heads.

Because the model does not produce text, no KV-cache is allocated in GPU memory. The compute profile is strictly bounded to the initial transformer forward pass. For a mid-sized variant like Decision-2.0-Lux-9B, this collapses the execution time down to approximately 18ms for median context lengths.

sequenceDiagram
    autonumber
    actor Client
    participant Gateway as Envoy ExtProc Gateway
    participant D2 as Decision 2.0 Engine
    participant Downstream as Specialist LLM Cluster

    Client->>Gateway: Inbound User Request
    Gateway->>D2: Call system_one(state, questions)
    Note over D2: Single forward pass over prompt & criteria<br/>No KV-cache allocation or token generation loops
    D2-->>D2: Compute pooled logits for choice & yes_no heads
    D2-->>D2: Validate logit margins against confidence floor
    D2->>Gateway: Return discrete route targets & probabilities (18ms)
    Gateway->>Downstream: Dispatch payload to matched specialist backend
    Downstream->>Client: Stream generative response

Crucially, this architecture resolves the confidence calibration flaw endemic to constrained decoding. When you force a generative model to produce structured JSON via schema masking, the token probabilities inside the output sequence are heavily uncalibrated; models routinely express extreme, unearned certainty on token logits while hallucinating the underlying classification choice. In contrast, Decision 2.0 extracts discriminative logits directly over the explicitly defined options. The resulting softmax distributions across choices and sigmoid outputs across binary gates represent true posterior probabilities. This allows me to set rigorous programmatic margins—such as requiring a top-choice probability delta of at least 0.25 before routing—and predictably shunt low-confidence queries into defensive fallback pipelines.

Evaluating Decision Primitives: Choice, Yes-No, and Multi-Question Batching

In production orchestration, routing decisions are rarely isolated. An ingress gateway typically needs to determine query destination, evaluate safety boundaries, and score input urgency simultaneously. Under standard architectures, this forces teams to assemble serial pipelines: a guardrail model (such as Llama Guard) runs first, followed by an intent classifier, followed by a priority scorer. Chaining these models multiplies infrastructure overhead and introduces cumulative latency that quickly blows past microservice service-level objectives.

Decision 2.0 solves this serialization penalty by structuring evaluations around three atomic primitives that execute concurrently over a shared input state in a single forward pass:

1. choice: A categorical dispatcher that evaluates the state against an arbitrary array of natural language descriptions. The model projects the contextual representation against the candidate definitions and returns a normalized softmax distribution over the candidate keys.
2. yes_no: A binary validation primitive that checks the state against a specific condition or policy rule. It bypasses multi-class competition to emit a single calibrated probability score for whether the predicate holds true, making it ideal for content moderation, policy enforcement, or PII detection.
3. score: A continuous evaluation primitive that maps the context against qualitative criteria to output a scalar value within a bounded numerical range, removing the need to parse synthetic numeric tokens out of generated strings.

Primitive	Prompt State Schema	Head Pooling & Transformation	Output Distribution Format	Primary Gateway Role
choice	Unbounded context state + list of named candidate route strings	Sequence hidden states pooled across route criteria tokens; evaluated via multi-class softmax	Discrete dictionary of normalized float probabilities across keys	Dynamic intent classification and downstream specialist agent routing
yes_no	Unbounded context state + target boolean policy statement	Task-specific classification head pooling; computed via single-logit sigmoid	Scalar float probability representing affirmative truth value	Real-time guardrailing, prompt-injection defense, and policy enforcement
score	Unbounded context state + continuous evaluation instruction rubric	Latent representation pooled into linear regression head	Bounded scalar float score	Query complexity scoring, urgency tiering, and model tier selection

The structural advantage of this architecture is its ability to handle multi-question batching. Instead of launching three separate model evaluations, I can pass a compound dictionary containing multiple choice, yes_no, and score targets into a single model.system_one() invocation. The underlying transformer processes the shared state representation through its attention layers once, branching at the output stage into the corresponding pooling heads.

This compound evaluation allows an API gateway to determine whether an enterprise customer's query is safe, extract its technical intent, and compute its urgency tier in a single 20ms pass. While this pattern demands strict monitoring of schema context size to prevent attention dispersion across long prompt definitions, it collapses the latency overhead of multi-stage ingress validation into a single, predictable compute step.

Sizing the VRAM Footprint: When to Run Kai-0.6B Versus Vega-27B

Deploying a discriminative model into an ingress critical path requires sobering hardware math. It is easy to look at a 15–20ms execution profile on a 9B model and declare victory over autoregressive decoders, but VRAM economics quickly complicate that calculation. Running Decision-2.0-Lux-9B in standard BF16 demands roughly 18GB to 20GB of dedicated VRAM. If your downstream architecture serves an 8B or 14B worker model, dedicating an entire NVIDIA L4 or A10G instance exclusively to route traffic into a downstream worker of comparable parameter volume is an economic failure mode.

The Decision 2.0 lineup spans two distinct operational tiers: edge-viable lightweight sentinels (Kai-0.6B, Eos-0.8B, and Sol-2B) and heavyweight policy engines (Nox-4B, Lux-9B, and Vega-27B).

Model Variant	Parameter Size	BF16 VRAM Footprint	Max Context Bound	Target Hardware Profile	Primary Deployment Role
Kai-0.6B	0.6B	~1.3 GB	4k–8k tokens	CPU / edge node / shared GPU slice	High-throughput intent classification and basic syntax gates
Eos-0.8B	0.8B	~1.8 GB	8k tokens	CPU / T4 / edge accelerator	High-volume binary policy filtering and prompt sanitization
Sol-2B (incl. Reasoning)	2.0B	~4.2 GB	16k tokens	Single low-cost GPU (L4, T4)	Multi-clause business rules, state trees, and model dispatch
Nox-4B	4.0B	~8.5 GB	16k tokens	Dedicated L4 / shared A10G	Compound intent parsing and nuanced PII identification
Lux-9B	9.0B	~19.0 GB	16k tokens	Dedicated A10G / L40S / half-A100	Multi-predicate ingress orchestration across enterprise queues
Vega-27B	27.0B	~56.0 GB	16k tokens	Full A100-80GB / H100	Deep regulatory compliance, legal risk checks, and complex taxonomy mapping

My rule of thumb for hardware allocation centers on rule complexity rather than input volume. For 80% of standard gateway routing—separating billing questions, technical support, and general product inquiries—Kai-0.6B or Eos-0.8B easily captures the semantic boundary. They run with negligible memory footprints, can be hosted alongside gateway proxies or even served on high-core CPU instances, and resolve routing states within 5 to 10 milliseconds.

I only allocate dedicated GPU resources to Lux-9B or Vega-27B when the criteria strings require interpreting subtle regulatory postures or corporate policy exceptions with dense domain nuance. If an enterprise rule demands evaluating compliance across dense legal documents where definitions are legally fragile, the zero-shot representation of a 27B model prevents classification drift. When a decision tree involves nested conditional dependencies, however, stepping up to Vega is often the wrong remedy; that is the explicit domain where the dedicated Sol-2B reasoning variant outperforms raw parameter brute force by navigating multi-stage rule dependencies without bloating your infrastructure footprint.

Where Non-Generative Decisioning Breaks: Multi-Hop Logic and Attention Dilution

The single forward pass is Decision 2.0's core performance advantage, but it is also its primary architectural boundary. Because execution terminates directly at pooled classification heads without emitting intermediate scratchpad tokens, the model possesses zero capacity for internal chain-of-thought deduction.

When you ask a standard generative model to handle a multi-hop policy—such as "If the subscription is Enterprise and past due is less than 30 days, route to Retention; but if past due exceeds 30 days and the tenant has an open legal dispute, route to Risk"—it uses generated tokens as working memory to resolve intermediate predicates sequentially. Decision 2.0 cannot do this. Its hidden states must compress the entire contextual input and all candidate criteria in a single representation step. Under nested conditional logic, the model collapses into lexical correlation: it latches onto salient tokens like "Enterprise" or "dispute" independently, failing to synthesize the relational operator between them. For compound rules, expecting standard single-pass heads to behave like deductive rule engines leads to silent, high-confidence misrouting.

flowchart TD
    subgraph Attention Dilution & Semantic Bleed Breakdown
        Inbound["Inbound State (User Query + Context)"]
        
        subgraph Overloaded Ingress Prompt
            P1["Criteria A: 'Technical fault resolution'"]
            P2["Criteria B: 'System integration inquiry'"]
            Q1["Question 1: Routing Target"]
            Q2["Question 2: Compliance Violation"]
            Q3["Question 3: Urgency Score"]
            Q4["Question 4: Sentiment Assessment"]
        end
        
        Inbound --> AttentionLayer["Transformer Attention Block"]
        P1 -.->|Diffuse Attention| AttentionLayer
        P2 -.->|Diffuse Attention| AttentionLayer
        Q1 -.-> AttentionLayer
        Q2 -.-> AttentionLayer
        Q3 -.-> AttentionLayer
        Q4 -.-> AttentionLayer
        
        AttentionLayer --> Pool["Classification Pooling Heads"]
        Pool --> Flap["Split Logits: 51% / 49% (Route Flapping)"]
        Pool --> Dropped["Degraded Recall on Later Questions (Q3, Q4)"]
    end

The second failure mode is semantic bleed across non-orthogonal criteria. When defining candidate strings for the choice primitive, developers frequently supply overlapping categories, such as "Technical integration inquiries" versus "Product setup and faults." If an incoming query contains elements of both, the model's discriminative head projects roughly identical cosine similarity across both candidates. The resulting softmax distribution flattens—frequently yielding split confidence distributions like 51% to 49%. Under this condition, minor paraphrasing or client prompt edits trigger route flapping. If your criteria boundaries are not strictly orthogonal, the model cannot establish clean separation.

Finally, operators must guard against attention dilution caused by schema bloat. While Decision 2.0 supports batching multiple questions into a single forward pass, every added question and verbose criteria rubric consumes sequence length within the shared attention window. In lower-parameter variants like Kai-0.6B and Eos-0.8B, attention weights become diffuse across bloated prompt schemas. The model struggles to attend effectively to the actual input state, leading to severe recall degradation on questions placed near the end of the schema. Treating the dynamic prompt like an unbounded configuration file undermines the very latency and reliability gains the architecture was built to deliver.

Hardening the Gateway: Margin Thresholds, Supply Chains, and Context Clamping

Deploying a discriminative foundation model into the hot path of an ingress proxy requires a fundamentally different operational posture than running a standard generative LLM behind an async queue. If an autoregressive model hallucinates, downstream parsing layers or defensive schemas can catch the failure mode. If a single-pass classifier misroutes at the gateway, the error propagates downstream immediately. Hardening Decision 2.0 requires addressing three core operational realities: probability margin thresholding, model loading supply chains, and sequence-length context clamping.

First, never wire routing execution directly to an argmax output. When evaluating dynamic criteria in natural language, borderline queries or overlapping definitions cause probability mass to split across competing classes. A query yielding a 51% versus 49% distribution between two routes will cause persistent route flapping under trivial lexical variations. In my own gateway configurations, I enforce an explicit margin delta threshold:

$$\Delta(P_{\text{top1}}, P_{\text{top2}}) \ge 0.25$$

If the probability margin between the primary and secondary class fails to clear this floor, the gateway treats the decision as ambiguous. Instead of rolling the dice on a marginal winner, the request falls back to a deterministic path: escalating to a human-in-the-loop review queue, defaulting to a conservative triage agent, or spinning up an autoregressive model to execute deliberate chain-of-thought analysis.

Second, the trust_remote_code=True requirement is an immediate blocker in hardened enterprise environments. Pulling custom Python execution logic dynamically from the public Hugging Face Hub at container startup introduces severe supply-chain exposure. To run this safely in production, teams must vendor the vllm-sr architecture files directly into an immutable, internally audited base image. Isolate the dependencies, verify the head definitions in a vulnerability scanner, and mount weights from internal blob storage or a locked private model registry. This decoupling also removes runtime network dependencies from your cold-start budget.

Finally, operators must strictly enforce aggressive context clamping on the input state. Decision 2.0 advertises context lengths up to 16k tokens, but attention remains $O(N^2)$ in compute complexity. Feeding an entire unbounded conversational history or multi-megabyte document into model.system_one() causes the forward pass latency to degrade from 18ms to several hundred milliseconds, blowing through proxy timeout budgets and destroying the gateway's time-to-first-token advantage. Ingress controllers must pre-truncate conversational state to the minimal salient window—extracting only the immediate user turn, the prior assistant turn, and compact metadata flags—before passing the payload to the discriminative heads.

Three Production Architectures Built on Foundation Decision Models

Foundation decision models shine brightest when an architecture requires zero-shot steerability without paying the latency and compute taxes of sequential decoding loops. I see three primary blueprints where deploying Decision 2.0 fundamentally outclasses traditional embedding matchers and generative guardrails.

1. Unified Ingress Gateway: Compound Routing and Policy Enforcement

In modern gateway design, separating policy validation from routing logic doubles infrastructure roundtrips. A unified ingress gateway consolidates both steps into an Envoy proxy running an External Processing (ExtProc) filter that communicates with a local inference sidecar hosting Decision-2.0-Lux-9B over gRPC.

flowchart LR
    Client["Client Request"] --> Envoy["Envoy ExtProc Filter"]
    Envoy --> Sidecar["Python Sidecar: Lux-9B (vLLM-SR)"]
    
    subgraph SFP["Single Forward Pass (18ms)"]
        Sidecar --> CheckPolicy["yes_no: Compliance Violation"]
        Sidecar --> CheckRoute["choice: Specialist Model Target"]
    end
    
    CheckPolicy --> Block{"Violation Detected?"}
    Block -- Yes --> Reject["HTTP 403 Gateway Block"]
    Block -- No --> Dispatch["Route to Target Specialist LLM"]

When an inbound user prompt hits Envoy, the ExtProc filter formats the raw payload into a unified input state. The sidecar passes this context into a single model.system_one() invocation evaluated against one yes_no compliance question (checking for policy violations and corporate data leakage) and one choice question (directing traffic between specialized downstream models such as billing, coding, or customer support). Evaluating both predicates concurrently in roughly 18ms keeps the entire ingress overhead well within strict microservice SLO budgets.

The vulnerability to defend against here is adversarial prompt injection. Because untrusted user text enters the state context directly, malicious prompts can attempt to manipulate attention representations to force a false compliance classification. Protect this interface by wrapping user text in strict structural delimiters and enforcing fixed system-level instructions that the discriminative heads are calibrated against.

2. Confidence-Gated Model Tier Dispatcher with Ambiguity Escalation

Cost optimization gateways often attempt to predict whether an incoming query requires a heavyweight frontier model or a cheaper small language model. Building this pattern on Decision-2.0-Sol-2B wrapped with a FastAPI layer provides a dynamic, low-latency dispatcher. The model runs two primitives concurrently: a score primitive that measures qualitative task complexity on a bounded scale, and a choice primitive that maps the request across technical domains.

The service uses the continuous complexity score to set the initial model tier, but gates the final dispatch decision on class confidence. The API layer computes the probability delta between the top two domain classes. If ambiguous phrasing causes criteria drift and flattens the probability margin below 0.25, the gateway avoids a misroute on the smaller model by automatically escalating the request to the frontier model tier. This delivers reliable cost reduction across high-volume commodity queries while preserving strict execution quality for difficult edge cases.

3. Agentic State Machine Router with Zero-Shot Branch Qualification

In autonomous workflows orchestrated via LangGraph or custom state graph controllers, agents frequently need to inspect intermediate state to decide the next step: calling a database retrieval tool, continuing text synthesis, or requesting human clarification. Replacing autoregressive JSON routing with Decision-2.0-Nox-4B eliminates multi-hundred-millisecond pauses at every node transition.

The graph controller passes the current agent execution state into a multi-question schema that qualifies tool necessity via choice and evaluates task completion via yes_no in a single forward pass. To maintain execution speed across long-running multi-turn agent sessions, implement strict context clamping that isolates only the last two conversation turns and immediate tool execution outputs. Restricting the context length insulates the attention window from quadratic latency scaling, keeping branch qualification snappy, predictable, and deterministic.

Resources

• 🎲 Decision 2.0 - a vllm-sr Collection
• vllm-sr/Decision-2.0-Lux-9B · Hugging Face
• vllm-sr/Decision-2.0-Sol-2B · Hugging Face