PhoneLLM Alpha 1: Retraining Nemotron 3 Nano for the 650-Millisecond Voice-Agent Budget

The Latency Ceiling That Disqualifies Most Frontier Models

Voice agents live or die on a number most LLM comparisons never print: voice-to-voice latency, the time between the caller stopping and the agent's voice starting. The PhoneLLM model card from the Pipecat team puts the empirical bar at 1,500 ms P95 — their phrase, backed by what they call lots of data about caller tolerance: people are happy talking to an agent that answers within that envelope, and noticeably unhappy past it.

That 1,500 ms has to pay for everything: network overhead, audio capture and processing, application logic, speech-to-text inference, the language model's time-to-first-token, and text-to-speech synthesis. Their published breakdown of a well-optimized agent adds up to 1,293 ms, inside which the LLM's time-to-first-token target is 650 ms. Hold those two numbers and the design constraint becomes brutal: against a 650 ms budget, they report the P95 time-to-first-token of GPT 5.6 Terra in fast mode at about 1,900 ms. The LLM alone, on the fastest tier of a frontier API, exceeds the entire voice-to-voice target before a single audio frame is synthesized.

This is why a voice team fine-tuned its own model instead of prompting harder. No prompt engineering changes inference speed. The only levers are: a smaller or sparser model, infrastructure you control, and workload-specific serving optimizations. PhoneLLM Alpha 1 is the artifact of pulling all three, and it is worth studying even if you never deploy it, because the derivation — from latency budget to architecture choice to cost per agent-minute — is the actual methodology of building production voice systems in 2026.

What PhoneLLM Alpha 1 Actually Is

The facts from the model card, without the announcement polish:

• Lineage: a full-parameter supervised fine-tune of NVIDIA's Nemotron 3 Nano 30B-A3B, trained with the NVIDIA NeMo framework. Public base weights (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16), public result (pipecat-ai/phonellm-alpha-1).
• Architecture: hybrid Mamba-Transformer mixture-of-experts — 30B total parameters, 3.5B active per token. The sparsity is the point: you pay memory for 30B parameters of knowledge but compute only 3.5B, which is what lets one GPU host dozens of concurrent voice sessions.
• Context: 262,144 tokens. PhoneBench evaluation ran at 32,768 tokens with 32 sequences; the production-aligned serving recipe uses 65,536.
• Serving: vLLM or SGLang using the standard Nemotron 3 Nano recipes, with trust_remote_code=True because the Nemotron-H architecture ships custom code.
• Required generation settings: temperature=0 and thinking disabled via chat_template_kwargs={"enable_thinking": false}. The card is emphatic that these two settings align with how the model was trained — this is not a suggestion, it is part of the model's contract. The shipped generation_config.json even contains do_sample: true, which the tested deployment overrides explicitly at the server level.
• Language: English only. For a model aimed at US-style customer service verticals (financial services, healthcare, retail, hospitality are the card's examples), that is a scoping decision, not an oversight — but it is a hard limit for anyone else.
• License: BSD 2-Clause, no commercial restrictions, with NVIDIA Nemotron license obligations underneath (more on that below).

The training-data story is deliberately underspecified — Daily's end-to-end stack, real-world telephone and customer-support workloads, production agent traces from enterprise work. What is specified is the purpose: a model that invokes tools accurately in long multi-turn conversations with thinking disabled, which is precisely the operating mode voice latency budgets force you into.

Say/Do Consistency: The Failure Mode This Fine-Tune Targets

The card names the failure with unusual bluntness: models will say "Yes, I've booked that table for you" without ever calling the booking tool. In a chat product this is a quality bug. In a phone agent that just promised a caller a refund, a rescheduled appointment, or a canceled subscription, it is an incident — the caller hangs up believing something happened that did not.

Why this happens in voice conditions specifically is worth internalizing. Thinking modes — the chains of internal reasoning that dramatically improve tool-use accuracy on benchmarks — are exactly what the 650 ms budget cannot afford. Strip thinking out, and tool-call reliability degrades across model sizes; degrade it further with long multi-turn context, noisy ASR transcripts, and interrupt-happy callers, and general-purpose models start hallucinating confirmations. The fine-tune's entire reason to exist is to make tool invocation correct in the no-thinking regime: call the right tool, with the right parameters, at the right conversational moment, and — critically — not claim success the protocol did not deliver.

flowchart TD
    CALLER[Caller audio] --> STT[Streaming transcription]
    STT --> CTX[Conversation context<br/>multi-turn, includes partial ASR noise]
    CTX --> LLM[PhoneLLM Alpha 1<br/>temperature 0, thinking OFF<br/>3.5B active params]
    LLM -->|"text response"| TTS[Text-to-speech]
    LLM -->|"tool call"| VALIDATE{Tool executed and<br/>result confirmed?}
    VALIDATE -->|"yes"| SAY[Say/do consistent reply<br/>claim only what the tool returned]
    VALIDATE -->|"no, retry or error"| SAYFAIL[Repair turn:<br/>apologize, retry, or escalate]
    SAY --> TTS
    SAYFAIL --> TTS
    TTS --> AUDIO[Audio back to caller<br/>whole loop P95 target: 1,500 ms<br/>LLM first-token budget: 650 ms]
    AUDIO --> CALLER

The card's before/after fine-tuning comparison shows the same caller turns handled by base Nemotron 3 Nano versus PhoneLLM — the base model needing thinking tokens to approach correct tool behavior, the fine-tune doing it directly. I have not run this comparison myself; what I can verify from the published evaluation methodology is what they chose to measure, which tells you what they consider load-bearing.

PhoneBench v1: An Eval Built for Judged Subjectivity

PhoneBench v1 is released alongside the model, and its design choices are as instructive as the model itself. Voice-agent output is free-form and mostly not programmatically checkable — speaking style, factual grounding, and relevance resist assertion-based tests. Even tool usage resists it: a benchmark that demands one specific tool with specific parameters at a specific turn will fail a model that legitimately looked up the answer a different way.

The approach: a panel of LLM judges grades each turn against high-quality reference samples, with the judges calibrated against human labels. Measured dimensions read like an incident review checklist for a deployed phone agent: telephone speaking style, tool-call accuracy, say/do consistency, factual grounding, conversation coherence, authentication and escalation discipline, and caller outcome.

Two methodology points deserve scrutiny from anyone who consumes benchmark numbers:

1. Bench/train separation. The scenarios, tool lists, and system prompts in PhoneBench are kept separate from the data used to train PhoneLLM. The card frames this as ensuring the model generalizes to unseen scenarios, business types, tool sets, and prompts. That is the right claim structure — but note it is asserted, not proven, in the text I read; the separation discipline is described, while a leakage audit is not.
2. Aggregate reporting discipline. The numbers quoted for quantization decisions are means over ten complete PhoneBench runs at temperature zero, thinking disabled. Ten-run means on a judged benchmark is a reasonable hedge against judge variance, and publishing the variance-bearing process is better practice than most model cards manage.

The headline claim — performance on par with GPT 5.6 Terra on typical voice-agent tasks, 94 percent cheaper, with 1,300 ms faster P95 time-to-first-token — comes from this benchmark plus their latency sweeps. I take it as a calibrated vendor claim: directionally credible given the methodology, and also produced by the same team that trained the model. The mitigation is that they published the benchmark itself, so the claim is checkable rather than testimonial.

The Cost Derivation: $0.00025 Per Agent-Minute, Step by Step

The card's cost section is the most useful piece of infrastructure writing in it, because it shows the full arithmetic from GPU list price to per-agent-minute cost, with every assumption exposed. Reconstructed as a table:

Step	Value	Source/assumption
B200 base price (Modal)	$6.2496/hour	Modal list price
Region pinning (1.5x)	$9.3744/hour	Voice agents pin sessions to one cluster for cache locality
Utilization adjustment (÷0.70)	$13.392/hour	Target 70% utilization
Per-GPU cost per minute	$0.2232/minute	$13.392 ÷ 60
Concurrent agents per node	88	44 processes per B200; two GPUs per node = 88 pinned agents
Per-agent cost	$0.00025/minute	$0.2232 ÷ 88

Three things in this derivation deserve attention. First, the concurrency number is not theoretical peak — it is the result of sweeps against a "not to exceed" P95 time-to-first-answer-token target of 600 ms, i.e., the maximum load the hardware takes before the latency tail violates the voice budget. Cost and latency are one optimization, not two. Second, the 44-processes-per-B200 figure is credited to the Nemotron 3 Mamba-Transformer MoE architecture scaling efficiently on modern NVIDIA hardware plus workload-specific optimizations Modal built for PhoneLLM's traffic — the card says the AutoEndpoints configuration approximately doubles max concurrency versus the generic vLLM cookbook config at that same 600 ms target. Inference-stack engineering moved the cost-latency curve by 2x; that is a bigger lever than any model choice in the derivation. Third, every number is inspectable — they published a spreadsheet replicating per-minute cost across API and self-hosted models, and explicitly invite people to copy and audit it.

For scale: at $0.00025/minute, 10,000 hours of agent talk time costs about $150. Whatever your own assumptions do to these figures, the shape is the message — a sparsity-3.5B model on batched Blackwell serving makes voice-agent marginal cost a rounding error next to telephony minutes themselves.

Quantization: Where the FP4 Points Are

The BF16 release is 30B parameters of bfloat16 safetensors. The official NVFP4 checkpoint (pipecat-ai/phonellm-alpha-1-nvfp4) is the production-oriented artifact: 52 shards, 19.4 GB, selective NVFP4 weight quantization at group size 16 via ModelOpt 0.46.0rc1, targeting Blackwell — tested on a single B200.

The quality table, means over ten PhoneBench runs:

Weights	KV cache	PhoneBench mean	Delta vs BF16
BF16	BF16	72.055800	—
NVFP4	FP8	71.512482	-0.543318
NVFP4	BF16	72.019090	-0.036711

Read the middle row carefully, because it is the trap: quantizing weights to NVFP4 while also running an FP8 KV cache costs 0.54 benchmark points, but keeping the KV cache in BF16 costs only 0.04. The recommended production recipe is therefore NVFP4 weights with a BF16 KV cache — and the card discloses the price of that choice too: at the tested allocation, BF16 KV reduced available KV-token capacity by about 40.2 percent versus FP8. Quality, capacity, and context length are a three-way trade, and they printed all three terms.

The calibration methodology is unusually concrete: 1,000 deterministic, PhoneBench-disjoint rows spanning 273 conversations and all 692 observed strata, with the 50 longest eligible rows forced into the sample. Sensitive modules stay unquantized — the lm_head, selected attention and Mamba projections, and Mamba convolution modules — which is the standard pattern of protecting output heads and state-space recurrence from precision loss. The index counts are published down to the tensor (24,147 tensors; 5,968 of each scale type) alongside an immutable release manifest with per-file hashes.

Running It: The Tested Recipe and the Unvalidated Edges

The tested deployment is specific: vLLM 0.27.1 on one NVIDIA B200, with a long explicit flag list that encodes real architectural knowledge — --kv-cache-dtype bfloat16, --mamba-ssm-cache-dtype float32, --mamba-cache-mode align, FlashInfer attention, the flashinfer_cutlass NVFP4 MoE backend, async scheduling, chunked prefill, prefix caching, and — reflecting the no-thinking contract — a nano_v3 reasoning parser loaded from a plugin shipped inside the checkpoint, --default-chat-template-kwargs '{"enable_thinking":false}', and --override-generation-config '{"temperature":0}'. Tool calling is wired through --enable-auto-tool-choice --tool-call-parser qwen3_coder.

Two honesty markers in the card that I rarely see and respect: "Generic vLLM installations or non-Blackwell GPUs have not been validated for this checkpoint," and a warning against assuming Modal's AutoEndpoints product path applies the NVFP4 release recipe — the AutoEndpoints documentation currently covers the BF16 model. On the BF16 checkpoint, single-request P95 time-to-first-token is reported below 100 ms on a B200, which is what makes the 650 ms budget trivially meetable at concurrency one and becomes an interesting engineering problem only as load rises. SGLang and vLLM both work with standard Nemotron 3 Nano configs as starting points; Modal deployment of the BF16 model is one command (modal endpoint create --model pipecat-ai/phonellm-alpha-1).

sequenceDiagram
    participant C as Caller
    participant P as Pipecat framework
    participant S as STT (streaming)
    participant L as PhoneLLM (vLLM/SGLang on B200)
    participant T as Tool backend
    participant V as TTS
    C->>P: speaks
    P->>S: stream audio
    S-->>P: transcript tokens
    P->>L: context + tool schemas (thinking disabled, temp 0)
    Note over L: first answer token target ~650 ms P95<br/>measured single-request P95 < 100 ms
    L->>T: tool call (booking / lookup / auth)
    T-->>L: confirmed result
    L-->>P: response text claims only what tool returned
    P->>V: synthesize
    V-->>C: agent audio (whole loop P95 target 1,500 ms)

Licensing: BSD-2 on Top, Nemotron Underneath

PhoneLLM Alpha 1 ships under BSD 2-Clause — no commercial restrictions, self-host anywhere. But it is a derivative of a model under the NVIDIA Nemotron Open Model License, and the card spells out the chain: under Section 3 of the NVIDIA license, redistribution of PhoneLLM or your own derivatives requires including the NVIDIA license text and retaining NVIDIA copyright and attribution notices; BSD-2 terms govern Pipecat's modifications and the model as a whole, as that section permits. Practically: commercial deployment is fine, but your compliance checklist gains one line — ship the NVIDIA notice file with any redistribution, including container images you share externally. For internal use, nothing extra.

What I Would Test Before Trusting It in Production

This is an alpha (the name says so; 4.7K downloads at the time I read the card), and the card's own specificity about what is not validated gives the checklist:

• English-only output. Any non-English caller population takes you off the trained distribution entirely. Test code-switching and accented speech end-to-end, not just STT quality in isolation.
• Tool-schema fit. PhoneBench measures tool-call accuracy on its tool lists. Your tool names, parameter shapes, and error semantics differ. Wire the qwen3_coder parser to your schemas and run a say/do audit: log every confirmation utterance against the tool-call ledger and alert on claims without calls.
• The judged-eval circularity risk. PhoneLLM was trained by the PhoneBench authors on data disjoint from benchmark scenarios. Discipline was described, but I would still run my own scenario suite before believing transfer to my vertical — authentication flows and escalation rules are exactly where fine-tunes quietly overfit vendor conventions.
• Concurrency at your tail. The 88-agents-per-node and 2x-optimization figures come from Modal's tuned stack. If you serve on vanilla vLLM or non-Blackwell hardware, re-run the concurrency sweep against your own 600 ms target before sizing any fleet; the card tells you plainly that its numbers do not transfer.
• Interrupt and barge-in behavior. Everything published is turn-level. Real phone calls overlap, interrupt, and backtrack; the interaction with a no-thinking, low-latency model is exactly where I would expect untested seams.

Where This Fits in the Voice Stack

Option	Latency profile	Cost profile	Data/control	When it wins
PhoneLLM Alpha 1, self-hosted	Sub-second first token achievable; P95 depends on your serving work	~$0.00025/agent-min at published concurrency, on your hardware	Full — weights, logs, and traces stay in-house	Regulated verticals, high volume, teams able to run vLLM/SGLang
Frontier API (e.g., GPT 5.6 Terra class)	Reported P95 TTFT ~1,900 ms in fast mode — over budget alone	Per-token, reported 94% higher than PhoneLLM for these workloads	Vendor	Bursty low volume, broad general knowledge, no infra team
Mid-tier API models	Varies; watch thinking-token leakage (the card flags one Flash-class model for exactly this)	Lower, but thinking tokens inflate it	Vendor	Prototypes and non-latency-critical flows
Bigger self-hosted general model	Competent but needs the same serving investment for worse density	Poor model-size arbitrage	Full	Only when task breadth genuinely requires 70B+ capability

The strategic claim I buy from this release is not "PhoneLLM beats frontier models." It is the model-size arbitrage argument the card states outright: the path to cheaper self-hosting is a small model that is a perfect fit for the workload, plus serving engineering that moves the cost-latency curve. And the roadmap claim — training runs cheap enough to refresh weights monthly against production traces and targeted evals — describes a different lifecycle than the annual-frontier-model cadence most voice stacks are built around. If it holds, the durable skill in voice AI shifts from prompt crafting toward eval construction and serving tuning. PhoneBench and that fully-flagged vLLM recipe are, respectively, templates for both.

Resources

• Pipecat PhoneLLM Alpha 1 model card
• Pipecat PhoneLLM Alpha 1 NVFP4 model card