Beyond Silence Timers: Dissecting TYGER and the Mechanics of Full-Duplex Voice Turn-Taking

The Full-Duplex Dilemma: Why Cascaded Voice Stacks Choke on Conversational Latency

Human conversation does not run on an alternating half-duplex walkie-talkie protocol. In everyday human dialogue, the average gap between conversational turns hovers around 200 milliseconds. We achieve this tight coordination because listeners do not wait for complete silence to plan a response; we process syntactic trajectories, pitch declination, and breath intake in real time to anticipate exactly when the other speaker will yield the floor.

Traditional cascaded voice stacks—chaining an energy-based Voice Activity Detection (VAD) engine like Silero, a streaming speech recognizer, an LLM, and a streaming TTS provider—shatter this rhythm. Because energy-based VAD algorithms only detect the presence or absence of raw sound amplitude, they have no concept of prosodic resolution. If you pause for 250 milliseconds to search for a word, a fast VAD triggers an interruption or an accidental turn yield. To prevent the agent from cutting users off mid-thought, engineers routinely configure silence fallback timers between 800 and 1,200 milliseconds. By the time that silence window elapses, the partial transcript finishes streaming to the LLM, the first token generates, and the TTS engine buffers its initial chunk, the total conversational latency balloons past 1.5 seconds. The interaction feels sluggish, mechanical, and brittle.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                           Voice Agent Paradigms                                 │
├─────────────────────────┬─────────────────────────┬─────────────────────────────┤
│  1. Cascaded + Rule/VAD │  2. Dedicated Decision  │  3. Native End-to-End S2S   │
│     (The Status Quo)    │      (TYGER's Lane)     │     (Full Monolithic)       │
│                         │                         │                             │
│ Mic → VAD → ASR →       │ Mic + Ref Audio         │ Dual Audio In               │
│ LLM → TTS               │         │               │         │                   │
│ (Silero, WebRTC VAD,    │ [TYGER Decision Model]  │ [Moshi, GPT-4o Realtime,    │
│  LiveKit heuristics)    │         │               │  PersonaPlex / Gemini Live] │
│                         │ Events to Cascaded Stack│         │                   │
│                         │ (Yield/Barge-in/Chime)  │ Direct Audio Out            │
└─────────────────────────┴─────────────────────────┴─────────────────────────────┘

The industry's knee-jerk reaction has been to pivot toward monolithic speech-to-speech models like Kyutai Moshi or proprietary realtime audio endpoints. These end-to-end architectures fold turn prediction directly into neural audio tokens, achieving sub-300ms responsiveness and natural conversational overlap. But that responsiveness comes at a massive architectural cost: you surrender the entire modular stack. In a monolithic audio model, deterministic business logic, dynamic schema validation, multi-step tool execution, and voice-cloning fine-tunes become black-box nightmares.

This is the exact wedge that TYGER targets. By isolating conversation control into a dedicated, open-weight acoustic arbiter that runs alongside your pipeline, you decouple turn arbitration from text reasoning. The LLM remains responsible for knowledge, tool orchestration, and content generation, while a specialized temporal model governs the conversational floor. You gain full-duplex conversational mechanics—instantaneous barge-in handling, prosodic endpointing, and passive listener acknowledgments—without abandoning the reliability and observability of modular infrastructure.

Inside the 80ms Loop: How Nemotron Representations Feed TYGER's Causal GRU

At the architectural core of TYGER lies a continuous, multi-channel processing loop evaluated every 80 milliseconds. Rather than ingesting an isolated microphone feed, TYGER accepts two strictly synchronized mono 16 kHz audio streams: the user's microphone capture and a loopback reference of the agent's active TTS playback.

Feeding the agent's playback audio directly into the turn-taking model resolves one of the nastiest failure modes in conversational voice engineering: self-interruption caused by acoustic leakage. Even the best software Acoustic Echo Cancellation (AEC) routines struggle when a user's phone or laptop speaker reflects high-gain playback into the microphone capsule. By exposing the reference stream directly to the acoustic feature extractor, the model learns to correlate incoming audio against the agent's own speech patterns, discriminating between a user cutting in and the agent's own voice bouncing off an office wall.

Both 16 kHz streams are routed into an unmodified NVIDIA Nemotron 3.5 ASR Streaming 0.6B backbone (~638M parameters). Instead of running full text decoding to infer intent from partial transcripts, TYGER extracts hidden representations directly from Layer 15 (1,024 dimensions) without lookahead. The model operates on 320 ms streaming hops, re-encoding a sliding 8-second audio window per stream at each evaluation step.

User Mic (16 kHz Mono) ────┐
                           ▼
                  [Nemotron 3.5 ASR] ────► Layer 15 Embeddings ────┐
                     (Layer 0-15)          (1024-dim, no lookahead)│
                           ▲                                       ▼
Agent TTS Ref (16 kHz) ────┘                              [Causal 3-Layer GRU]
                                                            (Width 768, ~14M)
                                                                   │
                                                                   ▼
                                                          Discrete States &
                                                          Continuous Scores (80ms)

Tapping Layer 15 gives the downstream decision logic access to rich acoustic and prosodic contours—such as fundamental frequency slope, lengthening syllables, and terminal pitch drops—long before an acoustic model could crystallize those frames into text tokens. A flat or rising pitch contour at the end of an utterance indicates that the user is pausing to think; a falling pitch accompanied by a decay in vocal energy indicates completion. Layer 15 preserves these continuous suprasegmental features without requiring transcript-level parsing.

These 1,024-dimensional feature frames feed into TYGER's turn-taking module: a causal 3-layer Gated Recurrent Unit (GRU) with a hidden width of 768 (~14M parameters). Total model footprint reaches 652M parameters, which occupies roughly 2.61 GB of VRAM in FP32. The standout engineering win here is the execution state: because the GRU is causal and operates across the extracted frame sequence, the recurrent conversation state requires only ~9 KiB per session. Outside of the rolling audio buffers, the orchestrator does not need to persist massive key-value caches or heavy token contexts between turns. The GRU updates its internal state every 80 ms, continuously projecting whether the floor is clear, contested, or yielded.

The Mechanics of Conversational State: Thresholds, Fallbacks, and Chimes

TYGER translates its recurrent hidden state into two simultaneous representations every 80 ms: a discrete operational state (LISTEN, HOLD, SPEAK, BACKCHANNEL) and four continuous probability heads for hazard, barge-in, backchannel, and endpoint.

Critically, TYGER does not feed its own previous state predictions back into the GRU input. State conditioning relies strictly on external ground-truth state flags provided by the voice pipeline orchestrator: LISTEN while waiting for user input, and SPEAK while the TTS engine is actively rendering audio to the user. This design choice prevents self-reinforcing state hallucinations.

stateDiagram-v2
    [*] --> WARMUP: Session Init
    WARMUP --> LISTEN: 6 Warm-up Frames (~480ms)
    
    state "LISTEN (Agent Idle)" as LISTEN
    state "HOLD (User Mid-Speech)" as HOLD
    state "SPEAK (Agent Playing TTS)" as SPEAK
    state "BACKCHANNEL (Agent Interjecting)" as BACKCHANNEL

    LISTEN --> HOLD: User Speech Detected (>=350ms, >-50 dB)
    HOLD --> LISTEN: User Pauses (Energy Drops)
    
    HOLD --> SPEAK: Endpoint Score >= 0.91
    HOLD --> SPEAK: 1.2s Silence Fallback
    
    SPEAK --> LISTEN: Agent Utterance Finished
    SPEAK --> HOLD: Barge-in Score >= 0.50 (Purge TTS)
    
    HOLD --> BACKCHANNEL: Backchannel Score >= 0.40 (Cooldown Expired)
    BACKCHANNEL --> HOLD: 1.2s Backchannel Cooldown Passed

Each discrete state transition is governed by explicit numeric boundaries and hardware-level safety guards:

1. Endpointing (HOLD to SPEAK): The default endpoint score boundary is set to 0.91 (tightened to ~0.93 in research evaluation). When the user is speaking and their prosodic trajectory reaches a conclusive boundary, the endpoint head crosses 0.91, instructing the orchestrator to seal the user prompt and initiate LLM generation. To handle dead air when an interlocutor trails off without clear syntactic or acoustic closure, a hard safety fallback forces an endpoint trigger whenever silence exceeds 1.2 seconds, overriding model scores.
2. Barge-in (SPEAK to HOLD): While the agent is in the SPEAK state, user mic audio is monitored against a default barge-in threshold of 0.50 (tightened to ~0.80 during rigorous benchmarks). If incoming audio crosses 0.50, the system treats it as an intentional interruption: the orchestrator cuts the streaming LLM context, flushes downstream TTS buffers, and yields the floor immediately.
3. Backchanneling (HOLD to BACKCHANNEL): While listening to an extended user explanation, the model monitors the backchannel head against a threshold of 0.40. If the user hits a momentary pause or a narrative beat that invites acknowledgment, the model triggers an interjection. The orchestrator plays an affirmative acoustic token (such as "mm-hm" or "yeah") without stealing conversational initiative or breaking the user's turn. To prevent repetitive, manic chattering, backchannels are governed by a mandatory 1.2-second rate-limiting cooldown.
4. Acoustic Safety Gates: To protect against ambient transients, false barge-ins, and session startup instability, TYGER enforces two baseline filters. Every session mandates an initial 6-frame warm-up window (~480 ms) to stabilize GRU internal hidden representations before scores are honored. Furthermore, transitions out of silence require at least 350 ms of sustained speech above a -50 dB power threshold, filtering out room reverb, mechanical clicks, and shallow breathing.

Architectural Tradeoffs: Dedicated Acoustic Models vs. Semantic VAD and Monolithic S2S

When I evaluate conversational voice architectures, I see teams repeatedly forced into an uncomfortable binary: accept the sluggish turn-taking of cascaded pipelines, or surrender their entire software stack to monolithic speech-to-speech models.

Traditional energy VADs like Silero or WebRTC VAD require negligible compute—running comfortably on a fraction of a CPU thread—but they operate completely blind to intonation. Even modern semantic augmentations like Deepgram Flux, which analyze streaming text transcripts to identify clause boundaries, cannot solve the fundamental problem: human conversational coordination is acoustic before it is lexical. A user elongating a vowel on a rising pitch is holding the floor, regardless of whether their grammar looks like an endpoint. Conversely, monolithic speech-to-speech architectures like Kyutai Moshi or proprietary realtime audio systems internalize timing, prosody, and turn transitions directly within their neural weights. Yet adopting them requires surrendering deterministic business logic, structured tool execution, and voice-cloning pipelines to a black box.

TYGER stakes out a compelling third path: decoupling turn arbitration from reasoning and synthesis without collapsing them into an uninspectable monolith.

quadrantChart
    title Voice Turn-Taking Approaches
    x-axis Low Modularity and Control --> High Modularity and Control
    y-axis Low Acoustic Depth --> High Conversational Responsiveness
    quadrant-1 Responsive and Modular
    quadrant-2 Responsive but Monolithic
    quadrant-3 Sluggish and Rigid
    quadrant-4 Modular but Deaf to Prosody
    Silero and LiveKit: [0.85, 0.22]
    Deepgram Flux: [0.76, 0.52]
    TYGER: [0.84, 0.86]
    Kyutai Moshi and GPT-4o: [0.18, 0.92]

By tapping Layer 15 of Nemotron 3.5 ASR Streaming alongside an aligned playback reference stream, TYGER grants a modular stack native prosodic awareness and genuine duplex capability. I can retain my choice of reasoning models, tool-orchestration pipelines, and streaming TTS engines while shedding the clumsy 1,000 ms silence timeouts that plague conventional agents.

Dimension	Traditional Heuristic VAD (e.g., Silero)	Semantic Streaming ASR (e.g., Deepgram Flux)	TYGER Acoustic Decision Model	Native End-to-End S2S (e.g., Kyutai Moshi)
Input Audio Channels	Mono microphone capture	Transcribed text tokens + timing	Dual aligned streams (Mic + TTS playback reference)	Multichannel raw audio tokens
Prosodic Awareness	None (pure energy and volume thresholds)	Moderate (text syntax + token cadence)	High (continuous Nemotron Layer 15 embeddings)	Maximal (native end-to-end learned prosody)
Echo Handling	Hard dependency on external software AEC	Hard dependency on external software AEC	Native reference stream acts as echo discriminator	Internalized into joint representation
Backchanneling	Unsupported (triggers accidental yields)	Rudimentary and rule-based	Native continuous probability head evaluated every 80 ms	Emergent natural dialogue behavior
GPU Footprint	Negligible (CPU-bound)	Cloud API or standard ASR GPU worker	652M parameters (~2.61 GB FP32 VRAM)	7B+ parameters (dedicated heavy GPU allocation)
Architectural Control	Complete modularity across components	High modularity across API components	Complete modularity (retains external LLM and TTS)	Minimal (monolithic neural black box)

That operational freedom comes at a steep price. Deploying a 652M parameter decision model that re-encodes an 8-second window across two streams every 320 ms demands serious GPU resources. In high-concurrency telephone gateways where Silero handles thousands of streams per box, running TYGER on every connection would explode infrastructure costs. Furthermore, because its representations derive strictly from Nemotron 3.5 ASR, TYGER is rigidly locked to English. For multi-tenant or multilingual deployments, those liabilities cannot be hand-waved away.

Where TYGER Breaks: Acoustic Drift, False Barge-Ins, and the 1.2-Second Trap

In production, TYGER's dual-stream architecture introduces failure modes that do not exist in conventional half-duplex pipelines. If you deploy it expecting a drop-in replacement for a standard VAD, you will quickly hit three operational traps.

The most catastrophic failure mode is acoustic clock drift between the client microphone capture and the server-side playback reference buffer. TYGER relies on the playback audio stream to identify self-echo: when the agent speaks through a user's laptop speakers, that audio bounces into the microphone capsule, and the model uses the aligned reference to recognize it as self-talk. If WebRTC packet jitter or an un-synchronized browser audio playback queue causes the reference stream to drift even a few dozen milliseconds out of alignment, that echo discriminator collapses. The model misinterprets its own delayed voice as an incoming user interruption, crosses the barge-in threshold, and terminates speech generation mid-word.

sequenceDiagram
    autonumber
    participant C as Client (Browser / Device)
    participant O as Voice Orchestrator
    participant T as TYGER (Nemotron + GRU)
    participant L as LLM / TTS Pipeline

    Note over C,O: Dual Audio Ingestion & Clock Sync
    C->>O: Send Mic Audio Frames (16 kHz mono)
    O->>C: Stream Agent TTS Audio
    C->>O: Ack Playback Queue Hardware Timestamp
    O->>T: Feed Mic Frames + Aligned Reference Audio
    
    Note over T: 80ms Continuous Evaluation
    T->>O: State: SPEAK (Hazard: 0.12, Barge-in: 0.82)
    
    Note over O,L: Atomic Interruption Handling
    rect rgb(240, 220, 220)
    O->>L: Cancel Streaming Context & Purge Inference Queue
    O->>C: Flush Client Playback Buffer Immediately
    O->>T: Update Ground-Truth State to HOLD
    end
    
    C->>O: Stream Fresh User Speech Frames

The second failure mode stems from TYGER's default barge-in threshold of 0.50. While the researchers tightened this to ~0.80 during evaluation benchmarks, the permissive default value combined with the baseline audio guard—requiring only 350 ms of sound above -50 dB—leaves the system vulnerable to non-speech acoustic transients. A sudden throat clear, heavy laughter, nearby office banter, or a dog barking readily exceeds that threshold, triggering false barge-in events that abruptly silence active answers. In any practical deployment, raising that threshold closer to 0.75 or 0.80 is non-negotiable.

Finally, there is the 1.2-second silence trap. To prevent dead air when an interlocutor stops talking without a conclusive prosodic drop, TYGER enforces a hard fallback: after 1.2 seconds of silence, an endpoint triggers unconditionally regardless of model scores. When a user pauses deliberately to formulate a complex thought or retrieve a number, that hard boundary cuts them off mid-sentence. Coupling TYGER with domain-aware conversational context—such as temporarily relaxing the silence timeout when an agent has just asked for a structured identifier—is essential to prevent frustrating user cutoffs.

Production Hardening: Calibration, Synchronization, and GPU Tiering

Transitioning TYGER from a local prototype to production requires treating acoustic timing and GPU infrastructure as tightly coupled systems. If you deploy the reference weights without operational safeguards, clock discrepancies and default boundaries will degrade conversational quality within minutes.

First, you must enforce strict hardware-level acoustic synchronization over your transport layer. Because TYGER correlates incoming microphone frames against the agent's playback audio to filter out self-talk, any latency jitter between audio packet ingestion and speaker emission breaks the model's acoustic reference. In a WebRTC architecture, implement RTCP Sender Reports to continuously correlate the client's actual speaker playback timestamps against microphone capture frames. If network turbulence or client-side queue delays cause the microphone stream to drift past 5 seconds relative to the playback reference buffer or wall-clock time, terminate the session immediately. Recovering a corrupted GRU recurrent state mid-conversation is impossible, and failing fast is far better than letting the agent enter an uncontrolled self-interruption loop.

Second, you must recalibrate the decision heads for production acoustic conditions. The repository's out-of-the-box defaults—a 0.50 threshold for barge-in and 0.40 for backchannels—are research baselines that prove far too eager in real-world environments. For production deployments, I push the barge-in boundary to 0.75 or 0.80 and elevate the backchannel threshold to 0.60 or higher. Furthermore, backchanneling must be contextually gated: pair the hardcoded 1.2-second cooldown with an orchestrator-level guard that disables backchannel evaluation entirely whenever the user is reciting alphanumeric strings, dates, or structured account details. An ill-timed affirmative chime in the middle of a credit card dictation derails the entire intake.

Finally, decouple your deployment topology into dedicated service tiers. TYGER couples its lightweight 14M GRU with a 638M parameter Nemotron 3.5 feature extractor, processing 320 ms streaming hops across sliding 8-second audio windows for two independent audio feeds. Running four concurrent full-duplex sessions alongside an LLM and streaming neural synthesis easily swallows 28 GB of VRAM. Packing these heterogeneous tasks onto a single machine causes severe GPU scheduling contention, directly threatening TYGER's strict 80 ms latency deadline. The operational fix is architectural separation: isolate the model into dedicated, high-frequency "ears" inference workers optimized for continuous tensor throughput. These nodes evaluate the dual audio streams and broadcast state transitions over an internal event bus to independent "mouth" workers handling reasoning and audio synthesis.

Hands-On Implementation: Three Full-Duplex Architectures to Build with TYGER

When architecting production systems around TYGER, the primary design challenge is binding its continuous 80 ms decision stream to your business logic and synthesis pipeline without introducing race conditions. Depending on your operational environment and latency budget, I see three distinct patterns worth implementing.

For enterprise intake, you can construct a low-latency customer support voice agent designed for immediate, natural interruption handling and active listener acknowledgments during unstructured problem descriptions. Here, a WebRTC gateway feeds dual 16 kHz streams directly into TYGER running on a dedicated inference worker. The model's continuous decision stream wires into a lightweight event bus that orchestrates an LLM like Claude or Gemma alongside a fast streaming synthesis engine like Cartesia or ElevenLabs. The core integration requirement is sub-millisecond cancellation: the instant a barge-in event fires, the orchestrator purges the downstream inference queue and dispatches an atomic flush instruction to the client playback buffer. To prevent office background chatter from aborting active answers, calibrate the barge-in threshold to 0.80 and endpointing to 0.93, while explicitly disabling backchannel evaluation during alphanumeric or form-field dictation.

For education and training, you can build an interactive interview and language practice coach that tolerates thoughtful pauses without premature interruptions. Standard voice activity timers routinely penalize candidates who pause mid-sentence to structure an answer. By pairing TYGER's continuous endpointing score stream with an intermediate client-side prosody buffer and an ASR partial transcript validator, you can construct an adaptive floor arbiter. The primary operational hurdle is TYGER's hardcoded 1.2-second silence fallback, which will prematurely trigger an agent turn during deep contemplation. You mitigate this by wrapping the endpoint trigger in a dynamic grace-period coordinator: if the partial transcript reveals an incomplete syntactic clause and the trailing acoustic pitch contour remains flat or rising, the coordinator holds the floor open and delays the LLM turn transition, granting the speaker room to think.

Finally, for edge deployments, you can assemble a local-first desktop voice assistant that runs entirely on a single workstation. This architecture leverages the reference implementation components found in tyger-s2s (app/plugins/ears/tyger/), pairing the turn-taking model with local VoxCPM2 streaming speech generation and an open-weight 8B reasoning model on an RTX workstation with at least 28 GB of allocated VRAM across four streaming sessions. The critical failure mode here is audio clock desynchronization between the operating system's loopback audio capture and the microphone input hardware. If the operating system's loopback buffer lags behind physical microphone capture, the acoustic reference track desynchronizes, causing the model to mistake its own speaker output for user speech and hallucinate self-interruptions. Enforcing an aligned circular ring buffer between the capture device and the loopback monitor is mandatory to preserve a clean conversational floor.

Resources

• nineninesix/tyger · Hugging Face
• GitHub - nineninesix-ai/tyger-s2s · GitHub
• 301 Moved Permanently