Audio8 ASR Infinite: What It Actually Takes to Listen Forever

The Unbounded Stream Problem That Streaming ASR Never Really Solved

Every streaming ASR system I've worked with is really a finite-stream system wearing a streaming costume. That's fine until somebody asks it to listen for eight hours straight, and then the costume falls off.

The failure is architectural, not a tuning problem. Take the transducer family — NeMo's FastConformer RNN-T/CTC line, sherpa-onnx's streaming Zipformer. These are genuinely frame-synchronous and causal. There's no generative text decoder accumulating key-value pairs, so memory is inherently O(1) and there is literally no position index to overflow. That's the good news, and it's also the limitation: without a heavy language-model decoder behind the acoustics, you get weaker semantic priors. Out-of-vocabulary phonetic errors, homophone confusion, and punctuation and casing that need a second model bolted on after the fact. For a voice agent, that's a real quality tax. For a 24/7 call-monitoring pipeline where you're reading the output and not just triaging intents, it's worse than that.

Then there's the Whisper-wrapper school — Whisper-Streaming, WhisperLive, various faster-whisper plus local-VAD rigs. These chop a batch model into sliding windows, run overlapping passes, and stitch tokens together by local agreement. It works beautifully in demos and deteriorates in production. When you cut context abruptly at a window boundary during silence, the decoder has nothing to condition on and it starts looping phrases or inventing them. The latency is also hostage to silence detection: mid-sentence pauses either split a segment you wanted whole, or push your wait times into the one-to-two-second range while the system decides.

So the two honest options have been: bounded memory with shallow language understanding, or rich language understanding with unbounded state and hallucination risk. Nobody had shipped a permissively-licensed open-weight model that took the third path — a causal audio tower feeding a generative decoder, with the decoder's state held to a fixed size so the stream can run for days without growing the cache or losing positional coherence.

The core tension is this: autoregressive quality comes from replayable text history, and replayable text history is exactly what makes memory grow. You want the model to remember the acronym someone defined forty minutes ago, and you want VRAM to be flat. Those two desires are in direct opposition unless you do something clever about both cache size and position encoding.

That's the premise Audio8-ASR-Infinite is built on — roughly 4B parameters, a causal audio encoder paired with a Qwen2.5-3B-Instruct decoder, and a fixed-size rolling KV cache with what Edge0 calls exact RoPE re-basing. The premise is sound. Whether it holds depends entirely on how well the mechanism survives millions of frames, and that's where I want to be specific rather than hand-wavy.

flowchart TD
    A[Continuous audio frames] --> B[Causal audio encoder]
    B --> C[Rolling KV cache, fixed capacity]
    C --> D{Token falls inside<br/>the window?}
    D -- yes --> E[Keep key/value pair<br/>position index unchanged]
    D -- no --> F[Evict oldest pair]
    F --> G[Shift sliding origin forward,<br/>re-base all remaining<br/>RoPE indices to new zero]
    G --> H[Relative distances<br/>between keys and queries<br/>preserved exactly]
    E --> I[Attention over window]
    H --> I
    I --> J[Decode next token]
    J --> K[Semantic turn detection head,<br/>predicted per decoded token]
    K -- completion predicted --> L[Emit final segment,<br/>reset utterance state]
    K -- continuation predicted --> M[Stay in same utterance]
    L --> A
    M --> A

Inside the Rolling KV Cache and Exact RoPE Re-Basing

Here's the mechanism I keep coming back to, because it's the whole ballgame.

A standard autoregressive speech decoder does the obvious thing: every generated token appends a key/value pair, and every new token's rotary position index is one larger than the last. Run for an hour and the cache is enormous. Run for a day and you're in genuinely unpleasant territory — position indices in the millions, memory climbing the whole way, and attention behavior degrading as the model is asked to reason over a context it was never trained to hold. This is the exact failure that makes "just run Whisper in a loop" a non-answer for continuous transcription.

Audio8 replaces that with a fixed-capacity cache. Once it's full, the oldest key/value pairs get evicted as new ones arrive. Capacity is constant, so VRAM is constant, and compute per step is constant. The stream can run for minutes or days and the resource profile doesn't move.

The hard part is what eviction does to positions. If you evict the oldest pairs but leave every surviving token's absolute position index untouched, you now have a window that starts at, say, position 400,000. The model is being asked to attend over a compressed slice of a much larger index space, and rotary embeddings don't take kindly to that — the relative distances encoded in the rotations no longer correspond to anything the model learned. Quality drifts.

"Exact" re-basing is the fix, and the word exact is doing real work. You shift the sliding origin forward and re-base every surviving key and query's position index to the new zero, so that the relative distances between any key and any query are preserved bit-for-bit. This is not interpolating a position encoding scheme, not scaling frequencies down, not approximating — it's a coordinate shift that leaves the attention geometry invariant. Applied at every eviction, across millions of frames, the map from "what the model sees" to "what the model learned" stays identical.

Why this is harder than it sounds: the failure is silent. At a cache boundary, if your re-basing isn't exact — if there's a rounding artifact in how you recompute the rotation for a token that's been re-based many times over — you don't get an exception. You get a subtle attention bias that accumulates. In FP16, and especially after INT4 quantization, you're shifting indices continuously for days and every frame that crosses a boundary is another chance for a small numerical error to enter and never leave. The research notes non-causal leakage across chunk boundaries as a related hazard. My read is that attention entropy degradation over long sessions is the failure mode you'd actually observe, and it would look like a model that's fine for the first hour and slightly worse for the fourth. That's a brutal thing to catch in a staging environment.

The concrete numbers matter here for deployment: ~4B parameters total, which lands at roughly 9 GB VRAM in FP16 and about 2.2 GB after INT4 quantization. Those are the two configurations I'd plan around — 9 GB is a single modest GPU per stream, 2.2 GB opens up edge and multi-stream-per-card topologies that a transducer-free LLM decoder normally wouldn't allow.

The maintenance risk is real and I want to name it plainly. This lives in a patched fork of vLLM, distributed through custom Docker Compose setups. You cannot drop it into stock upstream vLLM or a standard TGI/Triton deployment — you'd be porting continuous audio ingestion, a custom chunking scheduler, and the rolling-cache modifications yourself. Every upstream security patch, CUDA kernel improvement, or engine release becomes a manual merge and re-verification exercise. On one hand that buys you checkpoint reuse; on the other hand, I'd isolate that container as its own microservice behind a gRPC or WebSocket contract so engine-level churn never leaks into the services consuming the transcript.

Semantic VAD: What the Decoder Predicts That Energy VAD Can't

The differentiator I'd lead with if I were demoing this to a skeptical team is the turn detection, because it's the one piece that changes what the product can be.

Traditional endpointing is acoustic. Silero and friends watch frame-level energy and cut when the waveform goes quiet for long enough. That's a reasonable proxy for "the speaker stopped," and it's wrong in a very specific way: silence correlates with turn boundaries, but it isn't the same thing as one. Humans pause mid-thought. We stutter. We say "so, the thing is—" and then go quiet for a full second while we assemble the rest of the sentence. Energy VAD hears silence and cuts, and you get a transcript fragment that ends mid-clause and a downstream agent that thinks it's the user's turn.

Audio8 delegates this to the decoder. The Qwen2.5 decoder emits a turn-completion prediction at the token level, drawn from the same language stream that's producing the transcript. So the decision is conditioned on what was actually said — on grammar, on whether the utterance is semantically complete — rather than on whether the room got quiet. A subordinate clause that trails off into a thinking pause reads to the decoder as an incomplete thought, and it waits. That's the whole argument, and it's a good one.

It also has failure modes I'd design around explicitly. A grammatically complete subordinate clause followed by a long pause is the nasty case. Picture: "If we look at the quarterly report…" then a genuine three-second gap before "…we can see the miss." The main clause reads as complete, the acoustic gap is long, and the decoder can commit a final segment prematurely. You get an utterance boundary in the middle of what was one sentence. Rapid conversational overlap — interruptions, crosstalk, two people talking at once — biases the other direction: the model can hallucinate boundary markers or stall on a commitment it should have made, producing delayed finals that make a live agent feel laggy.

The audio clock is where you tune this. The model consumes native chunks at selectable step intervals — 80 ms, 120 ms, or 160 ms. At 80 ms you're in sub-300 ms interactive territory, which is what a voice agent needs; the research frames 240–560 ms of transcription delay across the clock range. At 160 ms you're processing more audio per step, which packs more streams per GPU and suits batch broadcast monitoring or call-center ingestion where nobody's waiting on a spoken reply. I'd benchmark both against real traffic rather than trusting either number, and I'd pick per use case, not globally.

One thing semantic VAD does not do: gate silence. Energy VAD earns its keep by telling you there's nothing to transcribe. A decoder-level turn head will happily keep working through prolonged background noise because it's reasoning about language, not acoustics — which is wasted GPU cycles and, at the margins, hallucination surface. Front-end AEC and noise suppression upstream isn't optional here; it's the thing keeping the expensive decoder pointed at actual speech. This is the same lesson that shows up in any ambient capture pipeline: the preprocessing layer is where the product lives or dies, and the model is the easy part.

stateDiagram-v2
    [*] --> SingleSpeakerActive
    SingleSpeakerActive --> PauseCandidate: acoustic pause detected
    PauseCandidate --> SemanticDecision: decoder evaluates token-level completion
    SemanticDecision --> CommitAndReset: completion predicted
    SemanticDecision --> ContinueUtterance: continuation predicted
    CommitAndReset --> SingleSpeakerActive: final segment emitted, new utterance
    ContinueUtterance --> SingleSpeakerActive: same utterance, clock advances
    SingleSpeakerActive --> OverlapCrosstalk: second speaker detected
    OverlapCrosstalk --> RunOnDetection: boundary commitment delayed
    RunOnDetection --> CommitAndReset: eventual boundary
    RunOnDetection --> FalseTurnTrigger: premature commitment under crosstalk
    PauseCandidate --> FalseTurnTrigger: complete subordinate clause plus long gap
    FalseTurnTrigger --> PrematureBreakdown: utterance split mid-sentence
    PrematureBreakdown --> SingleSpeakerActive: downstream correction required
    ContinueUtterance --> ComputationStall: commitment delayed too long
    ComputationStall --> SingleSpeakerActive: delayed final emitted

Dimension	Audio8 semantic VAD (token-level)	External energy VAD (Silero)	Whisper-Streaming local agreement
Decision basis	Decoder predicts turn completion from the language stream	Frame-level acoustic energy and silence duration	Token-prefix agreement across overlapping re-decodes
Thinking pauses / stutters	Handled — incomplete grammar holds the turn open	Poor — silence triggers a cut mid-utterance	Partial — depends on window and agreement threshold
Complete subordinate clause + long gap	Vulnerable to premature final segment	Cuts reliably, often at the wrong place	Cuts on window boundary, no grammar awareness
Overlap / crosstalk	Delayed commitment, risk of hallucinated boundary tokens	Cuts on whichever speaker hits silence first	Overlapping re-decodes produce unstable prefixes
Silence and noise gating	Weak — decoder keeps working on non-speech	Strong — this is what it's built for	Relies entirely on external VAD
Latency character	Deterministic within the audio clock (80/120/160 ms)	Latency spikes tied to silence thresholds, often 1000–2000 ms	Jittery; bounded below by the overlap window
Upstream dependencies	Needs front-end AEC/NS to avoid wasted compute	None beyond the VAD model itself	Requires VAD plus a stitching/agreement layer

Three Architectural Paradigms: Where Audio8 Actually Fits

The streaming ASR world has settled into three architectures, and they fail in different places. Picking one is picking which failure you can live with.

The transducer family — NeMo's FastConformer RNN-T/CTC, sherpa-onnx's streaming Zipformer, the SenseVoice-Small line out of Alibaba's FunASR — does frame-synchronous causal processing over a fixed lookahead. No generative decoder accumulates history, so memory is inherently O(1) and there is no position index that can overflow. These models are genuinely tiny by modern standards, most under 300M parameters, which means you can pack a lot of concurrent streams onto one GPU. The cost is semantic depth. Without a language-model decoder behind the acoustics, you get weaker handling of out-of-vocabulary names, homophone confusion where context would disambiguate, and punctuation and casing that require a second model bolted on afterward. For keyword spotting or intent triage, that's fine. For a product where a human reads the transcript, it's a quality tax you pay on every line.

The wrapper school takes an offline batch model — almost always Whisper — and slides a window over it. Chunks of a few seconds get re-decoded with overlap, and prefix tokens are matched by local agreement to stitch a transcript. It works in demos and degrades in production, for structural reasons. The 30-second context window forces arbitrary cuts, and cutting context during silence is exactly when a generative decoder with nothing to condition on starts looping phrases or inventing them. Latency is hostage to silence detection: mid-sentence pauses either split a segment you wanted whole or spike your wait times into the 1000–2000ms range while the system decides.

Audio8 takes the third path, alongside proprietary decoder-centric services like AssemblyAI's streaming models and Deepgram's offerings. A causal audio tower feeds a generative decoder — here a Voxtral Realtime 4B audio tower into a Qwen2.5-3B-Instruct decoder — and the decoded token stream doubles as the turn-detection stream. The repositioning is the point. You are not getting a strictly better model; you are trading compute footprint and language breadth for deep semantic context and deterministic latency. Roughly 4B parameters against sub-300M transducers is an order of magnitude more GPU per stream, and the bilingual English/Chinese tuning is narrow next to Whisper's 90-plus languages or NeMo's multilingual checkpoints.

What you buy with that compute is bounded-but-deep state: a fixed-size rolling cache that never grows, exact RoPE re-basing that keeps attention geometry stable across millions of frames, and turn decisions made on grammar rather than acoustics. Whether that trade is correct depends entirely on the application. A 24/7 monitoring pipeline where a human or a downstream LLM reads the output wants the semantic layer. A high-concurrency keyword spotter does not, and shouldn't pay for it.

Dimension	Non-LLM transducers (NeMo FastConformer, sherpa Zipformer, SenseVoice)	Sliding-window wrappers (Whisper-Streaming, WhisperLive)	Decoder-centric streaming (Audio8-ASR-Infinite, AssemblyAI)
Memory behavior	Inherently O(1); no KV cache to manage	Bounded by the 30s window; state resets at each cut	Fixed-size rolling KV cache; constant VRAM regardless of session length
Latency character	Frame-synchronous, stable, low	1000–2000ms spikes tied to silence thresholds; jittery	Deterministic within the audio clock; 240–560ms across 80/120/160ms steps
Semantic reasoning	Shallow — no generative language decoder	Strong within a window, lost at every boundary	Deep — full LLM decoder context, bounded by cache capacity
Compute footprint	Sub-300M params; high streams per GPU	Batch encoder-decoder; expensive per pass	~4B params; ~9 GB FP16, ~2.2 GB INT4 per stream
Language coverage	Multilingual variants available per model	90+ languages (Whisper)	English and Chinese, mid-sentence code-switching included

Failure Modes That Only Appear on Day Three of a Session

The rolling cache is what makes this deployable, and it's also the source of the failures nobody sees in a five-minute test.

Eviction is permanent forgetting. Any context older than the cache window is gone, and in a 24/7 session that means the terminology established early is not available to disambiguate later. Someone defines an acronym at hour one; at hour six the model has no idea what it stands for and transcribes it phonetically. A speaker's name introduced in the first ten minutes stops being recoverable, so homophones that should resolve by context don't — their versus there, a product name versus a common noun, jargon that only makes sense given an antecedent now evicted. This is not a bug to be patched. It is the direct consequence of trading unbounded context for a flat VRAM curve, and it means long-session consistency has to come from somewhere outside the model.

Position re-basing is the second structural exposure. Exact roping re-basing is exact in the algebra, but you are executing it in FP16, and under INT4 you're executing it on quantized weights, continuously, for days, across every cache boundary the stream crosses. Each boundary is a chance for a small numerical error to enter the rotation and never leave. The visible symptom is not an exception — it's a slow attention entropy drift that would present as a model slightly worse in hour four than hour one. I've learned to distrust systems whose only long-horizon failure is silent.

INT4 is where the quality floor sits, and it degrades unevenly. The categories that suffer most are accented speech, code-switching — Chinese and English traded mid-sentence, which is precisely the bilingual use case this model targets — and alphanumeric strings: serial numbers, license plates, order codes. Those are exactly the tokens a business pipeline cannot afford to get wrong, which is an uncomfortable place to save 7 GB of VRAM. I'd treat INT4 as a capacity mode, not a default, and validate it on your specific accents and entity types before trusting it.

Finally, the fixed audio clock is rigid by construction. At 80 ms, fast speakers and acoustic transients that straddle a frame boundary can lose edge phonemes before the audio tower ever sees them. It's a small effect per boundary and a compounding one across hours of dense speech. Between evicted context, quantized re-basing, and rigid frame steps, none of these are theoretical risks — they're guaranteed by the architecture, and the only real question is whether your use case notices.

Serving It in Production: Fork Lock-In, Stateful Workers, and Recovery

Two things about this deployment will bite you before any acoustic question does: the engine is a fork, and the worker is stateful.

Start with the fork. Audio8-ASR-Infinite runs on a patched vLLM streaming engine, distributed through custom Docker Compose setups. You cannot drop the weights into stock upstream vLLM or a standard TGI/Triton deployment and expect continuous audio ingestion, the custom chunking scheduler, and the rolling-cache modifications to be there. Someone has to port them. That means every upstream security patch, every CUDA kernel improvement, every vLLM release becomes a manual merge-and-re-verify exercise against a diff you own forever. The mitigation is architectural, not procedural: wrap the engine as an isolated microservice behind a gRPC or WebSocket contract, so the fork's internals never leak into your consumer services and an engine upgrade is a container swap rather than a migration. Treat the Docker Compose stack as an appliance with a version number, and pin it.

The statefulness is the harder problem. The rolling KV cache and the audio stream live on the GPU worker. You cannot round-robin a stateful stream across pods, which kills your standard stateless load balancer. You need session affinity — consistent hashing on a session ID, sticky routing, whatever your gateway supports — and you need to accept that worker restarts, CUDA OOMs, and node evictions terminate a session's entire history. The recovery protocol is the fix: on reconnection, replay the last several seconds of audio into a fresh worker to warm the rolling cache before resuming live. Five seconds is a reasonable starting point. Without it, every reconnect is a cold start with blank context.

If you're building a 24/7 call-center transcription service, this is the shape of the whole system: clients push 80 ms audio chunks over gRPC, the server holds rolling state per session, recovery replays the tail on reconnect, and a downstream fuzzy entity correction layer fixes the client-specific product names and acronyms that KV eviction has forgotten. Pair it with hotword biasing — a dynamic vocabulary or shallow-fusion layer — because the rolling cache will evict the entity defined at hour one by hour six, and no amount of re-basing brings it back. Same story for a real-time voice agent on an 80 ms clock: latency stacks (chunk wait plus decode plus semantic turn commitment), so measure end-to-end under load before you promise sub-500ms, and give the agent pod the same recovery replay so a restart doesn't cost you the conversation.

For an on-prem legal or medical review deployment, the calculus shifts. Air-gap it, run INT4 to fit the hardware, use 160 ms clocks for throughput, and put RNNoise or WebRTC AEC upstream — semantic VAD doesn't gate silence, so the expensive decoder will happily grind through background noise. And watch code-switching under INT4: it degrades first, and it's exactly what a bilingual model gets asked to do.

What I'd Reach For It For, and What I Wouldn't

I'd reach for Audio8-ASR-Infinite when the session is long, interactive, and semantic context is load-bearing. Call-center ingestion where an agent or a downstream LLM reads the transcript and acts on it. Voice agents that need turn detection on grammar rather than silence, because the mid-thought pause is the thing that breaks the interaction. Meeting transcription where domain terminology has to survive hour three and the speakers are bilingual. In those settings, the combination of bounded state and semantic turn detection is genuinely new in open-weight territory, and the Apache 2.0 license plus INT4 at ~2.2 GB means you can run it on hardware you control rather than renting it from Deepgram or AssemblyAI per audio hour.

I wouldn't reach for it in three cases.

Batch offline transcription of a large corpus does not need any of this. No stream, no turn detection, no rolling cache — just run Whisper, get 90-plus languages, and pay attention to your throughput rather than your statefulness. You'd be paying a 4B compute tax for capabilities your workload never exercises.

High-concurrency, low-cost streaming is the second no. If you're doing keyword spotting or intent triage across thousands of simultaneous calls, a sub-300M transducer like NeMo FastConformer or sherpa's Zipformer gives you O(1) memory, stable latency, and an order of magnitude more streams per GPU. You lose semantic depth; you don't need it for triage.

And if you need language breadth beyond English and Chinese, this isn't the model. The bilingual tuning is narrow by design, and INT4 makes code-switching the first thing to degrade. Whisper or a multilingual NeMo checkpoint is the honest answer.

My overall read: the 4B footprint is a real cost, and the fork lock-in is a real operational liability. But the specific combination — a fixed-size cache that survives days of streaming with position geometry intact, and turn detection decided by the language model rather than by acoustic energy — is not something I've seen elsewhere under a permissive license. If your product lives or dies on whether the system knows the user finished their thought, that trade is worth it. If it doesn't, you're buying an expensive fix for a problem you don't have.

Resources

• GitHub - Edge0-AI/Audio8-ASR-Infinite · GitHub
• Edge0/Audio8-ASR-Infinite · Hugging Face