Living dossier

Audio, TTS, Voice

Back to topics•Blog
Mehran Mozaffari·
9 resources0 related posts

The Voice Stack Finally Has a Permissive Open Layer

Voice AI in production decomposes into three problems that are usually sold as one: text-to-speech (making a machine talk), automatic speech recognition (making a machine listen), and real-time voice agents (doing both inside a human conversational latency budget of roughly a second end to end). The defining change of the last two years is that the first two problems acquired genuinely permissive open solutions — not research-only weights, not watered-down demos, but Apache/MIT/CC-BY artefacts with published benchmarks that compete with commercial APIs on quality. Kokoro-82M is the emblem: an 82-million-parameter TTS model, Apache-2.0 weights, trained for about $1,000 of A100 time, whose author documents hosted-API rates under $1 per million input characters and 12 million downloads. That is not a research curiosity; that is a commodity price for a commodity task, published by a solo maintainer.

What gets miscategorized into this field matters for scoping decisions. Music generation and sound effects are adjacent generative-audio problems with different evaluation criteria and are excluded here. Full-duplex speech-to-speech models — where a single model maps audio in to audio out without an intermediate transcript — are the frontier the commercial labs are converging on, but almost everything deployable today in the open is still the modular pipeline: ASR, text reasoning, TTS, each swappable. That modularity is a feature for cost engineering (each stage can be chosen on its own price/quality/latency curve) and a liability for latency (three network hops instead of one), and most of the practical decisions in this dossier reduce to managing that trade.

The second structural fact is the licence map, which is no longer an afterthought. The permissive tier is real: Whisper and faster-whisper (MIT), Chatterbox (MIT), F5-TTS code (MIT), Kokoro (Apache-2.0), Piper (MIT), NVIDIA's Parakeet ASR (CC-BY-4.0). The cautionary tier is real too: Coqui's TTS toolkit — the most-starred Python TTS library ever, at 46k stars — is MPL-2.0 code, but its flagship XTTS model weights ship under Coqui's own public model licence with non-commercial restrictions, and the company behind it went under. The lesson the field learned from Coqui's collapse is now baked into how the newer projects present themselves: Kokoro's model card leads with the licence and the training-data provenance; Chatterbox ships with a built-in neural watermark. Licence due diligence is not paperwork in this field; it is the difference between a model you can ship and a model you can only demo.

Open TTS Crossed the Quality Line, Then the Price Line

Three open projects define the current usable frontier, each with a distinct thesis. Kokoro-82M proves the small-model thesis: StyleTTS 2 architecture, decoder-only (no diffusion), IPA-phoneme G2P via its misaki library, trained exclusively on permissive and synthetic audio for a few hundred hours. Its model card publishes the full training bill — 1,000 A100-80GB hours at roughly $1/hour — which is itself a watershed: the cost of reproducing a useful TTS model is now a line item a serious hobbyist can fund. English-first, multiple voice packs, and fast enough for CPU-adjacent deployments. My position: for English narration, IVR, and accessibility use cases, Kokoro is the default open choice, and its permissive licence removes the only reason to even evaluate proprietary alternatives for those workloads.

Chatterbox (Resemble AI, MIT, 26k stars) proves the family thesis: one architecture spanning Turbo (350M, English, low-latency, native paralinguistic tags like [laugh] and [cough]), Nano (110M, same architecture, runs 3× faster than realtime on 8 CPU cores for on-device use), Multilingual V3 (500M, 23+ languages, reduced hallucination), and single-language fine-tunes for six priority languages. Zero-shot voice cloning from a 10-second reference clip is standard across the family. The tips section is refreshingly operational — accent bleed from a mismatched reference clip is a known failure with a documented workaround (cfg_weight to 0). My position: Chatterbox is the right open choice when you need voice cloning with a licence that allows shipping, and Nano is the most credible on-device TTS option in the field.

F5-TTS (15k stars, MIT code) proves the research-transfer thesis: a flow-matching diffusion transformer whose value shows in its published serving benchmark — on a single L20 GPU with Triton/TensorRT-LLM, 253ms average latency at concurrency 2, RTF 0.0394 (roughly 25× faster than realtime), against 0.1467 for plain PyTorch. That benchmark table is the honest picture of open TTS deployment: the model is free, the engineering to serve it at conversational latency is the work, and the gap between a naive PyTorch deployment and an optimized runtime is nearly 4× on RTF.

The Coqui story is the counterpoint that keeps all three honest. XTTS v2 was the field's first genuinely usable open voice-cloning model (13 languages, then 16, sub-200ms streaming claims), and the toolkit remains the most complete training platform in open TTS. But the weights' licence restricts commercial use, the company is gone, and the repo is effectively archived. Every newer project in this list is, in part, an answer to what Coqui proved was possible and failed to sustain.

The Clone Question: Watermarking Is Now Part of the Model

Voice cloning stopped being a technical frontier and became a governance problem, and the field's response is visible in the model cards. Chatterbox embeds Resemble's PerTh perceptual watermark in every generated file — an imperceptible neural watermark that survives MP3 compression and audio editing, with detection exposed as a few lines of Python returning 0.0 or 1.0. This is the first generation of open TTS where provenance tooling ships in the default pipeline rather than as an opt-in add-on.

Kokoro's card takes the complementary approach on the input side: it documents the training corpus's provenance (public-domain and permissively licensed audio, synthetic data from closed models, explicit CC-BY attributions for Koniwa and SIWIS) and even warns users off scam websites squatting on the model's name. Between Chatterbox's output watermarking and Kokoro's input provenance, the emerging norm is that a responsible open TTS release documents both where the voices came from and how to detect its outputs. My position: treat the absence of either as a disqualifying yellow flag when evaluating anything newer than this list, because consent disputes over cloned voices are the most likely regulatory shock this field faces, and provenance tooling is the defense.

ASR Is a Solved Commodity — With One Frontier Left: Long Audio at Speed

Speech recognition's open story is more settled than TTS's, and the numbers explain why. OpenAI's Whisper (MIT, 108k stars) remains the reference: six model sizes from 39M to 1.55B parameters, multilingual recognition and translation, and the turbo variant (809M parameters, 8× the speed of large with minimal accuracy loss) that made Whisper-class quality affordable on a single consumer GPU (6GB VRAM). faster-whisper (SYSTRAN, MIT) is the deployment layer: a CTranslate2 reimplementation whose benchmark table is one of the most useful documents in the field — on a 13-minute file with an RTX 3070 Ti, original Whisper takes 2m23s at 4.7GB; faster-whisper fp16 takes 1m03s; int8 takes 59s at 2.9GB; and batched inference (batch_size=8) collapses it to 16-17 seconds. Four-plus times faster with 8-bit quantization, and integrated Silero VAD to filter non-speech.

NVIDIA's Parakeet TDT 0.6B v2 (CC-BY-4.0) shows where the frontier moved: a 600M FastConformer-TDT model with full attention that transcribes audio segments up to 24 minutes in a single pass, with an RTFx of 3,380 on the Open ASR leaderboard — three orders of magnitude faster than realtime — plus word-level timestamps and automatic punctuation. Its published WERs (1.69 LibriSpeech clean, 9.74 GigaSpeech, 11.16 AMI meetings) make the case that English ASR at this size is effectively solved; a multilingual v3 covering 25 European languages is already posted. My position: for transcription volume, self-hosting is now strictly rational above trivial scale — Whisper/Parakeet-class models on one modest GPU out-transcribe any per-minute API price — and the remaining vendor value is in streaming latency and vertical features (diarization, compliance), not accuracy.

Piper belongs in this survey as the embedded-edge counterpoint: MIT-licensed, fast enough for Raspberry-Pi-class hardware, dozens of languages, and the default answer when the compute budget is a microcontroller. Its original repo now redirects development to a GPL successor (piper1-gpl under OHF-Voice), a small but instructive licence migration worth checking before you pin a version.

Real-Time Voice Is a Different Product, Not Just Faster TTS

ElevenLabs' pricing page is the clearest public document for how the commercial side of this field structures itself, and its structure is the tell: the company sells three separate products — Creative (TTS/STT/dubbing), Agents (conversational voice AI), and API — because real-time conversation has a different cost structure, latency budget, and reliability envelope than generating an audiobook. The credit economics: Free gets 10k credits/month; Starter $6/30k with commercial licence and instant voice cloning; Creator $22 (first month half price)/121k with professional voice cloning; Pro $99/600k; Scale $299/1.8M with 3 seats; Business $990/6M with low-latency TTS "as low as 5¢/minute" and 10 seats; Enterprise adds HIPAA BAAs, custom concurrency, and SSO. Effective per-minute cost falls from ~$0.36 at Free to ~$0.17 at Pro — a 2× volume curve — and concurrency limits (2 at Free rising to 10+ at Business) are the real gate for voice-agent deployments, since concurrent-request caps, not monthly credits, are what breaks under load.

The open alternative to the real-time product is the modular pipeline, and its latency math is now publishable: F5-TTS at 253ms server-side, Chatterbox-Turbo targeting zero-shot voice agents, faster-whisper transcribing in fractions of realtime, and an LLM in the middle. The pipeline can hit conversational budgets on good hardware, but every hop adds jitter, and the commercial full-duplex models' remaining advantage is exactly the hop count — they hear, think, and speak in one model pass. My position: for agent workloads under tens of concurrent calls, the open pipeline on one well-chosen GPU beats commercial per-minute pricing by an order of magnitude; past that, concurrency management and multi-region latency push you back toward managed platforms, and the honest comparison is total cost per concurrent conversation, not per minute of audio.

flowchart TD
    MIC[Microphone or telephony audio] --> VAD[VAD gate: Silero or equivalent]
    VAD -->|speech detected| ASR[ASR: faster-whisper or Parakeet<br/>streaming or batched]
    VAD -->|silence| MIC
    ASR --> TXT[Transcript with timestamps]
    TXT --> LLM[Text reasoning: your agent stack]
    LLM --> RESP[Response text]
    RESP --> TTS[TTS: Kokoro, Chatterbox-Turbo, or F5<br/>streaming synthesis]
    TTS --> PLAY[Audio playback to user]
    PLAY --> MIC

The second diagram is the same pipeline as a conversation, because the sequence — not the components — is where voice-agent latency is won or lost:

sequenceDiagram
    participant U as User
    participant P as Pipeline server
    participant W as ASR model
    participant L as LLM
    participant T as TTS model
    U->>P: speaks
    P->>W: stream audio chunks
    W-->>P: partial transcript
    P->>L: transcript when sentence boundary detected
    L-->>P: response text, streamed
    P->>T: stream sentence one for synthesis
    T-->>P: audio chunk one
    P-->>U: playback begins while sentence two is still generating
    Note over U,T: end-to-end budget is one second or less<br/>each hop must stream, never wait for the full utterance

The Options, Side by Side

Project Role Licence Scale / cost shape Latency story My position
Kokoro-82M Open TTS, English-first Apache-2.0 82M params; hosted APIs under $1/M chars; $1k to retrain Fast; CPU-adjacent Default for English narration and IVR
Chatterbox family Open TTS with cloning MIT Turbo 350M / Nano 110M / Multilingual V3 500M Nano 3× realtime on 8 CPU cores; Turbo for agents Right open choice for shippable voice cloning
F5-TTS Open TTS, flow matching MIT code Published Triton/TRT-LLM serving: 253ms, RTF 0.039 Best-documented optimized serving path Choose when you will run your own GPU fleet
Coqui TTS / XTTS Legacy open TTS toolkit MPL-2.0 code; XTTS weights non-commercial 46k stars, archived company XTTS streamed <200ms historically Study it; do not build a business on its weights
Piper Embedded TTS MIT (dev moved to piper1-gpl, GPL) Raspberry-Pi class Realtime on tiny hardware Edge and appliance deployments; check the new licence
openai/whisper Open ASR reference MIT (code + weights) 39M–1.55B params; turbo 809M at ~8× speed 30-second sliding window, offline bias The multilingual baseline; use via faster-whisper
faster-whisper ASR serving MIT int8: 2.9GB, 59s for 13min audio; batched: 16s Streaming-friendly; Silero VAD built in The default self-hosted transcription stack
Parakeet TDT 0.6B v2 Open ASR, long-form CC-BY-4.0 600M params; RTFx 3,380; 24-minute single pass Offline throughput king; v3 adds 25 languages Bulk transcription and subtitle pipelines
ElevenLabs Commercial voice platform Proprietary $0–$990+/mo tiers; ~$0.17–0.36/min effective; 5¢/min at Business Sub-200ms streaming; concurrency-gated Managed voice agents; pay for latency and SLAs, not quality

Choosing by Constraint: Licence, Latency, Compute

Licence first, always, in this field — it has the sharpest teeth. Shipping a product? Kokoro (Apache), Chatterbox/Whisper/faster-whisper (MIT), and Parakeet (CC-BY, attribution required) are all clean. Building on XTTS or any weights under a bespoke public-model licence means reading a non-commercial clause that probably kills your use case. And Piper's move to a GPL successor is a live reminder that an MIT label on a repo you found yesterday may describe a version that is no longer the one being developed.

Latency second, measured against your real budget. If "eventually correct" is acceptable (subtitles, voicemail transcription, audiobooks), everything above works and cost per minute dominates — self-hosted ASR wins outright. If conversational, you need the full streaming pipeline and your real constraint is the worst hop: a fast TTS model behind a non-streaming LLM still misses the budget. Voice-agent platforms like ElevenLabs' Agents product sell precisely the elimination of those hops plus concurrency headroom; at Business-tier rates the math can favor them before you account for the engineering time your own pipeline saves.

Compute third: the entire open stack now fits on one consumer GPU — faster-whisper int8 plus Chatterbox-Nano on CPU, or a single 8GB card running turbo Whisper and Chatterbox-Turbo. That fact, more than any benchmark, is what reset the field's price floor, and it is why the open-vs-commercial decision in 2026 is usually an operations decision (who runs the GPUs, who carries the pager) rather than a capability decision.

Where It Breaks

TTS hallucination and repetition. Generative TTS models produce loops, skipped words, invented continuations, and non-speech artifacts — Chatterbox's V3 release notes explicitly tout "reduced hallucination" and "reduced unwanted continuation, repetition, and off-prompt speech," which is a confession that V2 had these problems. Trigger: long text, unusual punctuation, out-of-distribution names and code tokens. Mitigation is chunked synthesis with validation (transcribe your own TTS output and compare), not hope.

Number, date, and normalization failures. Text normalization is the unglamorous half of TTS: "3.14.1" as a version, "1876" as a year, currency, abbreviations. Models with IPA-based G2P (Kokoro's misaki) and phoneme-tag markup handle these better but still fail on domain-specific strings. Trigger: any text containing identifiers — SKU codes, URLs, legal citations. The production answer is a preprocessing pass that rewrites non-words into speakable text, and budgeting for it is part of the real cost of "free" TTS.

Voice cloning consent and accent bleed. A 10-second reference clip is enough to clone a voice, which makes the consent question operational, not theoretical — and Chatterbox's own tips note that a reference clip in the wrong language transfers its accent into output. Trigger: user-supplied reference audio. The watermark helps detect, not prevent; policy (whose voices may be cloned, verified consent) is your application's job and your liability.

Whisper hallucinating on silence and noise. The original Whisper generates fluent phantom text — repeated phrases, invented sentences — over silence, music, or non-speech audio. Trigger: long recordings with quiet passages; this is precisely why faster-whisper ships the Silero VAD filter as a first-class option and enables it by default in batched mode. Un-vetted Whisper deployments that "worked in the demo" are a canonical production incident.

Latency collapse under concurrency. Every RTF number in this dossier is per-stream; RTF 0.04 at concurrency 1 does not mean 25 concurrent streams on one GPU. Trigger: voice-agent traffic spikes. The F5-TTS benchmark's own structure (latency measured at concurrency 2, client-server mode) is the honest template: benchmark your concurrency, not your demo.

Licence drift on pinned dependencies. Piper's community handoff changed the licence of the going-forward version; Coqui's weights were always trap-adjacent for commercial use. Trigger: building on a repo because of its stars rather than its licence file and governance. Re-check licences at every major pin — this field moves them.

Open Questions

The first is whether modular pipelines survive the full-duplex wave. If commercial speech-to-speech models reach pipeline-equivalent quality at agent-scale prices, the ASR→LLM→TTS stack that this entire open ecosystem supports becomes the "good enough" tier — which may be fine (the open pipeline's total cost keeps falling) or may hollow out (latency-critical agent work consolidates on closed models). The open community has no credible full-duplex answer yet; that is the field's largest gap.

The second is watermarking arms races. PerTh-style neural watermarks are cheap to embed and detectable "with nearly 100% accuracy" by their maker's own detector — but detection is only meaningful while the watermark survives and the detector is trusted. Whether watermark provenance becomes a regulatory requirement (and whether open models without built-in watermarks get regulated out of commercial use) is the policy variable that could reshape every licence decision above.

The third is the voice-data commons problem. Kokoro's $1,000 training run was possible because permissive and synthetic audio sufficed for a base model; genuinely multilingual, expressive, clone-safe TTS wants data no permissive corpus provides. The field will either grow its version of the LeRobot community-dataset model — consented, licensed, shared voice data — or stay dependent on synthetic-data distillation from the very closed models it competes with, which is a fragile foundation for an "open" stack.

Resources

Open TTS models and toolkits

Open ASR

Commercial pricing reference