How I'd Build a Real-Time Conversational Avatar: GPT-Live, LiveAvatar, and the Tool-Call Overlay Problem

The three layers this stack actually composes

The cleanest way to understand this architecture is to stop thinking of it as one system and start thinking of it as three planes that happen to share a session ID. Each plane has a different job, a different latency budget, and a different failure mode — and almost every production problem I'd expect from this stack traces back to someone collapsing two of those planes together.

The first plane is intelligence and audio generation: OpenAI's GPT-Live, the speech-to-speech model speaking the v3 contract as gpt-live-1. Its responsibility is narrow and hard: take continuous mic audio in, resolve turn-taking in full duplex, emit streaming voice audio, and — this is the part that makes the whole demo interesting — emit structured tool-call events concurrently with the audio. A cascaded stack would have to bolt that on after the fact. Here it's native, which is why the emotional expressiveness and barge-in behavior are so much better than an STT→LLM→TTS chain. The model is also the source of truth for when a tool call happened, relative to the speech that triggered it. Hold onto that, because it's the root of the sync problem later.

The second plane is audio-driven visual synthesis: HeyGen's LiveAvatar, driven through a LITE session WebSocket endpoint. Its responsibility is to receive raw PCM-ish audio and produce a lip-synced, face-animated video + audio track, streamed to the browser over LiveKit WebRTC. Note what LiveAvatar does not know about: transcripts, tool calls, session state, what the user asked. It is a rendering service. Treating it as one is the whole design.

The third plane is the client-side overlay: HyperFrames-style animated UI that reacts to tool-call events. Its responsibility is to draw things — term cards with pronunciation and meaning, canvas transforms that shrink the avatar into a corner and lay out a recap panel. This is DOM/vector work on the client, not pixels baked into the video track. That's a deliberate choice: crisp 60fps cards, no extra video bandwidth, no server-side compositing GPU cost.

The orchestrator in server/ is the router that keeps these three planes decoupled. It bridges browser mic audio to GPT-Live over WebSocket, routes the model's voice audio directly into LiveAvatar's LITE WS — the browser never touches that leg — and relays tool-call events down a separate WebSocket channel to the client. Two different transports, two different timing characteristics, one session.

flowchart LR
    subgraph Browser["Browser (web/)"]
        Mic["Microphone capture"]
        UI["HyperFrames overlay\nterm cards, canvas state"]
        Player["LiveKit media player"]
    end

    subgraph Server["Orchestrator (server/)"]
        Router["WS router +\nsession state"]
    end

    subgraph AI["OpenAI GPT-Live"]
        S2S["Speech-to-speech\ngpt-live-1 / v3"]
    end

    subgraph Avatar["HeyGen LiveAvatar"]
        LITE["LITE session\nWS endpoint"]
        Render["Real-time\nneural render"]
    end

    Mic -->|"WS uplink: mic audio"| Router
    Router -->|"WS: audio in"| S2S
    S2S -->|"voice audio"| Router
    S2S -->|"tool-call events"| Router
    Router -->|"WS: audio to avatar"| LITE
    Router -->|"WS downlink: transcripts + tool events"| UI
    LITE --> Render
    Render -->|"LiveKit WebRTC: avatar A/V"| Player

The separation matters for a specific reason: each plane can fail, scale, and be replaced independently. Swap the S2S model, swap the avatar vendor, swap the overlay renderer — the interfaces hold. And crucially, it lets the server be the security boundary: the browser holds no API keys, only LiveKit media and sanitized UI messages.

Why routing audio server-side to the avatar instead of through the browser

The natural first instinct — and honestly the one I'd sketch on a whiteboard — is to have the S2S model stream audio back to the browser, then have the browser forward that audio to the avatar service. It feels simpler. Everything terminates at the client. The client is already a WebRTC peer to LiveAvatar anyway.

It's the wrong call, and the reason is arithmetic.

In that naive topology, every chunk of synthesized speech makes a trip: model → orchestrator → browser → avatar service. Even on a good connection, that's an extra client round-trip on every audio chunk — realistically 40–120ms added per direction depending on geography and whether the browser is on wifi, 4G, or worse. Worse still, it's jittery round-trip: the browser's network path is the least predictable in the system. Audio that arrives late doesn't get nicer; it gets buffered, and buffering audio in a real-time avatar pipeline is exactly the thing that makes lips move after the mouth has already stopped.

The design here sidesteps that entirely. GPT-Live's voice audio lands at the orchestrator and goes straight out to LiveAvatar's LITE session WebSocket. The browser sits outside the audio-to-avatar loop completely. What the browser receives is the finished avatar A/V over LiveKit — one media leg, not two hops plus a client relay.

That matters because the end-to-end target is roughly 600–1000ms, and that budget is not generous once you itemize it. Network RTT to the model, model inference and first-audio-out, network RTT to the avatar service, avatar render start, WebRTC jitter buffer, playout. Pulling a client round-trip out of that chain is one of the larger single savings available, and it's the cheapest one — it costs you nothing in quality.

The cascaded paradigm shows why the serialization hurts even more. In an STT→LLM→TTS chain, each stage is a full handoff: STT must finalize a transcript before the LLM can start, the LLM must finish enough tokens before TTS can start chunking, and then TTS audio feeds the avatar. Each of those boundaries adds latency and, more painfully, add variance — chunked TTS handoffs are notorious for jitter that no amount of tuning fully flattens. Native S2S skips three of those boundaries.

There's a second, non-latency benefit worth naming plainly: the browser never holds OPENAI_API_KEY or LIVEAVATAR_API_KEY. All privileged calls happen in the orchestrator. That's not a nice-to-have — for anything beyond a demo, shipping keys to a client is a non-starter.

Dimension	Native S2S hub (GPT-Live → LiveAvatar)	Cascaded pipeline (STT → LLM → TTS → Avatar)	Monolithic engine (Tavus / D-ID)
End-to-end latency	Fast, ~600–1000ms; no STT/TTS serialization	Moderate-to-fast, ~800–1200ms; chunked TTS handoff adds jitter	Variable, ~900–1500ms; depends on vendor routing and region
Emotional expressiveness	High; full-duplex S2S captures tone, sighs, interruptions, pacing	Moderate; TTS leans on SSML or text sentiment, often slightly synthetic	Moderate to high; vendor-engine dependent, opaque to you
Tool-call synchronization	High; native tool events fire concurrently with audio over WS	High if the framework emits data packets alongside audio buffers	Low; hard to intercept sub-turn tool calls without a rich webhook
Cost per interactive minute	High; S2S audio tokens plus real-time video rendering, two vendors	Lowest and most flexible; swap in cheap LLM, fast TTS, cheap video	Fixed tiered pricing; bundled markup on compute and streaming
Interruption handling	Native duplex barge-in, but you must flush the avatar's audio buffer	High control; custom VAD lets you tune interrupt sensitivity exactly	Handled opaquely by the vendor

I'd reach for the native hub whenever conversational naturalness and UI synchronization are the product, and eat the per-minute cost. The cascaded stack wins when you need cheap or on-prem, or when you want to hand-tune barge-in with your own VAD.

The tool-call overlay clock: how WebSocket events and WebRTC tracks fall out of sync

Here's the failure mode I'd bet money on being the first thing that bites anyone taking this from demo to production: the avatar's voice and the term card it's annotating are traveling on two different clocks.

The avatar A/V comes over LiveKit WebRTC, which carries a dynamic jitter buffer. That buffer is doing its job — it smooths network variance so audio doesn't stutter — but its effect is that the media you see and hear is deliberately delayed by some adaptive amount, and that amount changes as the network changes. Tool-call events, meanwhile, arrive over a plain WebSocket with no ordering guarantee relative to the media stream. Nothing in the protocol says the card render and the spoken word are aligned, because they were never on the same timeline to begin with.

On a good connection, the offset is small enough to be invisible. The demo looks great. On a degraded connection — a phone on a lift, hotel wifi, a saturated office network — the jitter buffer stretches, and now the card lands half a second before or after the word it's supposed to annotate. For a language tutor showing a pronunciation card, that's not cosmetic: the card is supposed to attach to the utterance. If it appears after the model has already moved on, the learner is reading the previous term while hearing the next. For a sales copilot surfacing an objection-handling card, the same offset means the rep gets the cue late. "Good enough" for a demo becomes "noticeably broken" the moment the UI's whole value is being synchronized in time with speech.

sequenceDiagram
    participant GL as GPT-Live
    participant OR as Orchestrator
    participant LA as LiveAvatar
    participant LK as LiveKit WebRTC
    participant BR as Browser UI

    GL->>OR: voice audio + tool-call event (same instant)
    OR->>LA: WS: audio to avatar LITE session
    OR->>BR: WS downlink: tool-call event
    LA->>LK: avatar A/V track
    Note over LK: dynamic jitter buffer adds variable delay
    LK->>BR: avatar A/V plays out
    Note over BR: tool-card renders on WS callback,<br/>at a different wall-clock moment than the spoken term
    Note over BR: visible misalignment between card and speech

The mitigations I'd actually pursue, in order of how much control they buy you:

Timestamp tool events on the server against an audio frame clock. Instead of firing a card render the instant a tool call arrives, tag the event with the position in the audio stream it belongs to. The orchestrator already routes that audio; it knows the frame index. Now the client has something to align to rather than "whenever this WebSocket message happened to arrive."

Buffer UI renders to match an observation delay. If LiveAvatar is rendering frame N of audio while the client is playing frame N−k because of the jitter buffer, the overlay should target frame N−k too. Estimate or measure that offset on the server, and delay tool-card rendering by the same amount. This is the same trick video players use to align subtitles, and it works for the same reason.

And the case with no clean solution: when the jitter is client-side only — a burst of local packet loss that stretches the browser's own buffer — the server has no idea it happened. You can't compensate for a delay you can't observe. The honest fallbacks there are to render cards optimistically (show them, then correct placement) or to lean toward a slight intentional offset that favors catchability: a card that appears a hair early reads better than one that appears late. Either way, I'd instrument the offset in production and treat "median absolute card-to-speech misalignment" as a first-class metric, because it's the number that decides whether this feels like a tutor or a slideshow with a talking head.

Interrupt handling: what happens when the user barges in

Barge-in is where the three planes stop agreeing about what time it is. GPT-Live handles the conversational side natively: the moment the model detects the user has started speaking over it, it cuts off generation. No manual VAD tuning, no interrupt threshold to fiddle with, no "wait for the transcript to finalize" delay. That part is genuinely solved, and it's a big part of why native S2S feels better than a cascaded stack, where you'd be hand-tuning Silero VAD sensitivity to get the same responsiveness.

But the model cutting off does not cut off the avatar. GPT-Live stops emitting audio. LiveAvatar, meanwhile, is still holding whatever audio it had already buffered when the interrupt arrived, and it will happily keep rendering lip movement against that stale buffer unless someone tells it to stop. So the orchestrator has to send an explicit truncate or reset instruction down the LITE session WebSocket — essentially "drop everything after frame N, stop rendering the current utterance." Picture the avatar mid-word when the user barges in: without that flush, the mouth keeps shaping syllables for a sentence nobody is going to hear, and then it has to catch up mid-word once the new response starts. It reads as a glitch, and it's the single most obvious tell that a real-time avatar pipeline wasn't built carefully.

The tool-call side has its own reconciliation problem. If a tool event fired just before the interrupt — a term card was about to render — that event is now describing speech the user never heard fully. The clean choice is to clear pending tool-call events on flush: any event whose audio window was truncated gets dropped, and the orchestrator replays correct state on the next turn. The harder case is state that's already on screen. Because the recap panel is built from the server's session record rather than the model's context window, the server knows what it actually committed to before the cut. The client has to reconcile: what's on screen versus what the model thought it said versus what the server recorded. If those three diverge after a barge-in, the user sees a card for a term the avatar never finished saying.

stateDiagram-v2
    [*] --> IDLE
    IDLE --> SPEAKING: audio streaming into LiveAvatar buffer
    SPEAKING --> INTERRUPTED: user barge-in detected by GPT-Live
    INTERRUPTED --> FLUSHING: orchestrator sends truncate/reset to LiveAvatar LITE WS, clears pending tool-call events
    FLUSHING --> IDLE: buffer cleared, stale rendering halted
    IDLE --> SPEAKING: next turn begins

Honest cost and lock-in diagnosis

I'm not going to pretend the cost profile here is anything other than expensive. This architecture bills you per interactive minute from two vendors at once: OpenAI's S2S audio-token pricing, which is materially higher than text tokens because you're paying for continuous audio both directions, and HeyGen's real-time neural rendering rates, which are priced like what they are — GPU time driving a photorealistic face in real time. Multiply that by full-duplex sessions where the meter runs the entire time the user has the connection open, including the silences.

The cheaper path is real and I'd take it in plenty of contexts. Groq hosting Llama 3 handles the language side for a fraction of the cost; an open TTS like Kokoro, or Cartesia for something faster and more polished, handles voice synthesis; Simli handles lip-sync at low compute overhead and is deployable closer to your own infrastructure. Individually each of those is cheaper, and the combination lets you tune the trade-off per component. What you give up is the thing that's actually the product here: native S2S emotional expressiveness — the tone, the sighs, the pacing shifts, the way the model's voice carries a question mark or a laugh — plus barge-in behavior that doesn't require you to hand-tune interrupt sensitivity and won't feel robotic at the edges. TTS driven by SSML tags approximates expressiveness; it doesn't reproduce it. For a language tutor where pronunciation and natural prosody are the teaching material, that gap isn't a nice-to-have, it's the whole point.

The lock-in question is where the architecture's decoupling actually earns its keep. Because GPT-Live and LiveAvatar communicate only through the orchestrator — never directly, never sharing a session object — either one can be swapped behind its interface. If HeyGen's LITE session contract changes or the service degrades, you re-point the orchestrator's audio output to a different avatar endpoint; the S2S leg is untouched. If you want to move off GPT-Live, the reverse: keep the avatar, swap the intelligence plane. That isolation is the reason the premium is recoverable rather than a trap. What it does not buy you is a cheap exit — moving to the modular stack means rewriting the orchestrator's routing, adding VAD, and accepting the expressiveness downgrade, so the migration path exists but has real cost. The honest framing: you're paying a premium for conversational quality and paying a second premium in optionality, and the decoupling is what keeps that second premium from being pure lock-in.

Session state vs. model memory: the recap panel pattern

The subtlest decision in this reference implementation is one you can miss entirely if you're watching the demo instead of reading the code: when the avatar says "let's recap what we covered," that recap is not coming from the model's context window. It's assembled from the orchestrator's own structured session record and handed to the model as content to voice.

I've seen the alternative fail too many times to treat this as a stylistic preference. A context window after a forty-minute tutoring session is a pile of interleaved utterances, half-finished corrections, and tool-call chatter — the model can try to summarize it, and it will do so confidently, omitting and occasionally inventing terms. The server, by contrast, has an exact append-only ledger: every term introduced, every pronunciation, every tool call fired, every card the client rendered. That's a queryable fact, not a reconstruction.

So the split of responsibility is clean: the server owns what was covered, the model owns how to present it conversationally. When the user asks for a recap, the orchestrator pulls the relevant slice from its record, assembles a structured payload, and streams it to the client over the same WebSocket downlink that carries tool events. The client — the HyperFrames-style overlay layer — shrinks the avatar into a corner canvas and lays out the panel from that payload. The model's only job is the voice track narrating over it. This is exactly the kind of state decoupling the three-plane architecture is built to enable, and it's why that decoupling matters beyond latency.

The payoff is that the recap is correct by construction. It cannot hallucinate a term that was never taught, because the term was never in the state. The failure mode shifts from "the model invented a flashcard" to "the model phrased the narration awkwardly" — a far better class of bug to have. It also means the client can render instantly from a payload it already trusts, rather than waiting on a model round-trip to generate UI content.

flowchart TD
    A[GPT-Live emits tool-call event<br/>term + pronunciation] --> B[Orchestrator appends to<br/>structured session record<br/>in server-side state]
    B --> C[Client receives WS event,<br/>renders term card overlay]
    D[User asks: what did we cover?] --> E[Orchestrator queries<br/>session record, assembles recap payload]
    E --> F[Sends snapshot to client]
    F --> G[Client shrinks avatar canvas,<br/>renders recap panel from state]
    F --> H[LLM voices the recap<br/>over the panel]

That's the line between a demo and a product. A demo reconstructs state from the model and looks brilliant in a three-minute clip. A product keeps its own record, because the moment the session runs twenty turns longer than the demo, model memory is the wrong place to store anything you need to be true.

Two reference demos worth building on top of this

The reference repo ships a Japanese language tutor, and the tutoring pattern is the one I'd extend first. What makes it interesting is the remediation flow: a learner mispronounces a word, GPT-Live catches it conversationally — this is where native S2S earns its premium, because it's working from the actual audio, not a lossy transcript — and fires a tool call carrying the mispronunciation. The orchestrator routes the audio to LiveAvatar over the LITE session WebSocket while the WebSocket downlink pushes a term card to the client overlay. The card shows the correct term and a phonetic breakdown, so the learner sees the mouth shape, hears the model's pronunciation, and reads the decomposition in the same beat. Every correction gets logged to the server-side session record, which is what makes a "what did I struggle with today?" recap at the end not just possible but trustworthy.

Two things will bite you. First, alignment: when the tutor fires the corrective tool call and then immediately says the word, the card and the spoken word travel on the two different clocks I described earlier. If LiveKit's jitter buffer has stretched, the card can land before the audio — better than late, but still off. Second, interruption: if the user barges in mid-correction, you have to flush the LiveAvatar buffer and decide whether the card state should revert. A card for a correction the tutor never finished delivering is worse than no card at all.

The second demo I'd build is a sales copilot with live objection handling. GPT-Live runs the conversation, detects objection phrases, and fires tool calls; the orchestrator sends the generated response to LiveAvatar and simultaneously pushes a structured event to the client, which renders a counter-argument card or a comparison table in the overlay while the avatar keeps talking. The lead-qualification and framing work I've written about elsewhere — the RF-DETR pipeline piece covers a different domain, but the "don't trust the model to remember what happened, keep your own record" instinct is the same one at play. The two watch-fors are cost and recap. A thirty-minute sales call is a dual-vendor bill running the whole time from both OpenAI S2S and HeyGen streaming — that meter doesn't pause during silences, and it will shock you the first month. And the end-of-call summary panel has to come from server-side session state, the enumerated list of objections and counter-arguments, not the model's context window. By minute twenty-five that context has jumbled, and a recap that drops the objection the customer cared most about is a deal-killer.

Resources

• GitHub - heygen-com/liveavatar-gpt-live-demos · GitHub
• app.liveavatar.com
• liveavatar-gpt-live-demos/docs at master · heygen-com/liveavatar-gpt-live-demos · GitHub