Living dossier

Video Models

Back to topics•Blog
Mehran Mozaffari·
8 resources0 related posts

The Field Split in Two: A Closed Frontier and an Open Runtime Ecosystem

Video generation in 2026 is two markets wearing one label. The closed frontier — Sora 2, Veo 3.1, Movie Gen 2, Hailuo 3.0 — sells synchronized audio, physics-aware motion, 8K resolution, and seconds-per-clip turnaround as metered APIs and subscription products. The open ecosystem — Wan, HunyuanVideo, LTX, CogVideoX, Mochi, FramePack, and the ComfyUI graph that binds them — sells weights you download, VRAM you provision, and control you exercise yourself. The two markets interbreed constantly (closed-lab techniques show up in open releases within quarters), but the adoption decisions are entirely different: the closed side bills per second of video and hides its failure modes; the open side bills in gigabytes of VRAM and engineer hours, and hides nothing.

The most useful fact about the closed side is also the least advertised: products disappear. The Sora 2 announcement page — which introduced synchronized dialogue and sound effects and multi-shot storyboards — now carries the notice that the Sora product is no longer available. Any pipeline architected around a single closed video API carries that risk structurally; the open-weights ecosystem exists partly as insurance against exactly this.

What every model in both markets now shares is a common architecture: a diffusion transformer (DiT) operating on compressed video latents, a 3D causal VAE that compresses and reconstructs the pixel dimension, a text encoder conditioning the generation, and a scheduler that steps denoising. Understanding that shared skeleton is what lets an operator move between models without relearning the field.

The Anatomy: Why the VAE and the Scheduler Matter as Much as the DiT

The transformer gets the headlines; the pipeline around it gets the results. A text-to-video pass decomposes into: encode the prompt (often with a large language model — HunyuanVideo routes prompts through an LLM rewrite step before generation), map text embeddings onto noisy video latents via the DiT across denoising steps, then decode latents to pixels through the VAE. Each stage is a distinct cost centre and a distinct failure point.

The VAE choice is a genuine differentiator. Wan's Wan-VAE was designed to encode and decode 1080p video of arbitrary length while preserving temporal information, and CogVideoX shipped a 3D causal VAE that reconstructs video with near-zero loss — these are the components that determine whether long clips decode without flicker and whether I2V preserves the input frame faithfully. The scheduler and step count are the primary cost lever: distillation and cache-based acceleration (TeaCache reports roughly 2x speedup on Wan2.1; FastVideo ships consistency-distilled HunyuanVideo models and Sliding Tile Attention) trade a little quality for large multipliers in speed, which is usually the trade that makes a workflow close.

flowchart TD
    PROMPT["Prompt and optional input image"] --> ENC["Text encoder / LLM rewrite step"]
    LAT["Noise: compressed video latents"] --> DIT["Diffusion transformer<br/>denoising steps"]
    ENC --> DIT
    IMG["Input image (I2V)"] --> DIT
    DIT --> VAE["3D causal VAE decoder"]
    VAE --> PIXELS["Pixel frames: 480p to 4K, seconds long"]
    PIXELS --> POST["Frame interpolation, upscaling,<br/>sharpening, audio sync"]
    POST --> OUT["Delivered clip"]
    CTRL["Control: VACE, keyframes,<br/>pose, depth, trajectories"] -.-> DIT

Control is the second axis of the anatomy. Modern open models expose first-last-frame conditioning, multi-keyframe conditioning, depth and pose guidance, trajectory control, and editing models like VACE (Wan2.1's all-in-one creation-and-editing model). The controllability stack is where video generation stops being a slot machine: the same base model with a motion-control adapter (Wan-Move, accepted to NeurIPS 2025) or keyframe LoRA produces a repeatable shot instead of a lottery ticket.

The Open-Weights Field Is a Family Tree, Not a Leaderboard

The naive way to read this field is a ranked list. The correct way is a family tree, because the open models differentiate by lineage strategy, and each lineage carries a distinct bet.

Wan2.1 (Alibaba, Apache-2.0) bet on scale plus accessibility: a 14B flagship and a 1.3B model that needs only 8.19 GB of VRAM — a 5-second 480p clip on an RTX 4090 in about four minutes — plus Chinese and English text rendering, a strong Wan-VAE, and every task in the book (T2V, I2V, editing, text-to-image, video-to-audio). Its community tree is the widest in the field: VACE for editing, Wan-Move for point-level motion control, EchoShot for multi-shot portraits, LightX2V for inference on 8 GB cards, DiffSynth-Studio for FP8 quantization and LoRA training, TeaCache for 2x acceleration. Position: the default open foundation — if you standardise on one open video model today, this is the lowest-regret choice, because the ecosystem does your integration work for you.

HunyuanVideo (Tencent) bet on systematic scale and then on efficiency: a 13B DiT with LLM-encoded text conditioning, an FP8 weight release to cut GPU memory, parallel inference via xDiT, and a 1.5 generation described as a highly efficient new foundation. Its family grew sideways — I2V, Avatar (audio-driven human animation), HunyuanCustom for subject-customised generation, a Penguin Video Benchmark. Position: the strongest documentation-to-performance ratio in open video and the deepest acceleration ecosystem (FastVideo, FramePack, Sparse-VideoGen, RIFLEx all target it), with a custom Tencent community licence you must read before any commercial deployment.

LTX-Video / LTX-2 (Lightricks) bet on speed and audio-video unity: the first DiT model claiming all core capabilities in one system, now with LTX-2 adding synchronized audio-video generation, native 4K at up to 50 FPS claims, multi-keyframe conditioning, IC-LoRA control models, a latent upsampler, ComfyUI core integration, and LoRA training tools. Position: the real-time and audio-sync pick — the model to evaluate when the workflow is iterative (many short generations, fast feedback) rather than few-and-perfect.

CogVideoX (Zhipu/Tsinghua) bet on the accessible ladder: CogVideoX-2B open-sourced and explicitly re-licensed to Apache-2.0, runnable on a GTX 1080 Ti; the 5B tier on an RTX 3060; a 1.5 generation with 10-second clips; CogKit for fine-tuning. Position: the provenance pick with the clearest per-tier licensing, now more valuable as a fine-tuning base than as a generation frontier.

Mochi 1 (Genmo, Apache-2.0) bet early on high-fidelity motion with strong prompt adherence under a permissive licence, with LoRA fine-tuning and ComfyUI support. Position: historically important as the first model to "dramatically close the gap" between closed and open systems, now superseded at the frontier but still a clean reference implementation.

FramePack (Lvmin Zhang) bet on a different axis entirely: next-frame-section prediction that compresses input context to a constant length, making generation workload invariant to video length. The numbers are the story — a 1-minute 1800-frame video from a 13B model on 6 GB of VRAM, laptop GPUs included, with visual feedback per frame as generation progresses. Position: the anti-drift and accessibility pick for long-form generation on consumer hardware; treat its anti-drift claims as stress-testable rather than settled, per the author's own published test methodology.

The closed frontier, for contrast: Sora 2 shipped synchronized audio and multi-shot storyboards and its product page now states the product is no longer available; Veo 3.1 pushes real-time physics-aware motion into Gemini workflows; Hailuo 3.0 targets 10-second clips at lower cost; Movie Gen 2 targets 8K for advertising and VFX. These are capabilities to benchmark against, not platforms to build on, unless the vendor relationship is part of your risk budget.

ComfyUI Is the Actual Operating System of the Open Side

Every open model above ships with ComfyUI integration in its release notes, and that is not a coincidence — it is the field's real interface standard. ComfyUI is a node-graph engine for content creation: native support for the latest open models on release day, partner nodes for closed models, a desktop application and portable install across Windows, Linux, and macOS with NVIDIA, AMD, Intel, and Apple Silicon support, App Mode for exposing complex workflows as simple UIs, and API endpoints for production pipelines.

The operational meaning: a video generation workflow — prompt conditioning, LoRA loading, latent upscaling, frame interpolation, audio sync — is expressed as a saved graph, version-controlled as a JSON file, and executable headlessly. Teams standardise on ComfyUI graphs the way earlier teams standardised on Docker images: the graph is the reproducible unit of production. The cost of that standardisation is graph complexity — production ComfyUI graphs for Wan or Hunyuan are dense artifacts that encode cache settings, quantization choices, and seed management — and the benefit is that swapping Wan2.1 for HunyuanVideo is rewiring nodes, not rewriting a pipeline.

Two ComfyUI facts from its README round out the operational picture. Partner nodes expose closed models (Seedance, Hunyuan3D-class generators) inside the same graph that runs open weights, which makes mixed open/closed pipelines — draft with open weights, finish with closed — a node-level decision rather than a platform commitment. And App Mode plus the API endpoints mean a validated graph can be packaged for non-technical operators or called from production services, which is how the research-era tool grew into the delivery layer for commercial content pipelines. The same flexibility cuts both ways when debugging: a graph that can run any model can fail in model-specific ways, and reading someone else's production graph to find which node holds the bad LoRA or the wrong cache setting is the video-era equivalent of reading someone else's Makefile — possible, unpleasant, and occasionally necessary.

Comparison: The Named Options on the Axes That Decide Adoption

Model / tool Open weights and licence Scale and output VRAM class Distinctive strength Cost shape
Wan2.1 (Alibaba) Apache-2.0 1.3B and 14B; 480p/720p, minutes-long via community 8.19 GB (1.3B) up to multi-GPU (14B) Widest community ecosystem; VACE editing; bilingual text Free weights; your GPU hours
HunyuanVideo / 1.5 (Tencent) Custom Tencent community licence 13B DiT; 720p class FP8 weights to fit consumer cards Deepest acceleration ecosystem; Avatar and Custom variants Free weights; read licence for commercial
LTX-Video / LTX-2 (Lightricks) Open weights, custom Lightricks terms DiT; up to 4K/50 FPS claims; audio sync Efficient DiT tier Speed plus synchronized audio in one pass Free weights; check terms per deployment
CogVideoX / 1.5 (Zhipu) 2B Apache-2.0; 5B custom 2B-5B; 10 s clips (1.5) GTX 1080 Ti (2B) to RTX 3060 (5B) Most accessible ladder; CogKit fine-tuning Free weights at the accessible tier
Mochi 1 (Genmo) Apache-2.0 Preview-scale; high-fidelity motion Consumer GPU with offload Clean reference implementation, LoRA tuning Free weights
FramePack (Lvmin Zhang) Open code (check repo) 13B next-frame model; 60 s at 30 fps 6 GB minimum, laptop GPUs Constant-length context: length-invariant generation Free; your time is the cost
Sora 2 (OpenAI) Closed, API/product Synchronized audio, multi-shot storyboards Vendor-hosted Audio-video sync at the frontier Metered per generation; product availability has already changed once
Veo 3.1 (Google) Closed, Gemini-integrated Physics-aware motion, live iteration Vendor-hosted Real-time iteration inside Gemini Subscription/metered

The Production Loop: Generate, Select, Refine, Sync, Deliver

Real video workflows are loops, not one-shot generations, and the sequence that repeats per approved shot looks like this:

sequenceDiagram
    participant P as Producer
    participant G as Generation (Wan / LTX / Hunyuan)
    participant R as Review
    participant E as Edit and sync layer

    P->>G: Brief - shot list or reference frame
    G->>G: Generate N candidates (low-step preview)
    P->>P: Pick candidate
    G->>G: Full-step regenerate or VACE/keyframe refine
    P->>R: Review clip - consistency, motion, physics
    alt rejected
        R->>G: Adjust prompt, seed, or control input
    else approved
        R->>E: Send to upscale and audio sync
        E-->>P: Delivered clip at final resolution
    end

Three properties of that loop decide throughput. Preview quality: low-step generations must be good enough to select from, which is why distillation and cache acceleration matter even before the final render. Selection bandwidth: a producer reviewing five candidates per shot needs the loop to cost seconds, not minutes — this is where LTX-class speed and FramePack-style progressive feedback earn their place. And refinement control: VACE-style editing, keyframe conditioning, and IC-LoRA controls are what convert a rejected candidate into a targeted fix rather than a full regeneration. Teams that skip the control models pay for it in regeneration volume.

The Acceleration Stack Is Now a Layer of the Field

Video diffusion at usable settings is expensive enough that a distinct acceleration ecosystem formed around it, and it has become a selection criterion in its own right. The layers, from inference tricks to training changes: TeaCache reports roughly 2x speedup on Wan2.1 by caching; FastVideo ships consistency-distilled HunyuanVideo plus Sliding Tile Attention; Sparse-VideoGen accelerates generation with high pixel fidelity; Jenga does dynamic token carving; RIFLEx extends video length extrapolation; LightX2V packages engineering acceleration that runs Wan-class models on an RTX 4060 with 8 GB; xDiT provides multi-GPU parallel inference; and DiffSynth-Studio adds FP8 quantization, VRAM optimization, and video-to-video. On the hardware end, Helios — a Wan2.1-based research model — reports minute-scale synthesis at 19.5 FPS on a single H100.

The operator consequence: two teams with identical GPUs can see order-of-magnitude different throughput depending on which acceleration stack they run, and acceleration quality is not free — cache methods and distillation trade small fidelity losses for large speedups, and the losses compound when stacked. The discipline is to measure: same prompt, same seed, accelerated versus unaccelerated, judged on held-out frames before adopting an acceleration setting for production.

Synchronized Audio Is the New Battleground

The 2026 generation of releases converged on the same capability from both sides of the open/closed line: Sora 2 launched with synchronized dialogue and sound effects; LTX-2 ships synchronized audio-video generation as its headline feature; closed competitors push physics-aware and 8K output. The significance for production is that audio-in-model removes an entire post-production pipeline — separate dialogue generation, sound-effect timing, and mixing — and replaces it with a single conditioned generation. The open questions are lip-sync reliability on dialogue, sound-effect temporal accuracy, and whether audio quality holds at the resolutions video teams actually deliver at. Teams evaluating LTX-2 or its closed peers for dialogue-heavy work should budget evaluation time specifically for audio-video sync failures, which are more noticeable to audiences than any visual artifact.

Where It Breaks: Failure Modes and Their Triggers

Temporal drift on long generations. Trigger: minutes-long clips from models trained on seconds-long segments. Content drifts — identity, lighting, scene layout — because the model has no mechanism anchoring frame 1200 to frame 1. FramePack's constant-length context packing, Helios's minute-scale training approach, and segment-chaining with reference frames are three different mitigations; none is solved, and drift remains the default failure of long-form generation.

Prompt adherence collapses on physics. Trigger: prompts specifying object permanence, collisions, fluid behaviour, or cause-and-effect. Video models interpolate appearance statistics; they do not simulate mechanics. Hands, glasses filling with liquid, text on signs, and object interactions fail at rates that shot-level beauty masks in a demo. The production answer is to avoid asking for physics in one shot: generate, inspect, regenerate with adjusted prompts, or composite real footage.

VRAM exhaustion at the decode stage. Trigger: high resolution and long clip lengths in one pass. The DiT may fit while the VAE decode OOMs — memory peaks at different pipeline stages. Mitigations: CPU offloading, FP8 or GGUF quantized weights, tiled VAE decode, and shorter segments stitched. The LightX2V and DiffSynth projects exist because this failure mode is universal.

Shot-to-shot inconsistency in multi-shot work. Trigger: generating a sequence of clips that must share characters, wardrobe, and style. Text prompts are too weak a channel for identity. The workarounds — reference-image conditioning (Phantom), multi-shot native models (EchoShot), subject-customised architectures (HunyuanCustom) — are improving, but multi-shot consistency remains the gap between a clip factory and a story tool.

Licence traps in the open tier. Trigger: choosing weights by benchmark without reading terms. Wan2.1, Mochi, and CogVideoX-2B are Apache-2.0; HunyuanVideo and the larger CogVideoX tiers ship custom community licences with separate commercial terms; Lightricks' LTX weights carry their own conditions. The wrong discovery happens after the pipeline is built. Read the licence file of the exact checkpoint before architecting anything.

Counterfeit distribution. Trigger: popular desktop tools attract fake mirror sites. FramePack's README maintains an explicit blacklist of spam domains and warns against paying or downloading from any of them. Any one-click download should be traced to the official repository's release page — this is a malware channel, not a hypothetical.

Open Questions the Field Has Not Settled

Whether synchronized audio-video generation becomes table stakes or a premium differentiator is being decided right now by LTX-2 and Sora 2's launch generation — the capability is real, the workflow integration (dialogue lip-sync reliability, sound-effect timing) is young. Whether minutes-long coherent generation arrives through architectural fixes (FramePack's constant-context prediction, Helios's minute-scale training) or through segment stitching with reference anchoring is unresolved, and the answer determines whether long-form video becomes a generation problem or an editing problem. Whether open weights close the gap with the closed frontier or the frontier's audio-physics integration keeps a permanent moat is the commercial question every team building on open models is implicitly betting on. And the field still lacks a production-grade evaluation — text-to-video benchmarks measure brief alignment statistics, while production fails on character consistency, lip-sync accuracy, and edit-friendliness, none of which any leaderboard scores. Until then, the working rule stands: pilot with your own briefs, count your own regenerations, and price your own engineer hours — because in video generation, the render was never the expensive part.

Resources

Open-weights model repositories

Delivery layer and closed frontier