FFmpeg Skill: The Deterministic Control Plane for Media-Specific AI Agents

The Core Insight: An Agent Skill for Media Is a Control Plane, Not a Command Aide

The most damaging thing an LLM can do with FFmpeg isn't make an error — it's make a confident error. Ask a vanilla agent to "make this video vertical," and it will happily emit a filtergraph that assumes the source is 30fps SDR, that audio is stereo 44.1kHz, and that the pixel format is yuv420p. It will produce a washed-out, desynced, incorrectly-framed file and report success, because the process exited with code 0.

That's the exact failure mode ffmpeg-skill was architected against. Its author didn't build a catalog of FFmpeg incantations for an agent to rifle through. They built a constrained control plane — a workflow the agent cannot bypass, no matter how tempted it is to inline a quick ffmpeg -i command.

The design thesis is the four-step lifecycle, and each step exists to neutralize a specific class of hallucination:

probe.py first, always. The agent must never make a decision about media based on a filename or a user's casual description. "This video" could be 23.976fps or 60fps. It could be BT.2020 HLG or BT.709. It could have six audio tracks or none. Probing turns the agent's guesswork into grounded facts, and the entire downstream pipeline assumes those facts are accurate.

edit.py, lossless first. The strategy here is inverted from what an LLM naturally does. Most agents default to libx264 re-encode because it's the flag they know. This skill instead defaults to stream copy (-c copy) and container remuxing, and only escalates to re-encoding when the operation genuinely requires decoding — a tonemap, a re-frame, a filter. That single ordering principle keeps quality high and render times short for the operations where it's achievable.

check.py validates against delivery specs. Outputting a valid file is not the same as outputting a deliverable file. A 9:16 export that's 1920x1080 in landscape is technically valid and completely useless. The check step enforces target platform constraints — X's resolution and codec rules, for instance — before the agent ever reports success.

verify.py confirms the output is real. A/V sync, color gamut integrity, container conformance. This is where the agent catches the drift that stream-copy cuts introduce on variable-frame-rate footage, or the color shift that a bad tonemap produces. It closes the loop that most LLM-driven FFmpeg usage simply skips.

flowchart TD
    A[probe.py: inspect streams] --> B[edit.py: lossless -c copy remux first]
    B --> C[check.py: validate output integrity and platform specs]
    C --> D[verify.py: confirm A/V sync, color gamut, container conformance]
    A -->|resolution, framerate, duration, color space, audio channels| B
    C -->|X delivery rules| D

The result is a skill that treats an LLM as an operator who needs guardrails, not as a shell with a high-level vocabulary. The agent's job is to intend the operation; the skill's job is to make sure the operation is executed as a media engineer would execute it.

What It Actually Ships: The 21 Tools Are Not a Flat MCP Surface

Most MCP servers that expose FFmpeg do it as a directory listing — 40 tools, each a thin wrapper around a command. The agent picks trim_video or convert_format and hopes for the best. ffmpeg-skill ships 21 tools, but they're not a flat surface; they cluster into media-engineering domains, and each cluster has different requirements from the pipeline.

Reframing and resizing is where the tooling is most typed. Request "9:16," and the skill routes to an aspect-ratio conversion operation that knows it needs probe data about source dimensions to decide between crop and pad. If you're cropping a horizontal video to vertical, you're re-encoding — the filtergraph is applied to decoded frames. There's no stream-copy shortcut for geometry changes.

Silence cut is the classic lossless-vs-accuracy tension. The tool detects silent sections (using silencedetect logic) and trims them. When it does this with -c copy, hits are constrained to GOP boundaries, so the cut snaps to the nearest keyframe. That's fast and lossless, but it's not sample-accurate. The skill's typed parameters let the agent choose --accurate to re-encode and get frame-exact cuts when precision matters more than speed. This is exactly the kind of nuanced decision a generic MCP wrapper glosses over.

Subtitles and captions are another domain with a hidden dependency: subtitle burning is a filter operation. It decodes video, renders text overlays, and re-encodes. But generating subtitles is not an FFmpeg capability — the skill styles and burns existing .srt or .ass files, and karaoke-style text overlays are custom render paths. An agent that wants to "auto-caption" still needs a transcription service upstream.

HDR-to-SDR tonemapping is where the tooling's media-engineering polish shows. It isn't just a zscale filter chain; it's a color pipeline that respects primaries (BT.2020), transfer functions (HLG's arib-std-b67, PQ's smpte2084), and matrix coefficients. Skipping any of those produces the washed-out, milky SDR output you see from naive conversions. The skill's typed parameters are explicit about these steps.

Audio processing (LUFS normalization, dynamics, loudness targeting) and overlays/motion graphics round out the per-clip tools, while multicam and batch operations handle the project-level case — working across a directory of clips or combining multiple angles into a single render.

The packaging matters too. It ships as an npm package (npx ffmpeg-skill) and an MCP server integration (mcp/). That dual distribution means it's a one-liner to drop into Claude Code, Cursor, or Codex via the MCP path, and equally easy to invoke in a headless CI pipeline via npx. It requires only Python 3.9 standard library and local FFmpeg 5.0+, with no cloud APIs, no tokens, no network calls.

Tool Classification	What It Needs from Probe	Re-encode Trigger
Reframing / resizing	Source resolution, aspect ratio	Always (rotation, crop, pad all require decode)
Silence cut	Duration, stream presence, silence detection timestamps	Conditional — stream copy when GOP snapping is acceptable; re-encode with --accurate
Subtitles / captions	For burning: resolution for text sizing, pixel format	Always (text overlay is a decoded-frame filter)
Color / HDR tonemapping	Color primaries, transfer function, matrix coefficients, bit depth	Always (tone filter operates on decoded frames)
Audio processing	Channel layout, sample rate, LUFS levels, length	Conditional — audio-only adjustments may need re-encode of the audio stream only
Overlays / motion graphics	Resolution, framerate, duration for animation timing	Always (composition happens on decoded frames)
Multicam / batch	Per-clip stream info (resolution, fps, sample rate, codec)	Conditional — joins with matching codecs may stream-copy; heterogeneous input forces normalization and re-encode

The design principle I'd highlight here: the skill's job is to know when re-encoding is unavoidable, and to make that decision an explicit, typed one rather than an LLM's default behavior. That's the difference between an MCP wrapper and a media control plane.

The Lossless-First Default Is a Quality Over Speed Choice That Changes the Economics

The stream-copy default is often described as a speed optimization, and that's a misreading. Speed is a side effect. The primary reason ffmpeg-skill reaches for -c copy first is that it preserves the original encoder's decisions — the exact quantization, motion vectors, and bit allocation the source was mastered with. Re-encoding introduces generational loss that compounds across every subsequent operation. A rough cut that's remuxed losslessly is identical to the source. The same cut re-encoded at even a generous CRF is a different asset.

That cost is real for the rough-cut phase of a project, when you're making a dozen trims before finalizing the timeline. If each trim re-encodes, you've re-encoded the same material twelve times before the final render. The skill's design intent is to keep the interim artifacts bit-identical until the operation genuinely demands decode.

But there's a hard ceiling on what stream copy can do, and the skill's typed parameters make that explicit rather than hidden.

Dimension	Lossless Stream Copy (-c copy)	Frame-Accurate Re-encode
Quality preservation	Bit-identical to source; zero generational loss	Introduces re-encode artifacts; loss compounds across multiple passes
Speed	Near-instant; only demux/remux, no decode or encode	Full decode + encode; seconds to minutes depending on resolution and codec
VFR behavior	Cannot fix timestamp drift; desync persists and can worsen across cuts	Can enforce CFR (-fps_mode cfr), resampling timestamps to eliminate drift
Color space handling	Passes through untouched; fine for simple trims/remuxes	Required for tonemapping, color conversion, or any filter chain
Container compatibility	Must match codecs and stream structures across concatenated clips	Can normalize codecs, sample rates, pixel formats to unify heterogeneous inputs
When to prefer	Rough cuts, container format changes, spike removal, lossless concatenation of matching clips	Frame-exact trims, HDR-to-SDR, subtitles, overlays, joining heterogeneous footage, VFR correction

The operational rule I'd encode: reach for stream copy when the operation is structural — changing container, cutting at boundaries, reordering clips. Re-encode when the operation is transformative — changing pixels or audio samples. The skill's design bakes in that distinction, which is precisely what most generic MCP wrappers and raw LLM command generation miss. A raw agent defaults to re-encoding because libx264 is the flag it knows; this skill defaults to copy because that's the flag a media engineer would choose after probing the source.

The Failure Modes That Will Actually Bite in Production

The predictable failure modes in local FFmpeg automation aren't random bugs — they're the same five mechanisms, reproduced across every project. Here's what they actually look like and how each one manifests.

VFR desync. Smartphone footage and screen recordings are almost always Variable Frame Rate. The problem is that stream-copy operations cannot correct timestamp drift — they preserve whatever timebase the source has. When you cut a VFR clip at two points, each cut inherits the source's jittery timestamps, and those errors accumulate across sequential cuts. By the third or fourth trim, audio can be hundreds of milliseconds ahead of or behind video. The probe detects variable_frame_rate_suspected, but detection isn't correction. The only fix is a full re-encode with CFR enforcement, which is a deliberate choice the agent must make when it sees VFR footage in the probe output.

GOP-boundary snapping. When a cut is requested at 00:01.500 but the nearest keyframe is at 00:00.000 — common with a 2-second GOP on screen recordings — the stream-copy path cuts at the keyframe, producing either frozen initial frames or dropping the first second and a half of content. The result looks like a dead frame or a jarring jump-cut. The re-encode path with --accurate decodes from the keyframe and trims at the sample level, delivering the frame-exact cut the user asked for. The decision point is whether frame precision matters more than preservation of the original encode. For a rough cut of a talking-head interview, GOP snapping might be acceptable. For a music video synced to a beat, it isn't.

sequenceDiagram
    participant Agent as AI Agent
    participant Probe as probe.py
    participant CutLossless as cut.py (-c copy)
    participant CutAccurate as cut.py (--accurate)

    Agent->>Probe: Inspect source (duration, keyframe interval)
    Probe-->>Agent: VFR suspected, GOP = 2s, keyframe at 00:00.000

    Agent->>CutLossless: Cut at 00:01.500 (stream copy)
    CutLossless->>CutLossless: Snaps to nearest I-frame at 00:00.000
    Note over CutLossless: Output starts early — frozen frames for 1.5s

    Agent->>CutAccurate: Cut at 00:01.500 (re-encode)
    CutAccurate->>CutAccurate: Decodes from keyframe, trims sample-accurate
    Note over CutAccurate: Output starts exactly at 00:01.500

    Note over CutLossless, CutAccurate: VFR desync compounds across 3+ cuts —<br/>timestamp drift accumulates, audio/video drift

Concat heterogeneity. Joining an iPhone vertical clip with a desktop recording — one 44.1kHz stereo, the other 48kHz mono, possibly different pixel formats — produces a corrupted container when using the concat demuxer with -c copy. FFmpeg doesn't fail loudly here; it produces a file that plays but has broken audio or visual glitches at the junction. The mitigation is unified intermediate normalization — resample, pad/crop, and unify pixel formats before concatenation. That's a re-encode, and it's unavoidable when inputs differ.

Silent stream discarding. Operations targeting audio cleanup or export can silently drop secondary streams — subtitle tracks, alternate audio languages, timecode data, GoPro telemetry — unless explicit -map 0 handling is in the script templates. A cut that preserves the main video and audio but discards the embedded subtitles is a partial deliverable that looks complete. The verification step is where this surface, but you need stream-level checks, not just file-existence checks.

Intermediate disk blowup. A pipeline of Color Correction → Cut → Reframe → Silence Cut → Text Overlay writing CRF 18 intermediates at each step can consume tens of gigabytes on long-form footage before the final export. That's not a bug — it's the natural cost of daisy-chained re-encodes. The production habit is to use scratch filespaces that get cleaned up automatically, and to prefer single-pass filtergraphs for complex edits rather than chaining lossy scripts.

Where the Tool Must Be Paired: The Missing Neural Layer and the Environment Variability Problem

There's a capability boundary that the skill's documentation doesn't pretend to cross: FFmpeg doesn't transcribe audio. The skill can burn and style .srt and .ass files — karaoke overlays, timed text, styled captions — but it cannot generate subtitles from speech. When a user says "generate subtitles from this video," the agent needs an external neural pipeline upstream to produce timestamps before the skill can render them. In practice, that's a local Whisper invocation that outputs an .srt, which then feeds into the skill's subtitle-burning tool. The pairing is clean: Whisper handles speech-to-text, the skill handles the media-engineering and platform-compliance of rendering captions.

The second constraint is environmental, and it's more insidious. FFmpeg binaries vary wildly across package managers and platforms based on compile-time flags. A Homebrew build might have --enable-libass and --enable-libfontconfig for subtitle burning; a distro package might not. zscale — critical for color-managed HDR-to-SDR tonemapping — is optional and frequently absent from minimal builds. The skill depends on these capabilities existing, but the dependency isn't checked at install time.

This is where version drift and capability detection interact. The v0.9.1 changelog noted that FFmpeg's own CLI output — -filters, -encoders formatting — changes across major releases, which breaks brittle capability scraping scripts. A doctor check at agent boot is the mitigation: verify that the local binary has the required features (libass for subtitles, zscale for tonemapping, libx264 or hardware encoders for re-encode), and fail fast with a clear message if the environment is incomplete. Silent degradation is the worst-case outcome — an HDR conversion that outputs washed-out SDR because zscale was missing and the filter chain silently fell back to a naive conversion.

The production posture I'd recommend: pin a static FFmpeg build with known flags into your runtime environment, run the doctor check at startup, and treat the tool as a component that assumes a specific minimum capability set rather than gracefully degrading. Like all good guardrails, it works best when you've already removed the environmental ambiguity it can't resolve.

Positioning This Against Vanilla Agent Shell Calls, Remotion, and Cloud APIs

The alternatives to this skill aren't just different tools — they're different philosophies about where the intelligence should live.

Vanilla agent shell calls are the baseline, and they're actively dangerous. An LLM generating a raw ffmpeg bash one-liner from a natural language prompt will hallucinate parameters it can't verify, assume media properties it never probed, and default to libx264 re-encode for operations that could be lossless stream copies. The shell-injection and quote-escaping failures on complex filtergraphs are the tip of the iceberg; the deeper problem is that the agent treats a non-error exit code as proof of success. A washed-out, desynced, incorrectly-framed file that exits 0 is a silent failure that looks like a win. This skill's four-step lifecycle exists precisely to eliminate that class of hallucination — the agent can't skip probe even if it's confident, and it can't report success until verify has checked the actual output.

Code-first frameworks like Remotion and MoviePy occupy the opposite end of the spectrum. They offer pixel-perfect control and determinism, but they require the agent to write and debug full code scripts. For kinetic typography, complex motion templates, and multi-layered compositions with keyframe-based transforms, Remotion's React/Canvas/WebGL architecture is genuinely superior. But the runtime cost is heavy — Node + Chromium, or a Python virtual environment with OpenCV and friends — and the render speed for standard operations is slower than FFmpeg's optimized C codecs. This skill doesn't pretend to be a timeline compositor; if you need a title sequence with keyframed masks and per-layer opacity envelopes, reach for Remotion.

Cloud APIs (Shotstack, Creatomate, Editframe) are elastic and scalable — you can spin up a hundred parallel renders without local hardware constraints. But every render minute is metered, multi-gigabyte raw footage must be uploaded before processing, and privacy is a real concern when you're shipping private source material to a third-party endpoint. For high-throughput batch generation on a fixed timeline, the economics might work. For a single agent editing local footage, the latency of upload/download and the per-minute billing are counterproductive.

Where does this skill sit? Right in the gap between these three. It's a constrained control plane that gives the agent guardrails over FFmpeg's raw power, without the heavy dependencies of code frameworks or the cost and privacy tradeoffs of cloud APIs. The niche dominance is specific: local, free, zero-dependency, and specialized for the media-engineering edge cases that LLMs routinely fumble — VFR drift, HLG tonemapping, EBU R128 loudness targeting, platform-compliance validation.

flowchart TD
    A[Need to process video/audio?] --> B{What's the primary requirement?}

    B -->|Pixel-perfect motion graphics<br/>complex keyframe composition| C[Code-first frameworks<br/>Remotion / MoviePy]
    B -->|Elastic scale, no local compute<br/>or hardware constraints| D[Cloud APIs<br/>Shotstack / Creatomate]
    B -->|Privacy-critical, zero external<br/>dependencies, local footage| E{Need agent with guardrails?}

    E -->|Yes: LLM-driven editing with<br/>probe → edit → check → verify| F[ffmpeg-skill<br/>constrained control plane]
    E -->|No: single-shot, human-in-loop<br/>one-off command| G[Raw shell / direct FFmpeg command]

    C -->|Heavy deps, slower renders<br/>for standard ops| H[Tradeoff]
    D -->|Metered billing, privacy risk<br/>upload latency| H
    F -->|Local, free, good for<br/>media-engineering edge cases| H

Resources

• GitHub - kajisho5/ffmpeg-skill · GitHub
• raw.githubusercontent.com
• raw.githubusercontent.com