Shot Composer Deep Dive: Browser-Based 3D Blocking with an MCP Spine

What Shot Composer Actually Does

Shot Composer is a browser-based 3D previsualization engine that runs entirely client-side. No install, no account, no backend. You open it, you get a WebGL viewport (Three.js under the hood), and you start blocking shots the way a director would think about them — not the way a 3D artist would.

The core abstraction set is cinematography-first. Instead of exposing a generic viewport with raw camera transforms, it gives you presets built around standard film grammar: Over-the-Shoulder (OTS), Medium Close-Up (MCU), rule-of-thirds framing, and camera elevation controls. These aren't cosmetic labels over arbitrary coordinates — they encode relative spatial relationships. An OTS shot isn't just "camera at position X looking at Y"; it's a camera placed behind and slightly to the side of one character, framing another character across the shoulder. That's the entire point. Natural language is terrible at specifying that kind of spatial relationship, and this tool exists to bridge that gap.

Character posing works with stylized mannequins posed via joint rotations. You're not sculpting a realistic figure — you're blocking human anatomy well enough to serve as a spatial reference. The tool also supports motion keyframes across a timeline, so you can animate camera and object positions over time.

Everything persists to localStorage. Scene graphs, poses, keyframes — all of it lives in the browser's local storage quota, which is typically capped around 5MB per origin. That's fine for a handful of simple scenes and a problem if you start storing complex multi-character setups or embedded assets.

Here's the critical distinction: Shot Composer does not generate video or images. It produces spatial references. The output is a scene layout — camera angles, character positions, blocking — that you export as a JSON scene file or a rendered screenshot and feed into an external diffusion pipeline. That means ControlNet with depth maps or OpenPose rigs for Stable Diffusion, image-to-image workflows with Midjourney, or reference frames for Runway. The tool is the previz step, not the renderer. Understanding that division of labor is key to using it well — and to judging where it fits against the heavier alternatives.

The MCP Bridge: How It Talks to Agents

The interesting part isn't the 3D scene builder — it's that the whole thing is exposed over the Model Context Protocol, which means an AI coding agent can construct or modify scenes programmatically. The MCP server exposes tools for scene creation, camera positioning, character posing, and even code modification of the application itself. An agent running in Claude Code or Codex can call these tools to build a previz shot, tweak framing, or extend the tool's own feature set.

The architecture is worth being precise about because it has a structural gotcha baked in. The MCP server runs as a separate Node or Python process. The frontend is a Vite-based TypeScript app running in a browser tab with its own WebGL context, DOM canvas, and window object. These are disjoint runtimes. The MCP server has no native access to the browser — no shared memory, no direct canvas manipulation. Something has to bridge them.

The bridge in practice is a WebSocket relay or a CDP (Chrome DevTools Protocol) connection. The MCP process receives a tool call from the agent, translates it into a command, forwards it over the WebSocket to the browser frontend, and the frontend applies it to the live Three.js scene graph. The browser then sends back acknowledgements and scene state snapshots over the same channel.

flowchart LR
    A[AI Agent<br/>Claude Code / Codex] -->|MCP tool calls| B[MCP Server<br/>separate Node/Python process]
    B -->|WebSocket bridge| C[Browser Frontend<br/>Three.js / WebGL]
    C -->|ack + scene state snapshot| B
    B -->|tool results| A
    C -->|renders 3D scene| D[Canvas Viewport]
    E[Human User] -->|manual edits / export| C
    C -->|export JSON or screenshot| F[External Diffusion Pipeline<br/>ControlNet / image-to-image]

A typical agent flow looks like this: you ask your coding agent to create a scene — say, two characters with a specific camera angle. The agent calls create_scene, sets character positions, applies a camera preset, and maybe adjusts some joint rotations. The MCP server relays those commands to the browser, the scene updates live, and the agent gets back state snapshots it can use to verify or iterate. When the blocking is satisfactory, you export the scene as JSON or grab a screenshot and feed it to your diffusion model.

The failure mode I'd watch for is state desynchronization. If the browser tab isn't open, or the WebSocket connection isn't established, MCP tool calls either fail silently or persist to an isolated JSON state file that has no relationship to what's actually displayed. And if a human user is actively manipulating gizmos in the viewport while an agent sends batch mutations, you get optimistic update collisions — the agent overwrites the human's edit, or vice versa. There's no operational transform or CRDT layer here. It's a simple command-response loop, and it works well when the roles are clear: agent constructs, user reviews. It breaks when both write at the same time.

Posing Models and Camera: the Spatial Math trap

The system represents poses using Euler angles — X/Y/Z rotations applied to joints — and cameras using parametric presets built on relative spatial vectors. That's the sensible choice for a lightweight tool: Euler angles are human-interpretable, easy to serialize into JSON, and simple to hand-edit if you're an agent manipulating a scene through structured text. But Euler angles bring a specific class of problems, and they bite hard in this context.

Gimbal lock is the classic one. When two rotation axes align, you lose a degree of freedom and interpolation becomes unpredictable. An agent applying sequential rotations to a joint can push it into a singularity without any error — the math doesn't fail, it just produces something visually wrong. Related to that is the deeper issue: LLMs don't understand 3D rotational math. When an agent passes numeric rotation transforms over MCP tools, it's operating without spatial intuition. It doesn't know what a 75-degree Y-rotation looks like on a neck joint. It tends to produce hyper-extended elbows, inverted neck rotations, and self-intersecting meshes — exactly the failure modes you'd expect from a system doing forward kinematics without visual feedback.

Camera framing has the analogous problem from the other direction. High-level abstractions like OTS or MCU encode relative positioning — the camera must be behind a foreground actor, looking past them toward a background character, with the subject appropriately framed. That works beautifully when a human sets it up interactively, because they can see the viewport. An agent building the same scene from parameter values has no depth perception. It doesn't know if the camera is inside a character's head, behind a wall, or has its line of sight completely obstructed. The routing logic is sound; the spatial reasoning behind it isn't.

The mitigations are mechanical and straightforward. Joint constraint clamping — enforcing min/max rotation bounds per joint — prevents the worst hyperextension and self-intersection cases before they're committed. A raycast from the camera position toward the subject validates line-of-sight and detects geometry intersection, so you can reject obstruction-causing placements instead of shipping them. And for motion interpolation, falling back to quaternion-based slerp rather than component-wise Euler interpolation avoids the gimbal lock artifacts in keyframe animation — the coordinate drift and velocity jumps you get when interpolating Euler angles across a trajectory.

The deeper lesson here: this tool works best with a feedback loop. A human looking at the viewport, or a multimodal agent that can render frames and inspect them. Without that visual check, the spatial math trap isn't a corner case — it's the default outcome.

State Synchronization: Where the Bridge Falls Apart

The MCP bridge is a command-response loop, and it works fine as long as exactly one writer touches the scene at a time. The moment you have two, the architecture's lack of concurrency control becomes the dominant operational risk.

The disjoint state problem is structural, not incidental. The MCP server is a Node or Python process with its own memory space. The browser frontend has its own runtime that renders to a WebGL canvas and maintains the live Three.js scene graph. The WebSocket relay connects them, but it's a message pipe, not a shared memory model. When an agent calls a tool, the MCP server parses the request, mutates its internal JSON representation of the scene, and forwards the command. The browser applies it and sends back an ack. The issue is that the server's state isn't authoritative — it's a snapshot that can go stale the instant a human user starts dragging gizmos.

The race condition plays out like this: a human is scrubbing the timeline or pulling on a transform handle in the viewport. Simultaneously, an agent issues a batch mutation — reposition characters, apply a camera preset, adjust a few joint rotations. Both operations target the same scene graph nodes. There's no locking, no version number, no operational transform. Last write wins. If the agent's mutation commits after the human's drag, the human's edit is silently destroyed. If the human commits after, the agent's precisely calculated framing is now off by whatever delta the human applied.

The worst part is the feedback loop. The agent receives an ack with the scene state it believes it produced — but that ack may reflect only its own writes, not the human's concurrent edits. So the agent sees a clean result, proceeds with the next operation based on that supposedly correct state, and every subsequent mutation is mismatched against a reality it can't perceive. The scene drifts further from what either party intended.

sequenceDiagram
    participant U as Human User
    participant B as Browser Frontend
    participant M as MCP Server
    participant A as AI Agent

    U->>B: Drags gizmo (transform edit)
    A->>M: MCP tool call (batch scene mutation)
    M->>B: Forward command over WebSocket
    B->>B: Apply user's drag transform
    B->>B: Apply agent's mutation (overwrites user edit)
    B->>M: Ack with scene state (stale - reflects agent write only)
    M->>A: Tool result (false confirmation)
    A->>M: Next operation based on mismatched state
    M->>B: Forward next command
    B->>B: Mutates scene further from intended state

The mitigation has to be explicit concurrency control, and I'd reach for a simple command queue with versioning. The MCP server maintains a monotonically increasing scene version number. Every mutation carries the version it was computed against. If the version doesn't match the current scene state, the mutation is rejected with a conflict error, and the agent must re-fetch the latest state and recompute. That's much coarser than a CRDT — it doesn't attempt to merge concurrent edits, it just refuses to silently lost them — but for this workflow, explicit rejection is the right behavior. An agent can retry; a human's lost edit can't be recovered. You could also add a locking mechanism: while a human has an active drag or timeline scrub, the MCP server holds mutations until the interaction completes. That's simple to implement and covers the most common collision case.

Storage Limits and Asset Portability

Everything persists to localStorage, and that's the single most consequential design decision in the tool's persistence layer. The quota is typically capped around 5MB per origin. In practice, that means you can store a handful of simple scenes — a couple of mannequins, some camera presets, a few keyframes — before you hit the wall. Multi-character scenes with motion curves, custom meshes, or embedded GLTF assets will blow past it quickly. And the failure mode is nasty: localStorage writes fail silently. There's no exception thrown, no error dialog. The data just doesn't persist. An agent builds an elaborate scene, gets back a success response, and the next page refresh is an empty canvas.

The migration path is fairly obvious: switch to IndexedDB. It's available in every modern browser, doesn't suffer the same quota limitation, and supports structured data storage — you can store the scene graph as structured objects rather than JSON strings, which makes partial updates and incremental saves much more practical. The tradeoff is that IndexedDB's API is async and callback-heavy, so persistence logic becomes more complex. But the complexity is worth it, because the alternative is a silent data loss trap.

Feature	localStorage	IndexedDB
Capacity	~5MB per origin	Hundreds of MB (browser-dependent, often 50%+ of free disk)
API	Synchronous	Asynchronous
Data model	Strings only	Structured objects, binary blobs, indices
Performance	Blocking writes; fine for tiny payloads, janky for large ones	Non-blocking reads/writes; scales with larger scene graphs
Persistence	Survives page refresh; tied to origin	Same, plus better support for transactional writes
Migration effort	Baseline; requires rewriting persistence layer	Moderate — wrap a thin storage adapter, keep the frontend API stable

The bigger issue runs parallel to storage: there is no backend. No cloud sync, no shared server, no multi-user collaboration. Scenes exist in one browser on one machine. If Developer A's agent constructs a scene, Developer B can't see it — not without a manual JSON export and import cycle. That works for a single creator working locally, but it's a hard blocker for any team workflow where multiple agents or multiple humans need to iterate on the same blocking. The moment you need shared state, you need either a backend or a very disciplined JSON exchange protocol. The tool has no built-in mechanism for either.

What Agents Get Wrong in 3D: Empirical Failure Modes

When an LLM writes code for a 2D canvas, it's working in a space where it can reason about coordinates and patterns reasonably well. When it manipulates 3D objects through Euler angles, it's doing something fundamentally different — and it's bad at it.

The most common failure is in posing. Agents apply rotations to joints with no spatial intuition for what the result should look like. A 75-degree Y-rotation on a neck joint doesn't feel like anything to a model trained on text; it just produces a transform. The result is hyperextension — elbows bent backward past their anatomical limit, knees locked in impossible configurations, shoulders rotated completely out of the natural range. These aren't subtle errors; they're immediate, stark, and visually absurd. The deeper cause is that Euler angle manipulation via text doesn't encode any anatomical constraint. Every produced pose is plausible until it's committed to a joint that can't move that way.

The second major failure mode is camera placement. When an agent applies an OTS preset, it's relying on relative spatial vectors that it can't actually perceive. It places the camera at a parametric position that should frame a foreground actor and a background character. But without any line-of-sight check, that "correct" position can be inside the foreground character's geometry, behind a wall, or with the subject completely occluded by a scene prop. The spatial math is fine — the routing logic works — but the agent gets no feedback that the camera is buried inside a head. It proceeds as if the shot was properly composed.

Motion keyframes add a third problem: coordinate drift and pacing artifacts. When an agent interpolates camera positions across a multi-keyframe timeline, it's often interpolating each coordinate component independently. That produces erratic velocity jumps — the camera accelerates and decelerates unnaturally at keyframe boundaries. The trajectory curves look hand-drawn by someone who has never operated a camera.

The root cause is twofold: agents lack spatial reasoning, and there's no visual feedback loop. An agent writing a REST endpoint gets back structured JSON it can validate. An agent positioning a 3D camera gets back a scene graph that can't tell it the shot is blocked.

The fix is a capture_viewport_frame tool. The MCP server renders a headless frame — a canvas screenshot, or an offscreen render from a browser instance — and returns the base64-encoded image with tool results. That gives multimodal agents visual input: they can inspect the composition, see the hyperextended elbow, notice the camera is inside a wall, and iterate. It's not a complete solution — the agent still needs enough reasoning capability to interpret what it sees — but it converts a blind optimization problem into a vision-based one, and that's a massive improvement. This is the single highest-value MCP tool you can add: without it, agents are building 3D scenes with their eyes closed.

Comparing Approaches: Browser Blockers vs. Heavy 3D Suites vs. 2D Canvas Tools

The ecosystem for solving spatial ambiguity in generative video has fragmented into three distinct camps, each making a fundamentally different tradeoff. Understanding where Shot Composer sits requires seeing all three side by side, because the choice isn't about which is "best" — it's about which constraint you're willing to accept.

Criteria	Shot Composer	Blender + ControlNet	Runway / Luma Native Director Modes	PoseMy.Art
Setup friction	Zero — browser-based, no install, no account	High — full desktop install, GPU requirements, plugin configuration	Low — browser-based, account required	Zero — browser-based, no install
3D precision	Moderate — parametric camera presets, primitive geometry, stylized mannequins	Sub-millimeter — full skeletal rigs, physics, photorealistic lighting	Low — 2D keyframing and trajectory vectors, no true 3D coordinate space	Low-moderate — single or dual character posing, no camera tooling
Learning curve	Shallow — cinematography-first presets, not 3D software concepts	Steep — full 3D workflow, materials, rigging, render passes	Shallow — prompt-driven sliders and brushes	Shallow — drag-and-drop posing
MCP integration	Native — full MCP server for scene creation, camera positioning, posing	None — requires custom scripting or external process bridging	None — proprietary APIs only	None
Export path to diffusion	JSON scene export or screenshot → ControlNet (depth/OpenPose) / image-to-image	Depth/normal/OpenPose render passes → Stable Diffusion / ComfyUI ControlNet	Direct native generation — no external export needed	Pose image export → ControlNet OpenPose

The Blender pathway is the precision king, and I'd reach for it when you need actual answerability about lighting, physics, or custom rigs. But that precision comes with a brutal cost: hours of setup time, heavy hardware requirements, and a steep learning curve that assumes you're already comfortable navigating a full 3D suite. For the specific problem of "I need to block an OTS shot and quickly see if the composition works," Blender is overkill. It's the wrong tool for fast iteration on simple cinematic blocking — you'd spend more time configuring the scene than actually evaluating the shot.

Runway and Luma take the opposite approach: they collapse everything into 2D trajectory vectors and high-level sliders. You never touch a 3D coordinate space, which means the learning curve is gentle and the workflow is fast for simple camera moves. But they forfeit true 3D spatial consistency. If you need an OTS shot where a foreground actor partially occludes a background character, you can't reliably enforce that with a 2D brush. The tool might approximate it, but it won't guarantee the blocking you specified. That's untenable for anything beyond the simplest trajectory-based shots.

PoseMy.Art sits in the corner as a specialized poser. It's excellent at what it does — blocking human poses for illustration reference — but it has no camera tooling worth mentioning. No multi-camera timelines, no shot-size presets, no lens framing, and absolutely no MCP hooks. It's a one-trick pony that solves the pose problem in isolation.

Shot Composer's positioning becomes clear when you look at what it sacrifices: fidelity and generation. It runs on primitive shapes and stylized mannequins, it won't render your final frame, and it has no built-in diffusion pipeline. What it gives you in exchange is a zero-friction, MCP-first previz environment that an agent can drive programmatically. That's not a lateral trade — it's a fundamentally different axis. The tool doesn't compete with Blender on precision; it competes with the entire class of heavyweight tools on iteration speed and agent orchestration. For a director who wants to block a shot in minutes and hand the result to a diffusion model, that tradeoff is exactly right. For a technical artist building a photorealistic previsualization, it's not.

Self-Building Fragilities: When Agents Edit the Code

The open-source, agent-modifiable nature of Shot Composer is genuinely compelling in theory — an AI coding agent doesn't just construct scenes, it can extend the application itself. In practice, inviting an LLM to modify a WebGL/Three.js codebase exposes a class of failures that text-generation models are structurally poorly equipped to handle.

The most dangerous category is the rendering loop. The requestAnimationFrame cycle is the heartbeat of any Three.js application, and LLMs frequently break it in ways that don't produce immediate errors — the scene just freezes, or renders one frame and stops. The agent sees its code change compile successfully, gets a clean response, and moves on. But the canvas is dead. The failure is silent until a human looks at the viewport and sees nothing moving. The reason this bites so hard is that requestAnimationFrame invalidation is rarely caught by type checking or linting. It's a runtime behavioral issue that only manifests when the browser actually runs the loop.

Shader pipeline regressions are a close second. When an agent modifies material definitions, adds a post-processing pass, or touches the renderer configuration, it can silently break the GPU program pipeline. You get a scene that renders correctly in one viewport but produces garbage output from a different camera angle, or a tab that crashes with an WebGL context loss error. The agent has no way to know its shader change broke something — it's not getting visual feedback on what the GPU is actually doing.

Orphaned GPU resources deserve special attention because they're the most insidious. When an agent modifies scene disposal logic — changing how geometries, textures, or materials are cleaned up when a scene is torn down — it can leave textures and geometries alive in GPU memory with no JavaScript reference pointing to them. The tab doesn't crash immediately. But after enough scene creations and destructions, the browser's WebGL context exhausts its memory and the entire tab dies with a context loss. There's no error the agent can catch; the failure is delayed and dislocated from the code change that caused it.

UI state desynchronization compounds all of this. The inspector panels — Shot, Object, Motion, Pose — are typically bound to reactive state that mirrors the Three.js scene graph. When an agent modifies the underlying graph directly but doesn't update the reactive layer, or updates the reactive layer in a way that gets out of sync with the actual graph, the UI displays stale or contradictory information. An agent thinks it set a camera position to X, the graph reflects X, but the inspector shows Y. The agent's next operation reads Y and compounds the error.

My recommendation is a testing harness and strict version control, not just for the standard reasons but because they're the only mitigations that catch these failure modes. A headless browser test that creates a scene, renders a frame, and asserts specific pixel values catches shader regressions and rendering loop breakage immediately. An automated test that creates and destroys a hundred scenes and asserts no WebGL context loss catches orphaned resource leaks. And version control gives you a rollback point when an agent's "improvement" breaks the viewport. Treat the agent's code changes like you'd treat any other production change: test before merge, revert on failure. The open-source flexibility is powerful, but it amplifies both the upside and the downside of autonomous code modification.

Project Ideas: Building on Shot Composer

The architecture I'd want to see built on top of this tool is a three-layer evolution that addresses the most consequential gaps: persistence, visual feedback, and collaboration. Each is a well-scoped project on its own, and each has a specific technical trap worth planning for.

The persistence layer is the most immediately valuable. Shot Composer's reliance on localStorage with a ~5MB quota is a silent data-loss trap — agents build scenes, get success acknowledgements, and the data vanishes on refresh. A reader project here is an IndexedDB persistence layer with scene versioning. Using idb or Dexie, you'd replace the localStorage storage adapter with structured IndexedDB stores for scene graphs, character meshes, and motion keyframes. The versioning system is the clever addition: snapshot the scene before every agent mutation, maintain a history stack, and expose rollback in the UI. This connects the MCP server to the browser frontend's IndexedDB wrapper — the MCP server sends mutation commands, the frontend applies them transactionally within IndexedDB, and the version stack provides an audit trail. The watch-out is that IndexedDB's async writes can block the render loop if not debounced, especially when storing larger binary assets like GLTF meshes. You need to throttle persistence so it doesn't interleave with requestAnimationFrame frame updates.

The visual feedback tool is the highest-leverage addition. Without it, agents are building 3D scenes blind — accepting that a camera placement is "correct" because the math says so, even when the camera is inside a wall. A capture_viewport_frame MCP tool addresses this by rendering the current Three.js canvas to a data URL and returning base64 image data to the calling agent. That enables multimodal models to actually see the composition, detect clipping, notice hyperextended elbows, and iterate. Pairing this with joint constraint clamping gives agents both the feedback and the guardrails they need. The practical concerns here are real: frame capture is GPU-memory expensive and must be rate-limited, base64 payloads can grow large enough to slow down the agent's context, and the tool only works while the browser tab is open — you'd want a headless fallback for CI or automated testing environments where no human has a browser open.

The collaboration layer is the most ambitious project but arguably the most transformative. A lightweight WebSocket daemon that broadcasts scene changes across multiple browser instances, with CRDT-based merge (Yjs or Automerge) on the Three.js scene graph, would let multiple humans and multiple agents concurrently manipulate the same blocking without the optimistic-update collisions that currently plague the single-writer model. The daemon acts as a relay: each browser tab sends its scene updates, the daemon merges them using the CRDT, and forwards the merged result to all connected clients. Agents inject edits through the MCP server, which routes through the daemon. The watch-outs here are performance degradation with many simultaneous updates — CRDTs add memory overhead and latency that scales with scene complexity — and the tricky interaction between WebSocket messages and local optimistic updates, especially during concurrent gizmo drags. You'd want to verify your scene graph is small enough that the CRDT overhead stays acceptable, and test aggressively under concurrent editing conditions.

Each of these projects builds on a real architectural weakness rather than a hypothetical one, which makes them good candidates for actually shipping something useful.

Resources

• GitHub - Anujatk1999/open-media · GitHub