Code-Only 3D UIs: A Practitioner's Guide to Procedural Three.js Generation with Astra

What Actually Happens When You Ask Astra for a 3D UI

The first thing to understand is that when I say "Astra," I mean the coding agent—the multimodal model that takes inputs and writes code—not Google's Project Astra, which is a different thing entirely. The distinction matters because the workflow I'm describing is fundamentally about code generation, not about running a live multimodal assistant that watches your screen.

Here's what actually happens. You give the agent a video showing an interaction, a URL to a page whose visual style you want replicated, or just a text description of the UI you need. The model ingests that multimodal input and decomposes it visually—parsing spatial layout, hierarchy, color palettes, and motion patterns. Then it synthesizes the entire scene programmatically: not by fetching a mesh file, but by writing Three.js code that constructs geometry from mathematical primitives.

Let me make this concrete. If I ask for a button with a torus knot and a gradient shader, the agent doesn't look up a "torus knot button" asset. It writes THREE.TorusKnotGeometry with specific radius and tube parameters, creates a THREE.ShaderMaterial with a fragment shader that interpolates between two colors based on a varying UV coordinate, and builds a gradient falloff. It then sets up a THREE.Raycaster bound to pointer events, so hovering over the knot triggers a visual response. The generated code includes scene initialization, camera positioning, renderer configuration, the geometry math, the event listeners, and the animation loop that keeps everything alive.

The contrast with a traditional asset pipeline is stark. The typical approach involves loading a .glb or .gltf file via THREE.GLTFLoader, hosting that file somewhere, and either calling an MCP tool to fetch it or integrating with an asset library. That's network latency, storage overhead, and dependency on external services. With procedural generation, none of that exists. The entire thing runs from a single block of code, instantly executable in the browser.

There's a deeper implication here. Because the 3D model is the code, you can customize it with text prompts. Want the button's knot to be smaller and the colors more saturated? You ask, and the agent edits the parameters. No asset editing software, no re-exporting, no pipeline. The whole point is that the geometry is a function you can tweak, not a file you're stuck with.

Slicing the Architecture: How the Zero-Asset Pipeline Works

The pipeline has four distinct phases, and understanding each one matters because that's where you'll find both the elegance and the failure modes.

flowchart TD
    A[Multimodal Input: Video / URL / Text] --> B[Procedural Decomposition]
    B --> C1[Extract Spatial Layout]
    B --> C2[Extract Materials & Colors]
    B --> C3[Extract Motion & Animation Patterns]
    C1 --> D[Code Generation: Three.js or R3F Script]
    C2 --> D
    C3 --> D
    D --> E[Browser Runtime]
    E --> F1[Scene Graph Construction]
    E --> F2[Shader Compilation]
    E --> F3[Raycaster Binding]
    E --> F4[Animation Loop via requestAnimationFrame]
    F1 --> G[Zero Asset Downloads]
    F2 --> G
    F3 --> G
    F4 --> G
    G --> H[Interactive 3D UI in Browser]
    H --> I[User Edits via Text Prompt]
    I --> D

Input parsing. The agent receives whatever you throw at it—a screen recording of a dashboard animation, a URL to a landing page, a paragraph describing a product configurator. The multimodal capacity means it can read motion from video frames and infer structure from screenshots.

Procedural decomposition. This is the critical step. The model breaks the visual into spatial hierarchy (what sits where), materials (colors, reflectivity, transparency), and motion (rotation speeds, easing curves, trigger conditions). It maps these onto Three.js primitives and mathematical functions rather than attempting to locate a mesh file that matches.

Code generation. The output is standalone JavaScript or TypeScript—vanilla Three.js or a React Three Fiber wrapper. It initializes scene, camera, and renderer; instantiates buffer geometries with vertex and face indexing; sets up event listeners for mouse movement, clicks, and hover raycasting. This is where procedural textures come in: instead of loading a PNG for a wood grain or a noise pattern, the agent generates a canvas texture from a noise algorithm, or writes a shader that computes the pattern per-pixel. No external image files, no texture downloads.

Runtime execution. The scene graph gets built, and requestAnimationFrame drives the loop. Raycasting binds interaction to the geometry—you hover over a procedural mesh and the pointer event hits it because the raycaster intersects mathematically with the same geometry runtime that creates it. The feedback loop for edits is the power move: you prompt "rotate the left panel 15 degrees and make the shader more emissive," and the agent regenerates the code with those parameters changed.

The freedom from MCP servers, storage buckets, and loaders is significant. There's no tool call to fetch an asset, no API bridge to a generation service, no hosting overhead. The entire scene is computed client-side. That's why this paradigm works so well for agentic workflows—the agent can continuously refactor the 3D properties via code edits without waiting on external services.

What Procedural Code Can and Cannot Realistically Generate

Let me be blunt about boundaries, because this is where I've seen projects go sideways.

What works: Geometric, abstract, mechanical, and UI-oriented constructs. A car made from extruded boxes and cylinders—absolutely doable. A dashboard with sphere-based gauges and needle indicators—fine. A stylized robot with articulated arms from primitives—yes. These are all cases where the object's structure maps cleanly onto primitive composition, and the visual result is supposed to look stylized or mechanical anyway. The math is tractable.

What fails: Organic, photorealistic detail. Human faces, creatures with flowing fur, cloth simulation with realistic drape, highly detailed sculpted environments. The reason is topological flow. Primitives are closed shapes with simple topology—a box has six faces, a sphere has concentric rings of vertices. When you try to assemble dozens of these to approximate a face, you hit the ceiling: you can't blend the surfaces continuously, you can't achieve the fine curvature of a cheekbone or an eyelid. The seams show, and the result looks like a robot made of tubes, not a human.

Procedural UV generation is the second nail in the coffin. When code assigns BufferAttribute values for uv and normal, it rarely accounts for the distortion that happens when a flat texture wraps around a complex curved surface. The result is stretching, seam misalignment, and artifacts along curved regions—things you'd only notice once the object rotates. The agent can't see the render, so it doesn't know the texture looks wrong until someone runs it.

Here's the comparison in practical terms:

Visual Complexity Dimension	Pure Procedural Code	Generative Mesh Pipelines (Meshy, Tripo3D)	Visual Web 3D Tools (Spline)	Pure GLSL / SDF Shaders
Organic Details	Low—primitive composition breaks down on topology flow and UV mapping	High—dense polygonal meshes with UV maps and PBR textures handle complex anatomy	Medium-high—polished stylized rendering with baked lighting	High—SDFs mathematically define continuous surfaces, but with a distinct shader aesthetic
Geometric Constructs	High—excellent for mechanical, abstract, and UI elements	Low—requires 3D software or mesh tools to modify output	Medium—tied to Spline's UI and parameter exposure	Medium—math-heavy, hard for general agents to refactor
Code Customizability	High—full programmatic control over variables, materials, vertices	Low—binary asset, needs re-generation or 3D editing	Medium—proprietary platform constraints	Medium—compact but requires shader expertise

The strategic takeaway: if your application needs a hero animation with an abstract, geometric aesthetic, procedural code wins—instant, fully customizable, zero overhead. If you need a photorealistic product render with organic curves, you need the asset pipeline. The two approaches aren't competing; they're solving different problems, and you need to know which one you're in before you commit.

How Astra Stacks Up Against Claude, Codex, and Dedicated Mesh Generators

When I'm choosing a tool for a 3D UI task, the first cut is between code-only approaches and asset pipelines. Within code-only, the differentiators are subtle but consequential.

Astra's edge over Claude with React Three Fiber artifacts is the multimodal conditioning. Claude can absolutely generate R3F components—it handles declarative hierarchies and component lifecycles beautifully. But when you hand it a video of a specific interaction, it's translating visual motion into code from a text description of what it thinks is happening. Astra ingests the frames directly and maps the motion patterns to animation keyframes and easing curves. For UI translation tasks—"make my dashboard look like this URL"—that direct visual conditioning matters. The model sees the spacing, the hierarchy, the color relationships, rather than having them described to it.

Codex and Cursor take a different route. They generate vanilla Three.js scripts well, but I've found they default to importing external assets unless you explicitly prompt against it. You end up with code that calls GLTFLoader and references a model URL that doesn't exist yet, because the model's training data says that's how 3D works. Astra's procedural bias—inherited from its focus on code-only generation—means it reaches for THREE.BoxGeometry and THREE.ShaderMaterial by default rather than reaching for a file path.

Now the asset pipeline side. Meshy and Tripo3D produce genuinely impressive organic meshes—the kind of photorealistic detail procedural code simply cannot synthesize. But they impose a chain: generation API call, file download, hosting, CDN, GLTFLoader integration, and then you're binding events across a loaded scene graph you didn't write. Each link in that chain is a deployment surface. Spline is friendlier for designers—visual GUI, baked lighting, a runtime package—but you're locked into a proprietary ecosystem with limited parameter exposure for agentic refactoring.

The tradeoff matrix looks like this:

Metric	Astra Pure Code	Mesh Pipelines	Spline
Asset Overhead	Zero—instant client execution	High—multi-megabyte downloads	Medium—runtime bundle plus cloud assets
Agent Customizability	High—full control over variables, materials, vertices	Low—binary asset needs regeneration	Medium—platform parameter limits
UI Event Integration	Native—standard raycaster, pointer events, DOM overlays	Manual—traverse loaded scene graph	Built-in—visual state machine
Tooling Complexity	Self-contained—no MCP, no storage bucket, no API bridge	Complex—generation API plus file pipeline	Proprietary platform required

The strategic conclusion is clear. Procedural code is the right tool when you're building interactive widgets, mathematical visualizers, abstract hero animations, or any UI where the geometry is meant to feel stylized and you want the agent to keep refactoring it. Asset pipelines are the right tool when you need photorealistic organic detail and you're willing to accept the pipeline overhead. In my judgment, the sweet spot for code-only is broader than most teams assume—it's just that nobody tries it until they've already committed to the asset pipeline.

When It Breaks: Runtime and Production Failure Modes

Every approach has failure modes, but procedural generation has ones that are easy to miss because the code looks fine. Let me walk through the ones that actually bite.

GPU memory leaks. This is the most insidious because nothing fails immediately. Three.js WebGL objects—BufferGeometry, Material, Texture, WebGLRenderTarget—live in GPU memory, not JavaScript heap. They are not automatically garbage-collected when removed from the scene graph. When an AI-generated UI component unmounts or re-renders with updated parameters, the generated code typically creates new geometries without calling .dispose() on the old ones. If you're building an SPA with dozens of widgets each creating geometries on property updates, you'll exhaust WebGL contexts in a long session. The browser doesn't warn you until CONTEXT_LOST_WEBGL crashes the entire canvas. I've seen this happen in a demo where a dashboard updated every five seconds—the session needed a refresh after about twenty minutes.

Draw call explosion. This one you'll feel immediately. To approximate a single button using primitive composition, a generated script might stack twenty BoxGeometry meshes with three different materials. Multiply that by a full UI dashboard with forty widgets and you're at thousands of draw calls. Each unique mesh/material pair triggers a separate draw call, and the CPU-bound bottleneck will tank your frame rate below 60 FPS long before the GPU cares. Generated code rarely implements InstancedMesh or merges geometries with BufferGeometryUtils.mergeGeometries unless explicitly prompted. It compounds because the agent sees "twenty boxes" as a reasonable approach—it doesn't have the performance mental model a human engineer would bring to the scene.

Shader compilation stalls. Complex procedural materials—custom ShaderMaterial with noise functions or heavy fragment shaders—compile at runtime when the object first enters the viewport. That compilation runs on the main thread, blocking rendering and event handling. The first frame that introduces a new shader-heavy element will jank. The user sees a frozen spinner or a half-rendered scene for a few hundred milliseconds. Not catastrophic, but detectable.

Here's the lifecycle of a generated WebGL object, and where the missing disposal loop bites:

stateDiagram-v2
    [*] --> Created
    Created --> InRenderLoop: add to scene
    InRenderLoop --> Unmounted: component unmounts
    Unmounted --> MemoryLeak: no dispose() called
    MemoryLeak --> CumulativeGPUUsage: repeated unmounts
    CumulativeGPUUsage --> ContextLost: GPU memory exhausted
    ContextLost --> BlankCanvas: webglcontextlost fires
    BlankCanvas --> ContextRestored: browser restores context
    ContextRestored --> BlankCanvas: nothing rebuilds state
    MemoryLeak --> Disposed: explicit dispose() via traverse
    Disposed --> [*]

Every transition through that loop is a place where generated code typically fails. The Unmounted → MemoryLeak transition is the quiet one—nothing visually happens, so no one notices until the context dies.

The edge cases. Complex UV unwrapping for organic shapes is where procedural generation hits its ceiling; code-assigned uv and normal buffers distort textures on curved surfaces, and the agent can't see that until someone runs the render. Accessibility is a separate order of magnitude problem: a canvas-rendered 3D UI without DOM overlays is completely opaque to screen readers. Keyboard navigation doesn't work, assistive tech sees nothing. Mobile thermal throttling compounds everything—continuous requestAnimationFrame loops with fragment shaders drain batteries and trigger aggressive throttling that makes your UI feel broken on phones.

None of these are reasons to avoid the approach. They're reasons to build the guardrails before you ship.

Shaping Code to Survive in Production

The generation quality is only half the battle. The other half is making sure what gets generated doesn't destroy your runtime. Here's the engineering discipline I'd enforce on any team using AI-generated procedural 3D in production.

Resource cleanup is non-negotiable. I mandate a recursive traverse() cleanup hook on every unmount path. When a component unmounts, walk the scene graph and call .dispose() on every geometry, material, and texture. If you're using a framework wrapper, this becomes trivial—React Three Fiber already does it for you when components unmount. That's the strongest argument for standardizing on R3F: it handles the GPU memory lifecycle so the agent doesn't have to.

Framework wrappers are a force multiplier. Beyond disposal, React Three Fiber gives you DOM overlay capabilities via the Html component from @react-three/drei. That's how you solve the accessibility problem—the 3D canvas sits underneath, and you project interactive controls into DOM nodes that screen readers can see and keyboard navigation can focus. Without that overlay layer, your 3D UI is a black box to assistive technology.

Render only when something changes. frameloop="demand" is the single biggest mobile performance win available. The default behavior renders every frame at 60 FPS even when the scene is static. With demand mode, the render loop only fires when state actually changes—a user dragging a slider, a component mounting, an animation trigger. The battery savings are enormous, and the code doesn't need to change beyond the frameloop configuration.

Budget the draw calls before you merge. I integrate r3f-perf into staging and CI to enforce hard thresholds on draw call counts, triangle counts, and shader compile times. If a generated scene exceeds your budget, the agent gets a feedback message describing the violation and regenerates with instancing or geometry merging. This is the key difference between treating AI-generated code as a draft and treating it as shippable—you need automated verification, because the agent can't feel the frame drops.

Precompile shaders. Warm up the shader programs during idle time or before the component enters the viewport to avoid first-frame jank. Three.js exposes .compile() on the renderer—call it once with the scene to force shader compilation before interactions begin. It's a small addition that eliminates the most noticeable runtime hiccup.

And instancing. For repeated elements—a grid of particles, a row of identical UI widgets—instruct the agent to use InstancedMesh rather than individual meshes. One draw call handles hundreds of instances. The generated code needs that performance hint, and it needs it consistently.

These aren't optional best practices. When the code is AI-generated, nobody has manually caught the missing .dispose() call or the dozens of separate meshes for a single widget. The agent optimizes for visual correctness, not runtime health. Your production discipline is what converts a beautiful prototype into something that survives a real user session.

Interactive Code Demos: Three Projects You Can Build Today

The best way to internalize this paradigm is to build something with it. Here are three projects with clear scope, concrete components, and the failure modes I'd watch for in each.

Product configurator. Start with a rotating 3D watch—extruded case, sphere crown, a torus for the bezel. The geometry matters less than the interaction loop: color swatches in an HTML overlay that update a ShaderMaterial's uniform, hover feedback via Raycaster that highlights the component under the pointer, and OrbitControls for rotation. The agent can generate all of it from a prompt describing the watch and the UI pattern. The pitfall I'd flag is material disposal on color change. Every time you swap colors, if you're creating a new ShaderMaterial without calling .dispose() on the old one, you're leaking GPU memory with each click. The fix is trivial—dispose before reassigning—but it's exactly the kind of thing generated code misses because it renders correctly. Also, recalibrate your raycaster on window resize; pointer coordinates drift out of sync with the scene when the viewport changes, and hover feedback starts hitting the wrong objects.

Radial data dashboard. This is where the paradigm shines because the geometry is genuinely mathematical. Animated ring gauges for CPU and memory metrics: RingGeometry for the track, a shader that draws the progress arc based on a fractional uniform, and values updating from a data feed. The key insight is to update the uniform rather than re-creating the geometry on each data tick. Re-creating means new buffer allocations and potential leaks; updating a uniform is nearly free. I'd build this on React Three Fiber for lifecycle management and use frameloop="demand" so renders fire only when data changes—a dashboard that's static between data ticks shouldn't burn GPU cycles at 60 FPS. The draw call trap is real here: multiple rings, each a separate mesh, will stack up. Merge geometries or push the arc-drawing into a single shader pass. And include a DOM overlay for accessibility, because a canvas-only dashboard is silent to screen readers. Handle container resize with proper perspective scaling so the rings don't distort when the panel resizes.

Portfolio hero with particle field. An abstract animated vortex of particles with custom vertex and fragment shaders for motion and glow, plus mouse-reactive camera drift. This is the most visually impressive of the three and the one most likely to run into mobile performance walls. THREE.Points with BufferGeometry holds the vertex positions; a custom vertex shader animates them in the GPU without CPU-side updates each frame; a fragment shader adds the bloom-like glow. A simple raycaster highlights particles on hover by passing a uniform that the shader uses to brighten nearby points. The mobile problem is the particle count—tens of thousands of points updating every frame with a bloom shader will drain a phone battery in minutes and trigger thermal throttling. Keep the count modest, use frameloop="demand" so the loop only runs when the user interacts, and precompile the shaders on load to avoid the first-frame jank that comes from runtime compilation. This one also needs a deliberate accessibility fallback: a DOM-based description of the scene that renders if the canvas doesn't, since a particle vortex has zero semantic content for assistive tech.

All three are achievable with the current tooling. The difference between a demo and something production-viable is knowing the specific failure modes before you start.

Resources

• three.js docs
• docs.pmnd.rs