Code-as-Room: Thirteen Stages From a Top-Down Photo to a Render-Ready Blender Scene

The Task Contract: One Image In, an Executable Blender Scene Out

Code-as-Room (from Yixuan Yang and colleagues at Shanghai AI Laboratory, the Shanghai Innovation Institute, SUSTech, and Warwick; arXiv 2605.18451; Apache-2.0) commits to a task contract that is unusually crisp for this space: given a single top-down view image of a room, produce executable Blender code that reconstructs that room in 3D — geometry, materials, lighting, render settings included. The output is not a mesh blob and not a description of a room. It is a Python script that Blender runs, currently stage12_render/render_output.py at the end of a run directory.

I value this framing because it makes every quality claim checkable. Blender is a real interpreter with real constraints: the script either executes, errors, or produces something you can look at from any angle. There is no ambiguity of the "the output felt plausible" variety that plagues text-only evaluations. When the authors describe the system as an "MLLM-based agentic framework equipped with a structured execution harness," the harness is the part doing the quiet work — and the repository structure makes that division visible in a way most agent demos never bother to.

The requirements are modest and worth noting for anyone wanting to reproduce: Python 3.10+, Blender 3.6 or 4.x, and any OpenAI-compatible chat/VLM endpoint for the reasoning stages (the README's example configuration points at a gemini-3.1-pro-preview-thinking-class model), plus an optional image-generation endpoint for the texture stage. The pipeline depends on langchain-openai, langchain-core, openai, and pillow for the agent plumbing; Blender itself supplies bpy, bmesh, and mathutils, installed with Blender rather than pip — a distinction that sounds pedantic until you watch someone lose an evening to pip-installing a Blender shim that shadows the real one.

Why Blender Code Is the Right Output Representation

Before walking the stages, it is worth arguing the representation choice, because it is the smartest decision in the project. The obvious alternatives for image-to-3D are direct generation: predict a mesh, predict a point cloud, predict a Gaussian splat. Those representations are opaque — you cannot diff them, partially revise them, or talk about "the fourth object's material" without segmenting geometry after the fact. Blender code is the opposite: procedural, inspectable, editable, and composable. Every object has a name and a construction history in code. A wall is a function call with parameters; changing the room's width is editing a constant, not re-running inference.

Code representation also imports Blender's decades of accumulated structure — transforms, parenting, modifiers, material slots — for free. The model does not need to reinvent scene-graph semantics; it needs to emit bpy calls into a structure that already understands them. And because the artifact is code, the entire verification toolkit of software engineering applies: syntax checks before execution, static validation of API usage, patch-based repair of failures, and resumability at file granularity.

The tradeoff is completeness. The release plan says it plainly: "Code-only geometry can be insufficient for representing fine-grained small objects in downstream applications such as robotics," which is why 3D-asset retrieval checkpoints are on the roadmap. A procedurally-coded coffee mug is a cylinder with a torus handle unless the pipeline invests heavily in geometry synthesis — and the honest answer to fine-grained fidelity is retrieval or learned 3D generation, not more code. Knowing where your representation's expressiveness ends is the difference between a research demo and a product.

flowchart TD
    IMG[Single top-down room image] --> S0[Stage 0<br/>Scene classification<br/>residential vs lab routing]
    S0 --> S1[Stage 1<br/>Spatial semantic analysis<br/>scene elements + relationships]
    S1 --> S2[Stage 2<br/>Scene graph construction]
    S2 --> S3[Stage 3<br/>Base Blender code generation<br/>layout + walls + major objects]
    S3 --> S4[Stage 4<br/>Wall objects + minor placeholders]
    S4 --> S5[Stage 5<br/>Major object descriptions]
    S5 --> S6[Stage 6<br/>Detailed geometry<br/>parallel workers]
    S6 --> S7[Stage 7<br/>Surface-based small-object placement]
    S7 --> S8[Stage 8-9 optional<br/>small-object descriptions<br/>+ composite geometry]
    S8 --> S10[Stage 10<br/>Per-part PBR materials]
    S10 --> S11[Stage 11<br/>Texture generation + injection<br/>optional image endpoint]
    S11 --> S12[Stage 12<br/>Lighting + render script]
    S12 --> OUT[render_output.py<br/>run in Blender]

Thirteen Stages: Where Models Decide and Where Code Decides

The pipeline runs stages 0 through 12, and the README draws the architectural line explicitly: "LLM/VLM stages produce scene semantics, relation graphs, Blender layout code, object descriptions, detailed geometry, materials, texture prompts, and render settings. Deterministic code handles orchestration, validation, repair, memory, code integration, and several geometry/layout constraints."

That sentence is the whole design philosophy. The model does the perceptual and creative work — classifying the scene as residential versus lab, inferring that the desk sits against the north wall, describing what a cluttered bench contains. The deterministic layer does everything where probabilistic output is a liability: deciding what stage runs next, checking generated code parses, patching syntax errors, persisting memory between stages, integrating per-object code into one scene, and enforcing hard geometric constraints like "objects must rest on support surfaces, not float."

Stage 7 shows the pattern at its best. It parses Stage 6's detailed geometry, finds usable support surfaces in the scene, and places small objects grounded in the reference image on those surfaces — the deterministic surface-finder guarantees physical plausibility (nothing floats mid-air), while the model decides what objects belong there based on what the photo shows. Neither component could do the job alone: the model cannot guarantee support, the code cannot see the image.

The optional tail deserves a note: stages 8 and 9 (per-small-object descriptions and composite geometry) exist in this codebase but are explicitly "not part of the main paper pipeline," off by default, enabled with --detail-small-objects. The README is honest that they exist for clutter-heavy scenes — lab benches, kitchen counters, office desks — where primitive-shape small objects look wrong. Defaulting them off is a cost decision: each added stage is more model calls, more tokens, more surface area for a repair loop.

Prompts as Versioned Artifacts

The agent_prompt/ directory is the part I would make every agent-team lead study. Every stage's prompt is a checked-in file, and the repair prompts are separate artifacts: Stage1_task alongside Stage1_fix_template, Stage3_task alongside Stage3_fix_system, plus addenda like Stage1_task_lab_addendum and Stage1_task_residential_addendum, and a dedicated pair for rotation (Stage3_rotation_analyze, Stage3_rotation_fix). There is even a Stage_scene_classifier prompt that is just the routing stage's brain.

This is prompt engineering treated as configuration management rather than vibes. The addendum pattern in particular is how you scale a prompt system without monolithic prompts: the base spatial-analysis task stays stable, and domain-specific requirements (labs have fume hoods and benches; residences have sofas and rugs) layer on as separate files selected by the Stage 0 classification. The fix templates being separate from task templates means the repair loop's instructions can evolve independently of the generation instructions — and they will, because failure modes are discovered empirically while task requirements are designed.

The supporting utilities complete the picture: prompt_manager.py loads these artifacts, scene_classifier.py implements routing, memory.py persists stage outputs, validator.py and validators.py check structure, code_patcher.py and blender_code_syntax_fix.py repair, and composite_helpers.py assembles per-object code into composites. Every function an agentic system needs has a named module — nothing lives in a tangle inside a monolithic loop.

The Execution Harness: Generate, Validate, Repair, Execute

Stage 3 is where the system earns the word "harness." Base Blender code generation runs through a dedicated sub-pipeline (agent_utils/stage3/) with separate modules for the code-generation agent, an analyze agent, a fix agent, a code patcher, a validator, and composite helpers. The shape is a loop with typed roles rather than one prompt shouting "fix it" at itself:

sequenceDiagram
    participant CG as Code-gen agent
    participant V as Validator (deterministic)
    participant P as Patch/fix agent
    participant B as Blender (bpy)

    CG->>V: proposed Blender code
    V->>V: syntax check + structural validation
    alt valid
        V->>B: execute via bpy
        B-->>V: scene built or runtime error
    else invalid
        V->>P: errors + context from memory
        P->>CG: patched code for re-validation
    end
    B-->>CG: executed scene state
    Note over CG,B: rotation analysis/fix pair handles<br/>orientation errors found post-hoc

The rotation-specific analyze/fix pair (Stage3_rotation_analyze, Stage3_rotation_fix) is my favorite detail, because it shows the system growing organically against real failures. Rotation errors — a chair facing the wall, a monitor facing away from the desk — pass every syntax and structural check and only become visible when someone looks at the rendered scene or the scene graph. The team's response was not to hope the base prompt handles rotation better; it was to add a dedicated analysis stage and a dedicated repair stage that reason about orientation specifically. That is the empirical loop every production agent system eventually needs: new failure class, new typed check, new targeted repair.

Memory (agent_memory.jsonl in every run directory) ties the loop together — each stage reads prior stages' outputs from persisted memory rather than re-deriving them, which is what makes the whole pipeline resumable and debuggable. run_pipeline.py --status --run-dir <dir> shows memory state; --clear-stage stage7_small_objects --run-dir <dir> invalidates exactly one stage's output so you can rerun from there without paying for stages 0 through 6 again.

Resumable Runs: The Operational Design Most Demos Skip

Every run writes to an isolated, timestamped directory — run_YYYYMMDD_HHMMSS_<image>/ — containing agent_memory.jsonl, run_config.json, and one subdirectory per stage with its outputs: stage1/stage1_output.json, stage2/stage2_skeleton.json, stage3/_layout.json, through to stage12_render/render_output.py. The example directory in the repo ships a complete real run (run_20260521_104358_example1) so you can see the exact artifact shapes before running anything.

This is unglamorous and decisive. Because each stage's output is a file, the pipeline is: inspectable (you can read Stage 1's scene semantics to understand why Stage 6 generated a weird table), resumable (--run-dir <dir> --start 10 --end 12 runs only materials, textures, and render against existing geometry), and cheap to iterate on (regenerating one stage costs one stage's tokens, not the pipeline's). The --list-runs command turns the output directory into a history you can query. Compare this to agent demos where the entire state lives in a chat context window and a crash means starting over — the difference is not convenience, it is whether the system can be operated at all.

The final artifacts list reads like a contract for downstream tooling: the render script, the texture manifest (texture_manifest.json) alongside generated maps, the material configuration, and small_objects.json with placement data. Each is machine-readable and each is separately useful — a game-asset pipeline might want stage 7's placement data without ever touching Blender.

Batch Economics: Two Parallelism Flags That Mean Different Things

The batch runner (batch_run_pipeline.py) processes a folder of images and exposes a pair of flags that are easy to confuse and expensive to confuse:

Flag	Controls	Failure mode if misread
--parallel	Internal Stage 6 geometry worker count within one pipeline run	Setting it high hoping to speed up the batch does nothing across images
--max-concurrent	How many images/pipelines run at the same time	Setting it to 16 on a laptop spawns 16 pipelines, each calling LLM APIs and spawning Blender
--model-tag	Filesystem-safe folder name in the output bucket (<output-root>/<model-tag>/<label>/...)	Using long raw model names produces unwieldy paths; the bucket design exists to compare models on identical inputs
--label	Dataset or image-class folder under the model tag	Colliding labels mix datasets inside one comparison bucket
--stop-on-error	Halt batch after first failure	Default continue-on-error means silent per-image failures — read the logs
--dry-run	Preview the batch plan without executing	Skipping it on a large folder is how token bills surprise people

The output organization — by model tag, then dataset label, then run — reveals the intended use: controlled comparisons of model stacks on identical image sets. That connects to the repo's open benchmark item on the release plan: benchmark scaling is listed as resource-intensive, and the batch tooling is the substrate for it. The operational advice in the README is refreshingly concrete — on a laptop, --max-concurrent 4 to 6 "is usually safer because each pipeline may call LLM APIs and spawn Blender" — which is the kind of sentence written by someone who has actually watched a machine die at 16.

Scene-Type Routing and Its Honest Constraint

Stage 0 classifies the scene, and the classification changes downstream behavior through the addendum prompts — lab scenes get bench-and-equipment priors, residential scenes get furniture priors. You can force it: --scene-type lab or --scene-type residential overrides classification, which matters in production when a classifier misfires on an ambiguous image (a studio apartment with a workbench corner) and you know the truth. Wall treatment has its own dial (--wall-intensity subtle|bold|mural_like), a small but real acknowledgment that wall appearance drives perceived quality of a room render more than almost anything else.

The honest constraint is stated in the release plan: the pipeline "works best on rectangular or near-rectangular rooms," with irregular layouts on the roadmap, and single-room reconstruction only — whole floor plans are future work. Top-down photos of L-shaped rooms, curved walls, or multi-room spaces will hit the geometric assumptions baked into layout code. For anyone evaluating this for a real product, that constraint is the first thing to test against your actual input distribution, because it is exactly the kind of assumption that demos (curated square bedrooms) never surface.

What the Release Plan Admits

Release plans are where research repos tell the truth about their weaknesses, and this one is unusually candid on three points.

First, the planned web editor exists "to reduce both time and token cost compared with post-hoc correction inside the agent loop" — a direct admission that fixing a generated scene by talking to the agent is the expensive path, and a structured editor synchronized between the scene, the underlying code, and Blender is the cheap one. That is the same lesson every code-generation system learns: the repair loop is for the machine's errors, not for human preference iteration, and human iteration needs different (cheaper, deterministic) tooling.

Second, the asset-retrieval roadmap concedes the code-only representation's ceiling for fine-grained small objects, with robotics named as the downstream application that needs them. Third, the benchmark item acknowledges that "building and scaling the benchmark requires substantial time and token cost" — the evaluation is the bottleneck, which anyone who has built evals for generative pipelines will recognize as the truest sentence in the README.

For team leads, the transferable lesson from these three admissions: know which costs are token costs, which are asset costs, and which are evaluation costs, and budget them separately. Code-as-Room's architecture already reflects that accounting — stages are separable precisely so you can pay for detail only where your application needs it.

The Pattern Worth Stealing (Even If You Never Touch Blender)

Strip the 3D away and Code-as-Room is a reference implementation of a general architecture: perception models that turn unstructured input into typed intermediate artifacts, synthesis models that turn those artifacts into executable code, and a deterministic harness that validates, repairs, persists, and orchestrates. The stage decomposition — classify, understand, graph, generate, refine, detail, dress, render — maps onto any input-to-artifact pipeline: document-to-database, screenshot-to-frontend, sensor-log-to-dashboard.

The two decisions I would replicate first: make every stage's output a file on disk with a schema (memory becomes infrastructure instead of a context window), and split repair into typed paths (syntax repair, rotation repair) instead of one generic "try again" prompt. Both decisions cost nothing to adopt and both are the difference between an agent demo that works once on camera and a pipeline that runs overnight on a folder of inputs. The 229-star repo with its checked-in prompts, its isolated run directories, and its two-flags-that-mean-different-things batch runner is, more than anything else, a working answer to the question "what does operating an agentic code-generation system actually look like" — and the answer is: like a compiler with opinions, not like a chat.

Resources

• GitHub - YxuanAr/Code-as-Room
• Code-as-Room project page
• arXiv 2605.18451 - Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis