Scan to Structure: How Astra Turns Messy Photogrammetry into Artist-Modeled Procedural Scenes

The Core Problem: Why Raw Scan Geometry Is Useless for Editing

I've spent enough time staring at photogrammetry outputs in DCC tools to have a visceral reaction to them. The raw mesh that comes out of RealityCapture, Metashape, or a Gaussian-splat extraction is, structurally speaking, a catastrophe. It's a dense soup of triangles—hundreds of thousands of them for a single building facade—with no coherence at the scale an artist works at. There are no edge loops, no quad flow, no UV islands that respect material boundaries. Everything is triangulated, and the triangulation is determined by where the camera happened to be, not by the geometry of the object itself.

The deeper problem is manifoldness. Raw scans are riddled with non-manifold edges: places where more than two faces share an edge, holes where occluded geometry was never captured, and self-intersections where the surface folds back on itself. Any artist who has tried to select a wall in a scan mesh knows the pain—the selection bleeds through to the adjacent roof because there's no semantic boundary, just millions of connected triangles.

This isn't just aesthetic discomfort. It blocks entire production workflows. You cannot rig a scan mesh because there's no edge loop structure to support deformation. You cannot generate meaningful LODs because the mesh density is uniform and uninformative—a featureless wall has the same triangle count as a complex roofline. Dynamic textures are impossible because UVs are either absent or baked from the original capture with wildly inconsistent texel density; some surfaces get ten times more pixels per unit area than others. And forget about artist editing: the moment you try to nudge a wall, you're dragging a million-triangle boundary that cascades through the entire surface.

Contrast that with what an artist produces. Hand-modeled geometry has a semantic hierarchy: walls are separate objects with clean quad edge loops, windows are parametric elements that can be replaced by changing a dimension, and UVs are laid out deliberately with uniform texel density across same-scale surfaces. That's not just cleaner—it's editable. Change a wall dimension and the whole assembly updates coherently. Delete a window and you're left with a clean hole, not a ragged tear.

The gap between scan geometry and artist geometry is not a matter of decimation or retopology. It's a difference in information content. The scan knows where surfaces are. The artist knows what those surfaces mean. That's the pipeline we need.

Astra's Pipeline Breakdown: From Point Cloud to Parameterized Assemblies

The pipeline that Astra represents has a fundamentally different posture than retopology tools. Because it isn't trying to clean up the mesh—it's trying to understand it. The four stages each convert noise into meaning, and that meaning accumulates.

Stage one is semantic segmentation. The raw scan—whether it's a point cloud, a photogrammetry mesh, or a LiDAR surface—is passed through a deep learning network that assigns every surface region to a semantic class. Roof planes, walls, windows, terrain, vegetation, structural pillars. This isn't just classification; it's the first moment where the pipeline decides what the geometry means. The output is a labeled point cloud or a mesh with per-face semantic IDs. This is the stage where misclassifications happen—a green roof that gets tagged as canopy, a retaining wall that reads as terrain—and those errors cascade downstream unless the confidence scores are surfaced.

Stage two is vectorization. Now the pipeline has to condense those labeled regions into clean parametric primitives. For roofs, it fits planes and extracts height values, determining whether it's a gable, hip, or flat roof from the inferred geometry. For walls, it extracts 2D footprints—polygons that describe the building's outline at ground level. For windows, it detects rectangular openings and records their position, size, and orientation. The output is not a mesh; it's a set of 2D polygons, height measurements, and roof-type classifications that describe the building abstractly. This is where the pipeline stops thinking of the scene as a surface and starts thinking of it as a set of rules.

Stage three is shape grammar synthesis. Here's where the actual "artist modeling" happens. The extracted footprints, wall heights, and roof types feed into a procedural rule set—essentially a CGA-style grammar or a node graph—that generates clean geometry from scratch. Walls are extruded from the footprints with proper quad topology. Windows are instantiated as parametric assets at the detected positions. Roofs are generated according to their classified type, with the correct pitch and overhang. The output is a quad-dominant, UV-unwrapped mesh where each semantic component is a separate entity with its own edge loops and clean boundaries. This is the stage that produces something an artist could actually open and work with.

Stage four is material baking. The clean procedural geometry needs to look like the real thing, so the pipeline projects textures from the original scan onto the new surface. But that's not a straight projection—the scan textures carry baked-in shadows, specular highlights, and ambient occlusion from the capture environment. So the pipeline runs a de-lighting pass first, extracting pure albedo and normal information, and then projects those onto the newly unwrapped UVs. The result is a procedural assembly with clean UVs, consistent texel density, and appearance that matches the original capture without carrying its lighting contamination.

This is the critical distinction from retopology. Quad Remesher gives you a cleaner mesh—but it's still a single connected surface with no semantic information. The output of Astra's pipeline is a hierarchy: walls, windows, roofs, and terrain as separate, parameterized components that can be edited, swapped, or animated independently. It's not just cosmetic cleaning. It's a transformation in what the data actually is.

flowchart TD
    A[Raw Scan: point cloud / photogrammetry mesh / LiDAR] --> B[Semantic Segmentation]
    B -->|roof / wall / window / terrain classes| C[Vectorization]
    C -->|2D footprints, wall height, roof type| D[Shape Grammar Synthesis]
    D -->|wall extrusion, window placement, roof generation| E[Material Baking]
    E -->|UV projection, de-lighted albedo| F[Clean Procedural Assembly]
    F -->|labeled components| G[Editable quad-dominant mesh with semantic hierarchy]

How Astra Compares to Existing Solutions: Retopology, CityEngine, and Generative CAD

The landscape of tools that touch this problem space is crowded, but they occupy very different positions. Understanding those positions is the only way to see what Astra actually adds.

Quad Remesher and Instant Meshes live at the "direct geometry" end. They take a messy mesh and solve for a cleaner quad flow using direction fields and field-aligned quadrangulation. The output is a quad-dominant mesh with proper edge loops, and they work on any arbitrary input. But the result is a frozen mesh. It's cleaner to subdivide and sculpt on, but it carries no semantic information—there's no wall object, no window component, no understanding of what the surface represents. You've cleaned the triangle soup, but you haven't transformed it into an editable assembly.

Esri CityEngine takes the opposite approach. It's pure grammar-driven procedural modeling using CGA rules, and it's the industry standard for urban planning. It produces genuinely parametric buildings with full editability—change a rule and the whole city updates. But the input is clean GIS footprints, not messy 3D scans. It cannot parse a dense photogrammetry mesh and extract the structure from it. You have to hand it clean footprints and manually author the rules. It's a powerful generator, but it's not a converter.

Blackshark.ai, the engine behind Microsoft Flight Simulator, comes closest to what Astra does. It runs AI feature extraction over aerial and satellite imagery, detects building footprints and roof types from 2D data, and then runs shape grammars to generate clean building meshes at planetary scale. The mechanism is nearly identical to Astra's pipeline. But the input domain is fundamentally different: it's working from 2D imagery, not from dense 3D scans. That means it's constrained to buildings and terrain on a global scale—it isn't trying to understand an arbitrary scan of a mechanical part or a complex organic prop. It also idealizes the geometry, replacing real-world imperfections with standard parameterized shapes.

Generative reverse-CAD approaches—CSG-Stump, Point2CAD, and the emerging text-to-CAD systems—take yet another angle. They try to invert point clouds into constructive solid geometry or B-Rep trees, producing genuine parametric CAD history where you can edit a sketch or adjust a boolean operation. The editability is deeper than anything else here, since you're working with real parametric history rather than just a set of generated components. But these systems are limited to geometric primitives and mechanical parts. They fail on organic shapes and noisy architectural facades, where the geometry doesn't decompose cleanly into booleans.

I can lay this out side by side.

Approach	Output Type	Input Requirements	Editability	Domain Scope
Quad Remesher / Instant Meshes	Quad-dominant frozen mesh	Any arbitrary messy mesh	Nil—static mesh only	Generic, any surface
CityEngine (CGA)	Parametric city model	Clean GIS footprints, manual rules	High—parameter-driven	Architecture, urban planning
Blackshark.ai	Clean building meshes	2D aerial/satellite imagery	Moderate—grammar-driven	Buildings and terrain, planetary scale
Generative CAD (CSG-Stump, Point2CAD)	Editable CSG/B-Rep trees	Dense point clouds of parts	Very high—full parametric history	Mechanical parts, primitive-constrained
Astra	Procedural assembly with semantic components	Arbitrary dense 3D scans (photogrammetry, LiDAR, splats)	High—semantic separation, multi-level	General objects and scenes

What Astra occupies is the gap that none of these fill. It takes arbitrary dense 3D scans—not aerial imagery, not clean footprints, not primitive-constrained parts—and produces a semantic hierarchy of editable components. That's the domain-specific strength: it generalizes beyond the architectural constraints of Blackshark.ai and the primitive constraints of reverse-CAD, while delivering the structural editability that pure retopology cannot. The tradeoff, of course, is fidelity. The output is inevitably idealized; when the pipeline replaces a worn doorway with a standard parameterized door, the micro-surface imperfections vanish. You trade the exact texture of reality for the ability to change things at all.

Where Astra Breaks: Failure Modes in Segmentation and Grammar

The pipeline sounds clean on paper, but the failure modes are where the real engineering lives. I've seen enough of these systems to know that each stage has a characteristic way it breaks, and those breaks tend to cascade.

Segmentation misclassification is the first trap. When vegetation overhangs a structure—a tree canopy draping over a roofline—the segmenter frequently tags the roof plane as canopy. That's not a cosmetic error; it means the downstream grammar never extrudes that roof, and you end up with a building whose top floor is missing, replaced by a tree mesh that visually intersects the facade. Green roofs are worse. A terrace garden that's structurally a roof but visually reads as vegetation gets classified as terrain, and suddenly you have grass extruding through the living space. The observable result isn't subtle: it's a building with a hole in the wrong place.

Concave footprints are the second failure I watch for. Courtyards, stepped setbacks, and cantilevered floors are hostile to boundary-extraction algorithms. The vectorizer either collapses a concave alcove into a convex hull—clipping an entire interior courtyard out of existence—or produces self-intersecting 2D polygons that break the grammar downstream. I've seen this manifest as buildings with missing interior spaces: the courtyard that should be there is simply gone, replaced by solid wall.

Non-standard architecture breaks the grammar at the next stage. Geodesic domes, curved curtain walls, Gothic spires—these don't decompose into the rectangular or prismatic rules that shape grammars are built on. The system tries to force a standard gable roof onto a vault, and the result is geometry that looks like a drafting error: planar faces cutting through what should be a smooth curve. This is the "non-standard architecture cliff," and it's not an edge case—it's the entire class of bespoke buildings.

Oversimplification is a tolerance problem. The vectorization runs plane-fitting algorithms, and the tolerance threshold is the dial you're turning. Set it too loose and molding, balustrades, and recessed balconies merge into flat monolithic slabs—you lose the architectural detail that makes a building interesting. Set it too tight and you get jagged, non-coplanar face artifacts that look like a poorly decimated mesh. There's no setting that's right for all inputs, which is why this is an operator judgment call.

T-junctions and non-manifold seams appear at the intersection of procedurally generated elements. When a roof asset is snapped onto an extruded footprint, floating-point rounding produces unwelded vertices and micro-gaps. In a deferred renderer, that's a light-leak. In a physics engine, it's a raycast miss. The interesting part is that these are invisible in the DCC viewport—the seam looks closed—and only appear when the asset is lit dynamically.

Lighting contamination is the one that bites you last. If the de-lighting pass is skipped, the baked albedo carries the ambient shadows and specular highlights from the capture environment. The asset looks correct in static lighting, then breaks the moment dynamic lights hit it: shadows that should move stay frozen in place, highlights that should shift are baked into the diffuse map.

stateDiagram-v2
    [*] --> raw_scan
    raw_scan --> segmentation
    segmentation --> clean_parse
    segmentation --> misclassified
    clean_parse --> vectorization
    misclassified --> fallback_retopology
    vectorization --> footprint_ok
    vectorization --> concave_collapse
    footprint_ok --> grammar_synthesis
    concave_collapse --> fallback_retopology
    grammar_synthesis --> standard_shape
    grammar_synthesis --> non_standard_break
    standard_shape --> material_bake
    non_standard_break --> fallback_retopology
    material_bake --> de_lighted_ok
    material_bake --> baked_lighting_contamination
    de_lighted_ok --> final_output
    baked_lighting_contamination --> fallback_retopology
    final_output --> [*]

Each failure mode has an observable signature, which is the only reason this is production-feasible: you can build automated checks that catch a concave collapse or a misclassified roof before it reaches the artist. But you can't automate away the judgment calls. Which tolerance setting is right? Should a green roof be tagged as vegetation or structure? These decisions shape the output, and they require someone who understands what the building actually is, not just what the scan shows.

Production Realities: Cost, Scale, and Manual Overrides

The pipeline mechanics are half the story. The other half is what it actually takes to run this in production, and that's where the unglamorous engineering lives.

Processing time is the first reality check. A single building facade might process in seconds, but a city block—or a planetary-scale dataset like Microsoft Flight Simulator uses—is a different beast entirely. Semantic segmentation over millions of points is GPU-bound, but vectorization and grammar synthesis are largely CPU-bound, and they don't parallelize as cleanly. The pipeline is also memory-hungry: the raw scan, the intermediate labeled surfaces, the extracted footprints, and the generated procedural geometry all need to coexist in memory. I've seen 16GB machines swap heavily on modest scenes, which pushes you toward chunked processing and out-of-core workflows.

Scale drift is a silent killer. If your scans lack ground control points or RTK GPS positioning, metric scale drifts non-linearly across large datasets. A procedural rule set that expects a 2.1-meter door will place doors that look absurdly oversized or miniature depending on where in the dataset you are. The fix is calibration—but that requires knowing your capture was uncalibrated, which is often only discovered when the first asset looks wrong.

Occlusion holes produce hallucinated surfaces. Aerial scans have blind spots under eaves, behind vehicles, at recessed entrances. Unsupervised procedural systems fill those gaps with flat closure planes—which is occasionally correct, but often entirely wrong. You get a solid wall where a recessed doorway should be, or a flat foundation where the terrain slopes. The confidence threshold is your only mitigation: when the system is uncertain about geometry, it should either fall back to retopology or flag for artist review.

Automated de-lighting is not optional. I'd insist on it as a pre-pass before any texture projection. Without it, you bake static shadows into the albedo, and the asset breaks the first time it's lit dynamically. The de-lighting pass is also the only way to extract clean normal maps, which you need for the procedural geometry to respond to real lighting.

The artist-in-the-loop is the production backbone. Astra will not get every scene right, and those misses need a manual override path. Artists need to be able to edit vector footprints directly—nudge a wall boundary, delete a misclassified roof—without rebuilding the scene from scratch. They need to re-trigger the grammar on a single component rather than re-running the whole pipeline. And they need confidence scores surfaced so they know which elements to review without inspecting everything.

When to fall back to traditional retopology is a judgment call, but I'd set a threshold: if semantic confidence drops below what you'd consider an acceptable classification accuracy for your domain, or if the scene contains non-standard geometry that the grammar will just mangle, reach for Quad Remesher or Instant Meshes. The result is a frozen mesh, but it's a correct frozen mesh. For some assets—a scan of a complex organic prop that will never be edited—that's the right tradeoff. The pipeline should know when to surrender, and the engineering work is building that surrender point into the flow rather than letting it be a manual discovery.

Three Projects to Push This Pipeline Further

The gap between "this works on buildings" and "this works on my domain" is where the interesting engineering happens. Here are three concrete projects that would extend Astra's pipeline into territory worth owning.

City-scale procedural generator with fallback. Build a pipeline that takes drone scans of a city block, runs segmentation and vectorization to extract building footprints and heights, then drives grammar rules to generate clean building meshes. The critical piece is a confidence scorer: for each building, aggregate the semantic segmentation probabilities across its surfaces, and if confidence drops below 85%, fall back to quad remeshing with Instant Meshes instead of forcing a grammar. That threshold is where the architecture chooses between editability and fidelity—and it's the difference between a pipeline that produces useful output and one that produces hallucinations. What to watch for: segmentation errors on roofs with greenery (a tree canopy over a flat roof will tag it as vegetation and skip the extrusion), concave courtyards collapsing into convex hulls, and scale drift from missing ground control points. Use RTK GPS data if you can get it; without it, your 2.1-meter door rule produces absurd proportions across the dataset.

Interactive procedural editor for scanned buildings. Create a Blender addon that imports the procedural assembly, lets artists edit the footprint polygons directly or adjust grammar parameters (window spacing, roof pitch, wall height), and re-runs the grammar to regenerate the building in real time—without losing the clean topology. This is the project that turns the pipeline from a batch process into a working tool. The connection layer is the vectorized footprint data feeding a shape grammar engine, with Blender's Python API handling UI and geometry updates. The hard part is seam integrity: when you re-run grammar on an edited footprint, adjacent elements (a roof snapping onto a changed wall) produce non-manifold seams and micro-gaps from floating-point rounding. Fix by welding vertices and running mesh integrity validation before export—and make that validation part of the regeneration loop, not a post-processing step.

LOD generation driven by procedural hierarchy. Integrate the procedural output into Unreal or Unity, but don't treat it as a single mesh. The semantic components—walls, windows, roofs, terrain—are separate objects, which means you can generate LODs by simplifying each component independently. A complex roof with dormers and chimneys keeps its detail; a featureless wall drops to something minimal. The result is LODs that preserve detail where it matters and cut triangles where it doesn't, which is exactly what manual LOD authoring does—but automated. Watch for texture stretching and UV misalignment when simplifying individual components; guardrail by preserving UV seams and testing at distance, where stretching is visible even if the geometry looks fine up close.

Each of these projects exercises a different part of the pipeline under real pressure. The first tests the confidence fallback logic, the second tests grammar re-generation under edit, the third tests whether the semantic hierarchy actually earns its keep downstream.

Resources

• blackshark.ai
• 302 Found