CADGenBench Grades AI-Generated Mechanical Parts Like an Engineer Would: A Validity Gate, Keep-In Volumes, and Betti Numbers

The Gap It Names: Parts That Render Fine and Assemble Wrong

Generative 3D has a category confusion problem, and Hugging Face's CADGenBench (github.com/huggingface/cadgenbench, Python, Apache-2.0) is built to exploit it. The confusion: most public progress in AI-generated 3D — splats, meshes, text-to-3D showcases — measures how a model looks from novel viewpoints. Mechanical CAD lives under a different regime entirely. A part is not an image; it is a boundary representation with dimensions, mating interfaces, and a topology a machinist or a CAM tool will interrogate. A generated bracket can render beautifully, mesh plausibly, and still be useless because a hole is misplaced by two millimeters or the mounting face is not flat.

CADGenBench's framing, straight from the README: it "measures how well AI systems produce correct 3D mechanical parts." That word correct is doing all the work, and the benchmark's entire evaluation stack exists to give it engineering meaning instead of perceptual meaning. The repo (114 stars at the time I read it) packages three things: the scoring engine that the leaderboard Space runs server-side, the metric and submission documentation, and an optional reference baseline — an agent that writes CAD code, renders it, looks at its own renders, and iterates until the part is valid.

Two tasks, both grounded in real engineering workflows:

• Generation — from an engineering drawing of a part, produce a valid, geometrically correct 3D model.
• Editing — given an existing STEP file and a requested change, apply that change.

The second task is the one I find more consequential and less commonly benchmarked. Generation from a drawing is a hard cold-start problem; editing an existing STEP file against a change request ("move this boss 5mm outboard") is the day-to-day reality of CAD work, and it exercises a completely different capability: understanding existing parametric geometry well enough to modify it without breaking everything else.

Two Tasks, One Submission Contract

The benchmark is deliberately tool-agnostic. It does not care whether you produced the model with build123d, CadQuery, Autodesk Fusion, or Onshape. A submission is one candidate file per sample — a STEP/BREP file named output.step (or output.stp) — plus metadata. Each sample in the public dataset declares its task type (generation or editing) in a description.yaml, and the identical metrics and file contract apply to both tasks.

That design choice has consequences worth noticing. By standardizing on STEP/BREP — the interchange format every CAD system speaks — the benchmark evaluates the artifact, not the method. Anything that can emit a valid STEP can compete: an LLM writing Python against a CAD library, a fine-tuned generative model, a human with Fusion for that matter. This is the same insight that made image benchmarks fungible — the file format is the interface — applied to a domain where the interface choice actually matters, because STEP carries exact geometry (BREP), not tessellated approximations.

The datasets split in the way that keeps a benchmark honest over time:

Repo	Visibility	Contents
HuggingAI4Engineering/cadgenbench-data	Public	Inputs: descriptions, optional input STEP files and renders, plus sanity_check_submission.py
HuggingAI4Engineering/cadgenbench-data-gt	Private	Ground truth: ground_truth.step, optional jig sub-volumes, renders, authoring docs — read only by the leaderboard Space

Public inputs, private answers, and a local script that lets you exercise the same validity gate the Space will apply before you ever upload. Anyone who has run a public benchmark that later leaked its test set will recognize why this topology is the correct default.

Four Metrics and a Gate: How the CAD Score Is Built

The Space scores every candidate against ground truth on four axes, and the composition rule is the most engineering-sane detail in the whole design: validity is a gate, not a component. A candidate that fails validity zeroes everything else.

Metric	What it captures
Validity	Is the BREP well-formed and watertight — does it mesh into a closed manifold? Gate: failure zeroes the rest.
Shape similarity	Geometric distance: surface-distance F1 and volume IoU against ground truth.
Interface match	Mating-feature correctness, evaluated via authored keep-in / keep-out sub-volumes.
Topology match	Betti numbers (b0, b1, b2) of the tessellated boundary.

The CAD Score is a weighted combination of the applicable component scores, gated by validity. Let me take the four apart, because each one encodes a specific engineering failure mode that perceptual metrics cannot see:

Validity asks the question every CAD kernel asks before anything else: is this solid real? A BREP that is not watertight — that meshes into something with holes, degenerate faces, or non-manifold edges — is not a slightly bad part; it is not a part at all. Downstream tooling (meshers, CAM, simulation) will either reject it or produce garbage. Making this a gate rather than a penalty says something correct about the domain: 90% of a usable part scores zero, because 90% of a usable part is unusable.

Shape similarity is the closest thing to a conventional metric here, but implemented in geometry rather than pixels: surface-distance F1 (how well the candidate's surfaces sit where the ground truth's surfaces are) and volume IoU (how much of the solid overlaps). This is where dimensional accuracy gets measured — a part with every feature displaced slightly will bleed F1 and IoU continuously. Notably, this is a relative measure against ground truth, not an absolute tolerance check against the drawing; more on that in the critique.

Interface match is the metric I find genuinely novel. Details below; the short version is that it evaluates whether the part's mating features — the surfaces that have to meet another part — actually land where assembly requires, using authored keep-in and keep-out sub-volumes as the test jig.

Topology match compares Betti numbers of the tessellated boundary: b0 (connected components), b1 (loops/handle holes), b2 (voids/cavities). In machinist's terms: did you get the right number of separate bodies, the right number of through-holes, and the right enclosed voids? A part with the exact right outline but one extra hole has identical shape-similarity to within a few percent and identical validity — and it fails topology match, because it is a different object. Betti numbers are a cheap, robust, occlusion-independent fingerprint of "did you build the same kind of thing."

Interface Match Is the Most Interesting Metric

Most 3D benchmarks stop at geometry. CADGenBench's third axis evaluates the thing that actually makes a part a part: its interfaces with other parts. The mechanism, per the metric docs, is authored keep-in / keep-out sub-volumes — regions of space the ground-truth author designates where mating features must land (keep-in) and must never intrude (keep-out). Think of it as an inspection jig in software: bring your part, and we check whether the bolt bosses sit in the pockets where the mating flange expects them, and whether nothing pokes into the volume where the cable harness routes.

The evaluation outcome per interface is graded — the illustrations in the repo name fit / partial / fail cases for both keep-in and keep-out regions — which maps directly onto how assembly failures actually present: a bolt hole 0.2mm off-center might assemble (fit), one 2mm off might force reaming (partial), one 10mm off makes the part scrap (fail).

Why this matters methodologically: interface correctness is the failure mode that shape metrics systematically miss. Two parts can have high volume IoU and excellent surface F1 while differing in whether their mounting interface aligns with the rest of the assembly — a few percent of surface area, concentrated exactly where function lives. Authoring jigs is also the only scalable way to test this: encoding "this face must mate with that face" as CAD constraints would require understanding the generator's modeling tree; testing "geometry passes through this region of space" requires nothing but the solid. The jig abstraction trades annotation effort (someone authors keep-in/keep-out volumes per sample) for generator-agnostic evaluation — the same trade the rest of the benchmark makes with STEP files.

The Baseline Agent: Write Code, Render, Look, Repeat

The reference baseline is the most immediately useful artifact for practitioners, because it is a complete, working agentic CAD pipeline you can run, fork, or benchmark against. Its architecture is a tight perception-action loop:

• The agent writes CAD code — by default build123d Python, with CadQuery as an alternate backend; both export STEP/BREP.
• The resulting STEP is rendered in-process via PyVista/VTK — no Chromium or browser install, just system OpenGL on headless Linux (libgl1/Mesa); macOS works out of the box.
• The agent reviews those renders and refines its code, iterating until the part is valid.
• Cheat sheets for both build123d and CadQuery ship inside the baseline package as reference material for the model.

flowchart TD
    DESC[Sample description.yaml<br/>from cadgenbench-data] --> PROMPT[Task prompt + library cheat sheet<br/>build123d or CadQuery]
    PROMPT --> LLM[LLM via LiteLLM<br/>any provider/model string]
    LLM --> CODE[Write CAD Python code]
    CODE --> EXEC[Execute: build solid,<br/>export output.step]
    EXEC --> VAL{Validity check<br/>well-formed, watertight BREP?}
    VAL -->|invalid| FIX[Feed errors back to the agent]
    FIX --> LLM
    VAL -->|valid| RENDER[Render STEP in-process<br/>PyVista / VTK]
    RENDER --> REVIEW{Agent reviews renders<br/>against the drawing}
    REVIEW -->|geometry wrong| LLM
    REVIEW -->|matches| CAND[Candidate output.step<br/>ready for submission]

The loop is worth studying as an agent design independent of CAD: the validity check is a cheap programmatic gate applied every turn, reserving the expensive perceptual judgment (does this look like the drawing?) for candidates that have already passed structural sanity. That ordering — programmatic gate before model judgment — is the same pattern CADGenBench's own scoring applies, and the same pattern I would copy in any domain where generated artifacts have both checkable structure and judgeable quality.

Running it is deliberately boring: pip install -e ".[baseline,dev]" on Python 3.12, API keys in .env, then cadgenbench baseline run 101 --model openai/gpt-5.5 for one sample or --all --parallel 4 for the set. The --model flag takes any LiteLLM provider string, so the same harness sweeps providers by changing one argument — the README's own examples span openai/gpt-5.5, anthropic/claude-opus-4-7, and gemini/gemini-3.1-pro-preview. A package command bundles a results directory into a contract-compliant submission zip. The baseline only generates; scoring happens on the Space, which keeps the evaluation surface identical for everyone whether they used the baseline or a hand-rolled pipeline.

The Evaluation Topology: Public Inputs, Private Answers, Two-Tier Trust

The submission flow closes the loop in a way that handles the two classic benchmark pathologies — overfitting to public answers and unverifiable leaderboard claims:

sequenceDiagram
    participant P as Participant
    participant L as sanity_check_submission.py (local)
    participant S as Leaderboard Space
    participant GT as cadgenbench-data-gt (private)
    participant M as Maintainer team
    P->>P: generate output.step per sample (any tool)
    P->>L: run validity gate locally before uploading
    P->>S: upload submission.zip (per-sample folders + meta.json)
    S->>S: validate zip layout
    S->>GT: score candidates against private ground truth
    GT-->>S: per-metric scores
    S->>S: compute gated CAD Score
    S-->>P: leaderboard row + per-submission HTML report
    Note over S,P: rows publish as UNVALIDATED
    M->>S: methodology review for validated tier<br/>(docs/benchmark/validation.md)

Two design decisions here deserve credit. First, rows publish as unvalidated, and promotion to a validated tier is a separate methodology review by the maintainer team, with accepted evidence types documented. A leaderboard that distinguishes "a number we computed from your file" from "a number we will stand behind after checking how you got it" is rarer than it should be. Second, the per-submission HTML report means the benchmark returns diagnostics, not just a rank — you can see which axis failed, which is what makes a benchmark a development tool rather than a scoreboard.

What This Benchmark Does and Does Not Tell You

Because the failure modes it measures are narrower than "CAD ability," here is the honest capability map:

Question	Does CADGenBench answer it	Via
Can the system emit a structurally sound solid at all?	Yes, decisively	Validity gate
Is the geometry dimensionally faithful to the drawing?	Approximately	Surface-distance F1 and volume IoU against ground truth
Do mating features land where assembly needs them?	Yes	Authored keep-in / keep-out jig volumes
Does the part have the right holes, bodies, and voids?	Yes	Betti numbers b0/b1/b2
Can it hold explicit tolerances (e.g., ±0.05mm on a bore)?	Not directly	No GD&T / tolerance axis in the metric set
Can it edit parametric intent (feature trees) rather than just solids?	Partially	Editing task works on STEP solids; parametric history is out of scope
Can it do assemblies with kinematic constraints?	No	Out of scope

That last group is not a criticism so much as a boundary marking. Tolerance-aware scoring would require ground truth annotated with GD&T callouts and a very different comparison stack; STEP files carry exact BREP geometry, but the benchmark's ground truth is a single reference solid, so accuracy is measured relative to that solid rather than against annotated tolerances. In practice surface-distance F1 against a correctly built reference is a good proxy for dimensional discipline — you cannot get tight surface F1 while being loosely dimensional — but it will not tell you whether a system respects a ±0.02mm callout, and nobody should read it that way.

Who Should Do What With It

• If you are evaluating text-to-CAD or CAD-editing systems for engineering use: this is currently the most credible public harness I have seen for exactly that question — submit candidates and read the per-axis reports rather than the single score. A system with strong shape similarity but failing interface match is a very different procurement risk than one failing validity occasionally.
• If you are building an agentic CAD pipeline: the baseline is a better starting point than its "optional" label suggests — the write/execute/validate/render/review loop with programmatic gating is reusable even if you replace every model and library choice.
• If you are doing generative-3D research: note what the benchmark is telling the field. Visual fidelity metrics have been the field's scoreboard for years; this one is built so that a model cannot score well by looking right. Expect leaderboard gaps between perceptual benchmarks and this one to be the quantified version of the demo-versus-manufacturable gap.
• If you want to contribute samples: the authoring side (ground truth solids, jig volumes, and labeller-facing sanity-check scripts) is documented in the private repo's AUTHORING.md, and the benchmark lives or dies on sample diversity — part families, machining features, and editing operations are where I would want to see growth.

The one-line summary: CADGenBench moves the finish line for generative 3D from "looks like a part" to "would survive contact with an assembly," and the scoring engine — a validity gate, geometric distance, authored mating jigs, and topology fingerprints — is a genuinely well-designed instrument for that claim. What it measures today it measures correctly, and what it does not measure yet it labels clearly enough that you can notice the difference.

Resources

• huggingface/cadgenbench — GitHub repository and README