The Standard of Completion: How Factory's Three-Role Agent System Rebuilt gdal to 90 Percent Parity

Run one model at gdal twice and you get two different programs. As a single agent, Fable 5 wrote 17,000 lines of C++, reproduced 35.8 percent of the reference program's measured behavior, and stopped because it judged itself finished. Arranged into a three-role system with a validator that owned an independent measure of completion, the same model produced 115,000 lines and reached 90.3 percent on the same hidden grading suite. The model never changed. The arrangement of labor around it did, and that arrangement is the whole result.

Factory Research's write-up of the experiment, "What it Takes for Coding Agents to Complete Large Software Tasks," is the most concrete public evidence I've seen for a claim I've been forming from my own agent work: the ceiling on long-running coding agents is set by who owns the definition of done, and by whether that definition is executable. Below I walk through the benchmark, the architecture, the instrument, the receipts across 24 tasks and three models, the cost ledger, and the artifact Factory shipped, then lay out what I'd actually reuse from this in production agent systems.

A Self-Grading Agent Quit gdal at 36 Percent and Called It Done

gdal is the command-line face of the GDAL project, the geospatial workhorse that has been in development since 1998 and sits beneath QGIS, ArcGIS, and PostGIS. Roughly two million lines of C/C++ upstream, about 600,000 of them reachable through the CLI surface the benchmark configures. The single-agent run implemented, checked its own work, and decided when it was done. Its checks were real; the common paths worked. The run still covered just over a third of the behavior space.

The failure Factory diagnosed is structural, and I recognize it from every long refactor I've handed to an agent. A coding agent validates each piece in the context that produced it. Those checks inherit the scope of the work that generated them, so they can establish everything the agent thought to build while excluding features, interactions, and constraints it never represented. The agent then reads its own coverage as completion. It did not run out of time or budget; Factory states plainly that it stopped because, by its own assessment, it was done.

That is the specific defect the system arrangement attacks: an inventory of what must be established, procedures for establishing each part, and current evidence that those procedures pass, all constructed before implementation narrows the task into work items. Requirements state what must be true; they never measure whether the work achieves it. Somebody or something has to hold the whole outcome while the agent handles parts.

ProgramBench Turns the Reference Into a Black-Box Oracle

The substrate for the experiment is ProgramBench, a cleanroom software-engineering benchmark from Meta Superintelligence Labs, Stanford, and Harvard, described in arXiv 2605.03546 and maintained at programbench.com. Each of its 200 tasks hands the agent an executable binary, fixtures, and partial documentation, then asks for a from-scratch reimplementation. The binary is execute-only: it can be run without limit, but never read, decompiled, disassembled, or traced, and there is no internet access. Agent-driven fuzzing produced more than 248,000 behavioral tests across the task set, and grading happens against a hidden suite. The task list spans tiny utilities like peco and hexyl up to FFmpeg, SQLite, and the PHP interpreter.

On the public leaderboard, running a minimal mini-SWE-agent harness, Claude Opus 5 at xhigh reasoning fully resolves 4.5 percent of tasks and clears 95 percent of tests on 37 percent; GPT-5.6 Sol resolves 1.0 percent. Individual runs have cost up to $5,000. Rebuilding programs from scratch with no structural hints is brutally hard, and that hardness is the point: the model must discover both the behavior space and how to measure it.

Factory selected the 24 hardest tasks, weighted toward the lowest best-public scores (the selected set's median best-public score sat at 41.9 percent against 89.1 percent for the other 176), and dropped candidates like php-src, pueue, ditaa, quickjs, chroma, and miller during screening for persistent safety blocks, single-agent saturation, or scores gated by one undocumented environment variable. Three frontier models ran the comparison: Fable 5 at xhigh reasoning, Kimi K3 at high, and GPT-5.6 Sol at max. Each task-and-model cell was one campaign in each of two conditions, run without human intervention, not compute-matched, and graded exactly once on the hidden suite pinned at pb-1.2.0. Both conditions could execute the reference freely and could not read it, inspect the benchmark's tests, or touch the network.

Three Roles and a Wall Separate Judging From Building

The system condition wires the same model into three roles. The implementer builds the candidate program. The validator surveys the reference and, before any implementation starts, authors the instrument: a weighted body of test cases plus the comparison rules for judging output. The orchestrator sits above both, adjudicating findings and deciding what happens next and when to ship. The load-bearing feature is the wall between measuring and building: the candidate and clustered findings cross it; the instrument and its raw output never do.

flowchart TD
    REF["Reference program<br/>execute-only black-box binary"]
    IMP["Implementer<br/>writes and advances the candidate"]
    VAL["Validator<br/>measures the candidate, clusters failures"]
    ORC["Orchestrator<br/>adjudicates findings, issues directives, ships"]
    INST["Instrument<br/>weighted cases + comparison rules"]
    HS["Hidden suite pb-1.2.0<br/>grades once after ship"]

    REF -->|"run without limit"| IMP
    REF -->|"run without limit"| VAL
    ORC -->|"directive: what to fix next"| IMP
    IMP -->|"candidate"| VAL
    VAL -->|"findings clustered by root cause"| ORC
    ORC -->|"what to measure next"| VAL
    VAL --- INST
    IMP -.->|"never sees cases or raw output"| INST
    VAL -->|"final candidate"| HS

The wall holds in both directions, and the asymmetry is deliberate. The validator may expand the instrument as it learns more about the reference; it may never weaken or revise it to accommodate what the candidate happens to contain. The implementer never authors, runs, or sees the cases: once a sparse sample becomes visible, it becomes the optimization target, and passing it establishes those cases rather than the behavior space they sample. Every role ran the same model at the same reasoning level, so the gain came from information flow, and the cleanest evidence for that is the wall itself costing nothing but a shortcut.

Role	Owns	Receives	Never sees
Validator	The instrument: weighted cases and comparison rules	The reference binary, each successive candidate	Directives to weaken coverage
Implementer	The candidate codebase	Directives naming weak subsystems, reference access	Instrument cases, raw measurements
Orchestrator	Campaign control: next measurement, ship decision	Clustered findings from the validator	Raw instrument output

The Instrument Defaults to Byte Identity and Licenses Its Exceptions

Full behavioral parity is unmeasurable in principle; a program accepts an effectively unbounded input space, so any practical validation samples it. The instrument is that sample, and Factory specified it only in outline: which behaviors matter, how they are weighted, and what counts as evidence were the validator's calls. For gdal the validator wrote hundreds of cases. Two examples from the write-up:

D('hillshade.combined', ['raster', 'hillshade', '--variant', 'combined', 'dem.tif', 'out.tif']),
D('contour.levels', ['raster', 'contour', '--levels', '120,150,180', 'dem.tif', 'out.geojson']),

The grading policy is where this gets genuinely strict. For every case, the runner executes oracle and candidate sequentially in the same absolute sandbox path, wiped between runs, with identical environment, fixture bytes, and file mtimes, then byte-compares four channels: exit code, stdout bytes, stderr bytes, and the full work-tree delta, including every sidecar the run created or modified, down to .aux.xml, .ovr, .msk, shapefile companions, and tile trees. Byte identity is the default on every channel. Deviation exists only as a named normalizer attached to specific cases, each carrying recorded evidence that the reference itself cannot produce stable bytes there. Across hundreds of gdal cases the validator licensed exactly two relaxations: a masked heap address in debug traces and a date embedded in a file header.

This is the discipline most agent setups skip. My own eval suites default to spot checks and "looks right"; this defaults to byte equality and treats every exception as a documented, evidence-backed license. The stance transfers directly: if you cannot say why two runs of the same invocation should differ byte-for-byte, they should not.

The Outer Loop Turns Failures Into Weighted Directives

Once implementation begins, the loop is a four-beat cycle. The orchestrator chooses what to measure. The validator runs the current candidate against the instrument and groups failures by root cause. The orchestrator rejects noise and invalid measurements, then converts real problems into a directive at the level of missing features, subsystems, or behaviors. The implementer investigates the reference independently and advances the candidate. When the instrument stops revealing meaningful differences between successive candidates, the orchestrator can order an expansion of a weak area or targeted differential testing against the reference.

sequenceDiagram
    participant O as Orchestrator
    participant V as Validator
    participant I as Implementer
    participant R as Reference binary
    Note over V: instrument authored before any implementation
    loop until the instrument stops revealing differences
        O->>V: choose what to measure next
        V->>R: run oracle on selected cases
        V->>V: run candidate through the same cases
        V->>O: findings clustered by root cause
        O->>O: reject noise, adjudicate real failures
        O->>I: directive naming weak subsystems and weights
        I->>R: probe the reference independently
        I->>V: advanced candidate
    end
    O->>O: decide to ship
    Note over V: hidden suite grades once, outside the loop

The directive format is what makes the wall survivable. Here is part of one from the gdal run, telling the implementer where the candidate is weak without leaking the sample:

## 1. Stub frontier (dominant mass, ~60 verbs)

Ordered by measured weight (approximate case mass in brackets). Probe each against the oracle first, as always.

- vgrid: all 11 grid methods [14]
- mdim group on multidim VRT + classic-input pins [12]
- sozip: create / list / validate / optimize [11]
- DEM suite [~30]: contour [7], hillshade incl. variants [6], aspect [6], slope [4], viewshed [4], roughness, tpi, tri [1-3 each]
- rasterize [7], pixel-info [7], calc [7], mosaic [6] + stack, tile [5], footprint [5]

Weighted, verb-level, evidence-backed, and free of any case text the implementer could overfit. When I compare this to the way I usually feed failures back to a coding agent, full test output pasted into context, I can see the failure mode this prevents: the agent starts passing the printed examples instead of closing the behavioral gap.

The Receipts: Medians Jumped on Every Model, With Real Exceptions

Every score below comes from the hidden suite, which sampled the same behavior space no role ever saw, so the gains transferred from the self-built instrument to an independent measure. The medians across the 24 tasks: Fable 5 moved 56.7 to 89.3 with 73 percent of the single-to-system gap closed, Kimi K3 45.1 to 75.4 at 42 percent, GPT-5.6 Sol 48.6 to 66.2 at 25 percent. Wall time stretched accordingly: 8.5 to 96 hours for Fable, 9.0 to 64 for Kimi, 1.5 to 24 for Sol.

Task	Fable 5	Kimi K3	GPT-5.6 Sol
peco	92 to 100	79 to 98	77 to 95
sox	70 to 100	52 to 76	49 to 66
proj	97 to 99	44 to 88	27 to 90
stgit	93 to 97	37 to 88	78 to 84
7zip	54 to 95	76 to 81	78 to 94
gdal	36 to 90	57 to 72	58 to int.
pandoc	19 to 84	32 to 70	40 to 59
ctags	14 to 76	28 to 58	28 to 46
duckdb	34 to 80	31 to 41	30 to 29
lnav	38 to 84	18 to 66	43 to 21
tree-sitter	55 to 91	46 to 74	62 to 57
ffmpeg	9 to 40	9 to 13	11 to 17
cppcheck	30 to 54	t/o to 4	5 to 7

Official scores, rounded; t/o marks an evaluation timeout scored as a floor; int. marks a run interrupted and not rerun (gdal and gromacs on Sol, samtools on Kimi; 141 of 144 cells were graded before publication). The biggest absolute climbs: pandoc plus 65 points, ctags plus 62, lnav plus 46 on Fable, proj plus 63 on Kimi. Where the system frontier lands in the high 90s, sox at 99.5 and peco at 99.6 on Fable, the recreation is effectively done.

The exceptions deserve equal attention, because they map the technique's failure surface. Sol regressed tree-sitter from 62 to 57, lnav from 43 to 21, and nudge duckdb from 30 to 29; the weakest model converted the arrangement into a net loss on three tasks. Kimi's cppcheck system run scored 4.1 after its single-agent evaluation timed out, and its ast-grep evaluation hung twice with nothing graded. Evaluation infrastructure, and the model's ability to benefit from adversarial measurement, bound the technique as much as implementation skill does. Six Fable cells that were safety-blocked ran with Opus standing in (bedtools2, gromacs, pandoc, samtools, sox, tree-sitter), disclosed per cell. And every number is one run per cell with no variance estimate, so I read the medians as directional and the per-task deltas as case studies.

What the 14x Cost Premium Actually Bought

The gdal receipts make the price explicit. The system campaign consumed 3.00 billion credits against the single agent's 216 million, a 14x multiplier, and 196.9 wall hours against 15.0, a 13x multiplier. Within the system spend, implementation took 97 percent, validation 2 percent, orchestration 1 percent; the instrument itself cost 53 million credits to author. The candidate grew 6.7x in core lines and absorbed 3,450 file edits against 60.

Metric	Single agent	Three-role system	Factor
Official score	35.8%	90.3%	2.5x
Wall time	15.0h	196.9h	13x
Credits	216M	3.00B	14x
Core lines	17k	115k	6.7x
Core files	48	104	2.2x
File edits	60	3,450	9.0x

The decisive line in the write-up is that budget was not what separated the conditions: every single-agent campaign ended because the agent decided to end it. Additional compute does not help an agent that will not spend it. The system changed the judgment of completion, and the compute followed from that judgment. That reframing matters for anyone budgeting agent fleets: I should not ask "how much more compute does this task need," I should ask "who holds the standard, and would more spending even flow through it."

stateDiagram-v2
    [*] --> InstrumentAuthored
    InstrumentAuthored --> Implementing: first directive issued
    Implementing --> Measuring: candidate crosses the wall
    Measuring --> Implementing: findings become directives
    Measuring --> DifferentialProbing: instrument stops revealing differences
    DifferentialProbing --> Measuring
    Measuring --> Shipped: orchestrator judges parity reached
    Shipped --> [*]: hidden suite grades once

My working decision table for when I'd pay the premium:

Situation	My call
Focused change with a tight existing test suite	Single agent; the standard already exists in CI
Greenfield rebuild of a system with runnable truth	Validator system; the oracle is cheap to consult
Migration where the legacy system keeps running	Validator system; legacy binary is the oracle
Long-tail polish after the cheap wins	Single agent with the instrument's residuals as its backlog
Model that regressed under the arrangement (Sol on tree-sitter)	Pilot per model before committing credits

pb-gdal-fable Is the Artifact You Can Kick Yourself

Factory open-sourced the system run's output at github.com/Factory-AI/pb-gdal-fable, and the README is worth a slow read because it documents what near-parity without source access actually produces. The recreation is 115,163 lines of C++17 across 104 files and 125 command paths, produced over 8 days and 10 review passes against a GDAL 3.13.0dev binary trimmed to 11 drivers with no GEOS. It scored 1,184 of 1,319 on the held-out behavioral suite, 89.8 percent raw and 90.3 on the official metric.

The provenance numbers are the interesting part. Measured against all 2,952 upstream files, the recreation shares zero verbatim line runs of four or more non-trivial lines, and its code 7-gram overlap is 4.18 percent, all generic C++ idiom. The 21.2 percent shared string literals are GDAL's own error messages, reproduced on purpose because the reference prints them. No GDAL API convention survives: no CPLError, no GDALDataset type, no poDS. Four libraries are linked rather than rewritten because byte-identical output demands them (PROJ 8.2.1, libdeflate, zstd, libwebp), while libtiff, libgeotiff, libjpeg, libpng, and expat were available and went unused; the TIFF container, GeoTIFF keys, JPEG codec, and XML and JSON parsers are original code.

The repo also ships the epistemology. spec/ holds 6.2 MB of output captured from the reference, embedded into the binary at build time. tests/ contains the model's own differential harness plus roughly 10,000 pinned cases that run the reference and the build in twin sandboxes and byte-compare them. NOTES.md is 5,000 lines of derived behavioral laws with their evidence, which the README correctly calls the most interesting file: it is a written behavioral specification of gdal recovered by pure black-box experimentation. Known gaps are documented with the same honesty: multidimensional modes, verbs used inside pipeline, hidden option aliases like --z-field that appear in no help output and so cannot be probed, and geometry verbs that refuse with ERROR 6: GEOS support not enabled., exactly as the reference does.

The Pattern I'm Stealing: External Standards Before Implementation

Strip away the benchmark and three transferable mechanics remain. First, derive an executable standard of completion from the outcome's sources of truth before implementation starts, and keep it current until the work meets it. Second, keep measurement out of the implementer's context; hand over clustered, weighted findings and never raw eval output, because a visible sample becomes the target. Third, give a separate role the ship decision, since the agent that benefits from declaring done should never be the one declaring it.

Factory is explicit that this is the automation of safety-critical practice: requirements traceability and independent verification and validation, the way standards bodies run conformance suites. Humans rarely do it for ordinary software because a conformance suite amortizes across many implementations while a product team pays per application. Agents change both sides: they generate work faster than humans can inspect it, and they can apply the same capacity to building and maintaining the standard itself.

Independent work landed on the same shape from a different direction. Free Wortley's account of seven months running agent swarms against Xbox 360 game binaries documents a search engine that shipped two genuinely wrong rewrites because a naive "accept if the score goes up" rule rewarded them, including one that gained 3.25 points while being equal at a single input; the fix was making byte-exact recompilation the only terminal judge and demoting every other measure to advisory. The same campaign found bounded reasoning beating unbounded (capping thinking effort took one model from 4 of 7 functions byte-exact to 6 of 7) and model diversity beating model IQ (a cheap model and a frontier model agreed on only 8 of 13 solved functions; their union reached 18). Different domain, same conclusion: the verifier is the system, and its independence is not a nicety.

What I'd actually do with this in the next quarter: for any rewrite or migration my teams run through agents, generate the acceptance harness from the legacy system first, with byte-comparison as the default and named, evidence-backed normalizers as the only exceptions. Split the validator role out of the implementer's context and have an orchestrator turn failures into weighted directives at the subsystem level. Track the eval-timeout failure mode explicitly, because Sol's regressions and Kimi's hung evaluations show the measurement layer can lose the game on its own. The gdal result earns attention because it isolates one variable, the standard of completion, and moves it 54 points. That is the lever I want on my own fleets.

Resources

• What it Takes for Coding Agents to Complete Large Software Tasks — Factory Research's write-up with the full experiment, receipts, and method notes
• ProgramBench — the 200-task benchmark, leaderboard, and FAQ
• ProgramBench: Can Language Models Rebuild Programs From Scratch? — the paper behind the benchmark
• Factory-AI/pb-gdal-fable — the open-sourced 115k-line gdal recreation, its differential harness, and NOTES.md
• How agent swarms decompile games byte-for-byte — Free Wortley's independent account of agents, oracles, and reward hacking in binary decompilation