Harness Engineering Over Raw Weights: Why Modular Dev-Primitives Crush Monolithic Coding Agents

The Real Bottleneck in Repo-Scale Coding: Why Model Upgrades Plateau While Scaffolding Scales

Whenever a new frontier checkpoint drops, the immediate industry reflex is to rerun standard evaluation benches and declare whether coding is "solved." I have watched this cycle repeat across multiple model generations, and every time we step outside synthetic single-file patches into real repository refactoring, the same reality bites back: raw model capability upgrades yield sharply diminishing returns when dropped into naive single-agent loops. If the harness surrounding the model treats the codebase like a giant text file dumped into an expansive context window, performance plateaus regardless of parameter count.

The empirical verification of this bottleneck arrived with clarity in recent work on harness engineering by Jin et al. (arXiv:2610.07832, October 2026). Evaluating whole-repository migrations on SWE Refactor Bench, the authors paired GPT-5.6 Sol with the standard Codex harness baseline. The result was an abysmal 6.5% composite success rate. The model had access to state-of-the-art reasoning, tools, and expansive context capacity, yet it fell apart when migrating interdependent components across an entire repository.

The punchline came when the authors held model weights and compute budgets completely static, swapped out the monolithic baseline, and wrapped GPT-5.6 Sol inside a modular dev-primitive orchestration harness called HERMES. Success rates jumped from 6.5% to 31.0%—nearly a 5× leap. Across broader SWE benchmarks, the structural upgrade delivered a 12.4% average performance bump. Nothing about the underlying model changed; the execution harness simply stopped asking a single agent to hold the universe in its head.

SWE Refactor Bench (Whole-Repo Migration) Success Rate:
Codex Baseline + GPT-5.6 Sol:  [███                          ]  6.5%
HERMES Harness + GPT-5.6 Sol:  [███████████████              ] 31.0% (4.77x improvement)

In repository migrations, tasks do not fail because a frontier model cannot write a function or misunderstand syntax. They fail because of context saturation and interface drift. When an agent navigates fifteen files in a single terminal session, the interaction transcript balloons with shell output, intermediate diffs, and lint errors. By turn twelve, the agent suffers from semantic degradation: it forgets type invariants declared three files earlier, assumes stale method signatures, and introduces subtle contract discrepancies across boundaries. As I noted when exploring Hacking the Agent Loop: An Operator's Guide to In-Process Claude Code Mods, treating the agent loop as an unmanaged command prompt inevitably hits a cognitive wall. The true scaling frontier is not token length; it is the harness topology.

Beyond Monolithic Terminals: Four Architectural Paradigms for Autonomous Code Modification

To understand why traditional scaffolding collapses under repo-scale changes, we have to contrast the architectural patterns competing for automated software engineering: monolithic shell agents, functional role-based multi-agent teams, static graph retrieval engines, and decentralized dev-primitives.

quadrantChart
    title Structural Coupling vs Context Mode in Coding Harnesses
    x-axis Passive Retrieval --> Active Autonomous Actors
    y-axis Low Structural Coupling --> High Structural Coupling
    quadrant-1 Decentralized Execution Nodes
    quadrant-2 Graph-Augmented Retrieval
    quadrant-3 Monolithic Shell Baselines
    quadrant-4 Functional Role Hierarchies
    Aider and Devin: [0.18, 0.22]
    SWE-agent: [0.22, 0.26]
    Sourcegraph Cody: [0.24, 0.78]
    Cursor: [0.30, 0.72]
    MetaGPT: [0.75, 0.28]
    ChatDev: [0.82, 0.24]
    HERMES: [0.82, 0.84]

Monolithic tool-using shell agents like Aider, OpenHands, and Devin give a central agent access to a bash terminal, ripgrep, and edit commands. They shine at open-ended CLI exploration, but their flat execution model breaks when handling long-horizon refactors: interaction logs explode, and the model wanders down debugging rabbit holes.

Role-based frameworks like MetaGPT and ChatDev attempt to fix this by organizing LLMs into human organizational hierarchies—Product Manager, Architect, Coder, Tester. In my view, this is a conceptual mistake. Decomposing an engineering workflow by human corporate job titles rather than underlying code topology creates an organizational parody instead of an engineering harness. The "Coder" agent remains a monolithic bottleneck with passive context, struggling to coordinate the cross-file dependency graph. As I explored when looking at Dissecting Claude Dynamic Workflows: Anatomy, Failure Modes, and Operational Realities of 64-Way Server-Side Swarms, swarms only operate coherently when their boundaries reflect the problem's mechanical architecture rather than decorative persona prompts.

Static-graph engines like Cursor and Sourcegraph Cody take the code graph seriously, indexing abstract syntax trees and SCIP/LSIF symbol databases. However, they treat the graph as a passive retrieval index: symbols are extracted and stuffed into a central prompt for a single writer.

Decentralized dev-primitives, introduced by HERMES, abandon the passive context paradigm entirely. Instead of feeding code graphs to an agent, they turn the code graph into the agent network.

Architectural Pattern	Representative Systems	Core Scaffolding Mechanism	Primary Advantage	Repo-Scale Bottleneck
Monolithic Shell Agents	Aider, Devin, OpenHands, SWE-agent	Single central model executing terminal tools and file edits sequentially.	Unconstrained flexibility; zero upfront graph indexing.	Severe context saturation and semantic drift across multi-file dependencies.
Functional Role-Based Swarms	MetaGPT, ChatDev	Persona-based division of labor (Architect, Engineer, QA).	Specialized prompt boundaries for functional phases.	Organizational decomposition ignores file AST topology; the coder remains a bottleneck.
Static Graph RAG Engines	Sourcegraph Cody, Cursor	AST and SCIP/LSIF indexing retrieving relevant slices into prompt context.	Deterministic symbol resolution; compact prompt footprints.	Context remains passive; cannot handle distributed, concurrent interface negotiation.
Decentralized Dev-Primitives	HERMES	Code artifacts paired with resident micro-agents communicating along AST edges.	Strict context isolation; localized editing; failure backpropagation.	Graph parsing overhead; consensus stalls on cross-cutting changes.

Dissecting the Dev-Primitive: Turning Passive Source Files into Active Execution Nodes

The mechanical breakthrough in HERMES is treating code artifacts not as disk buffers to be read, but as stateful, executable micro-agents. Every major component in a repository—a source module, an internal library, a build manifest, or a test file—is wrapped inside a Dev-Primitive.

Each primitive encapsulates three distinct layers:

1. Local State: The raw source code and immediate configuration.
2. Structural Metadata: Local AST slices, caller-callee call graphs, and associated unit test references.
3. Structured Communication Channels: Protocol interfaces that restrict the node to communicating only with upstream callers, downstream dependencies, and dedicated test suites.

By bounding the micro-agent strictly to its local AST scope, the context window never degrades. The node does not need to know how the entire repository works; it only needs to know its own contract, who calls it, and what dependencies it consumes. When an interface changes, the caller and callee negotiate the contract change directly through structured message exchanges.

sequenceDiagram
    autonumber
    participant Spec as Task Specification
    participant Act as Activator
    participant Caller as Caller Primitive
    participant Callee as Callee Primitive
    participant Sandbox as Execution Sandbox
    participant Diag as Diagnostician

    Spec->>Act: Ingest migration requirement
    Note over Act: Traverse AST & dependency graph
    Act->>Caller: Activate node (scope: migration)
    Act->>Callee: Activate node (scope: migration)
    Caller->>Callee: Negotiate contract update (param types, signatures)
    Callee-->>Caller: Confirm interface agreement
    Caller->>Caller: Apply local patch
    Callee->>Callee: Apply local patch
    Caller->>Sandbox: Stage modified artifacts
    Callee->>Sandbox: Stage modified artifacts
    Sandbox->>Sandbox: Execute build & integration tests
    Sandbox-->>Diag: Return failure log & stack trace
    Diag->>Diag: Map stack frames to graph topology
    Diag->>Callee: Target reactivation (isolated error attribution)
    Callee->>Callee: Refine patch implementation
    Callee->>Sandbox: Restage resolved patch

Rather than awakening every primitive across thousands of files, HERMES relies on an Activator component. When a task specification arrives, the Activator analyzes the objective, traces the dependency graph from entrypoints, and selectively activates only the relevant subgraph. Primitives outside this boundary remain dormant, consuming zero tokens and injecting zero noise.

Once active primitives negotiate their contract updates and stage localized patches, the files are handed to the Execution Sandbox. If compilation fails or tests break, the Diagnostician module analyzes the error logs and stack traces. Instead of dumping a 500-line traceback into a central prompt, the Diagnostician maps each failing stack frame directly to the offending node in the dependency graph, reactivating only the implicated primitive to generate a targeted fix.

Execution-Guided Blame Allocation: How Diagnosticians Backpropagate Compiler and Test Failures

In a standard ReAct or bash-loop harness, runtime verification failures are handled with blunt force. When a suite of 200 unit tests fails after a multi-file migration, the harness dumps a 400-line pytest traceback directly into the central model’s context window. The agent must re-read the entire traceback, re-inspect its previous diffs, deduce which of the twelve modified files introduced the regression, and attempt another patch. After two or three failed runs, the context window is drowning in repetitive compiler outputs and fragmented error logs, triggering semantic drift and hallucinated API fixes.

HERMES replaces this brute-force retry cycle with execution-guided blame allocation, orchestrated by the Diagnostician. Instead of treating the build log as raw text for a single writer, the Diagnostician operates as an AST-aware triage compiler:

[Execution Sandbox: Build / Pytest Failure]
                     │
                     ▼
             [Diagnostician]
                     │
       ┌─────────────┴─────────────┐
       ▼                           ▼
[Log Frame Parser]        [AST Call-Graph Matcher]
       │                           │
       └─────────────┬─────────────┘
                     ▼
          [Blame Disambiguation]
                     │
        (Isolated Reactivation)
                     ▼
     [Offending Dev-Primitive Only]

When tests or compiler checks fail in the execution sandbox, the Diagnostician ingests the raw build outputs and stack traces. It tokenizes the stack frames, resolves file paths and symbol identifiers, and walks the repository’s AST call graph to locate the true point of deviation.

Crucially, the Diagnostician differentiates between the symptom site and the root cause. If an import in api_handler.py triggers an AttributeError inside auth_service.py because a method signature was modified upstream, a monolithic agent often perturbs auth_service.py to mask the symptom. The Diagnostician instead maps the call edge back across the dependency graph to api_handler.py, isolates the contract violation, and selectively reactivates only the offending primitive.

The reactivated primitive receives a clean, surgical context: its own file contents, its AST caller-callee metadata, and the exact failing invocation snippet extracted by the Diagnostician. Dormant nodes that compiled cleanly are never notified, preventing token churn and unintended file rewrites. Once the isolated primitive refines its patch, the changes restage to the execution sandbox for another automated pass. Blame travels along the graph edges in reverse, mimicking backpropagation in neural networks rather than an unconstrained trial-and-error shell session.

The Economics of Asymmetric Intelligence: Running Cheap Leaf Nodes Under Frontier Supervisors

One of the most consequential findings in Jin et al.'s work is that decentralized dev-primitives unlock structural cost asymmetry. When an orchestration harness assigns an agent to every module, naive implementations assume every node must run a costly frontier model. Running dozens of frontier-tier instances for routine file-level edits would implode inference budgets and introduce crippling rate-limit bottlenecks.

The authors demonstrated that leaf-node Dev-Primitives do not need frontier reasoning. When paired with a strong frontier model (GPT-5.6 Sol) acting strictly as the high-level Activator and Diagnostician, running lightweight Qwen3-8B instances as resident file primitives achieved a composite success rate within 4.5% of an all-frontier deployment on SWE Refactor Bench, while slashing total inference token expenditures by 26.2%.

Harness Configuration	Activator / Diagnostician Model	Leaf Dev-Primitive Model	Relative Inference Cost	Migration Success Rate	Wall-Clock Latency Profile
Homogeneous Frontier	GPT-5.6 Sol	GPT-5.6 Sol	1.00x (Baseline)	31.0%	High (Frontier API queue delays)
Heterogeneous Asymmetric	GPT-5.6 Sol	Qwen3-8B	0.74x (-26.2%)	26.5% (-4.5%)	Medium (Fast local/leaf inference)
Monolithic Baseline	Codex Baseline (Single-Agent)	N/A (Single Model)	0.82x	6.5%	Low (Early plateau / premature bail)

The mechanical reason this split works comes down to cognitive bounding. A monolithic agent attempting a multi-file migration must simultaneously maintain repository architecture, track cross-module call graphs, parse messy tracebacks, and synthesize exact code diffs—a task demanding frontier-grade reasoning.

Inside HERMES, that macro reasoning is entirely absorbed by the Activator and Diagnostician. By the time a task reaches an individual Dev-Primitive, the problem has been reduced to local self-modification within an isolated file boundary. The leaf node only needs to parse its local AST, respect the structured interface contract negotiated with its immediate neighbors, and apply the diff. An 8B-parameter model excels at localized AST-constrained transformations; it only fails when forced to hold cross-file dependencies in working memory.

For production platform teams, this asymmetry changes the economics of automated refactoring. You reserve expensive frontier tokens for topological planning and stack trace disambiguation, while offloading high-volume code synthesis to low-cost, high-throughput models running locally or on cheap inference endpoints. As I detailed in Turning Production Coding Agents into RL Environments: The Mechanics, Failure Modes, and Economics of Capture Proxies, optimizing the economic boundary of agent loops is what makes autonomous workflows commercially viable.

Ghost Topologies, Drift, and Stale Indexes: Where Decentralized Harnesses Break in Production

Despite the dramatic jump over monolithic baselines, decentralized dev-primitive harnesses are not magic bullets. When deploying graph-coordinated primitives against messy production codebases, several failure modes emerge that academic benchmarks tend to understate.

The primary mechanical failure point is graph invalidation. HERMES relies on static AST and call-graph metadata to establish communication channels between callers, callees, and test suites. But full-repository migrations do not simply edit function bodies; they rename base classes, consolidate modules, change import conventions, and move files across directory trees.

If Primitive A renames a public method or reorganizes an export, its downstream neighbors immediately hold stale AST references. Unless the harness runs incremental symbol re-indexing on every intermediate edit, primitives begin communicating over a ghost topology. Primitives negotiate changes based on contracts that no longer exist on disk, generating code patches that fail compilation before tests can even execute. Maintaining live AST parity across dozens of concurrent leaf edits introduces immense synchronization overhead.

[Primitive A Edits Export] ──(Delayed AST Re-indexing)──> [Ghost Topology Window]
                                                                  │
                                                                  ▼
[Primitive B Assumes Stale Interface] <───(Desynchronized Chat)───┘
               │
               ▼
   [Incompatible Code Patch] ──> [Staged Build Failure]

Second is the risk of consensus deadlock. When cross-cutting concerns touch shared contracts—such as migrating an error-handling wrapper across twenty modules—primitives negotiate peer-to-peer. Without a centralized architectural supervisor to enforce a definitive standard, neighboring nodes can enter infinite negotiation loops, alternately requesting that the other adapt to their proposed signature. Structured natural language protocols leave room for subtle semantic drift: two primitives may agree on an interface specification in dialogue, yet implement diverging type invariants in the actual source code.

Dynamic runtimes amplify this fragility. In languages like Python or JavaScript, reflection, metaprogramming, dynamic dispatch, and loose dependency injection defy static AST analysis. The Activator frequently suffers false negatives, omitting unreferenced configuration files, environment variables, or dynamically loaded plugins from the active subgraph. The active primitives migrate their scoped code cleanly, but the build fails because an unindexed dependency was left untouched.

Finally, we need a dose of operational sobriety regarding the numbers. Jumping from 6.5% to 31.0% is an impressive scientific milestone, but in an enterprise engineering setting, a 31% success rate means 69% of repository migrations still fail. A system that fails more than two-thirds of the time cannot run unsupervised in a continuous delivery pipeline. Decentralized harnesses do not eliminate the human operator; they shift the operator’s role from writing repetitive syntax to triaging desynchronized graph states, breaking inter-agent deadlocks, and resolving boundary mismatches.

Building Your Own Primitive Harness: Practical Implementation Patterns for Engineering Teams

If you want to move beyond monolithic terminal wrappers without buying into heavy academic scaffolding, you must build your harness around three core infrastructure primitives: live Language Server Protocol (LSP) indexing, strictly typed contract schemas, and transactional git checkpoints.

First, throw out naive text-based file viewers in favor of a persistent LSP daemon (such as rust-analyzer or pyright). When a leaf node mutates a file, the harness must query the LSP server to extract updated diagnostic diagnostics, caller-callee trees, and cross-file symbol definitions before downstream nodes execute. Second, abandon unconstrained natural language chat for cross-boundary negotiations. Force nodes to communicate using rigid JSON schemas or typed interface stubs—defining exact symbol names, parameter types, and return values—with strict turn limits to prevent endless handshakes. Third, wrap every diagnostic cycle in git-tree transactional checkpoints. If an applied patch increases net test failures or triggers compilation regressions, your harness must execute an atomic git rollback, discarding that search branch immediately.

[Target Codebase] ──> [LSP Daemon / AST Index] ──> [Typed Contract Broker (JSON Schema)]
                              │                                │
                              ▼                                ▼
                  [Git Tree Checkpoint] <─────── [Scoped Leaf Actor Execution]
                              │
                              ▼
                  [Automated Test Runner] ──(Regressions)──> [Atomic Rollback]

This structural pattern unlocks several practical builds inside modern platform engineering teams. For automated library migrations, you can build an LSP-backed file actor harness that assigns isolated agent actors to individual files during major SDK or framework upgrades, communicating interface diffs over typed RPC. By connecting an LSP daemon for dynamic symbol resolution, AST-grep for syntactic diff extraction, an SQLite state store for component contracts, and lightweight local models running inside Docker containers, you achieve clean token isolation. The main hazard to watch for is silent AST drift, where renames in one file invalidate imports downstream before the background language server finishes re-indexing.

Similarly, teams battling flaky CI regressions can construct an execution-guided diagnostic dispatcher for failing pipelines. This automated repair system parses JUnit or pytest XML test failures, walks call dependency graphs alongside git blame data, and dispatches targeted micro-agents solely to offending modules. Connecting a CI log parser, a call-graph indexer like PyCG, a sandboxed branch runner, and a frontier diagnostic coordinator allows surgical patch dispatch to leaf nodes. Here, the critical failure mode is misattributing blame when dynamic runtime errors or shared fixture corruptions trigger failures in innocent consumer modules rather than the root-cause provider.

Finally, for cross-service changes, you can assemble a contract-driven schema refactoring mesh for monorepos. This decentralized transformation tool updates database models and API schemas by negotiating changes across service boundaries using formal JSON Schema contracts. It chains an OpenAPI or Protobuf parser, a topological graph sequencer, an inter-agent structured contract validator, and an automated git branch reconciliation pipeline. The operational trap here is endless negotiation loops where two tightly coupled services reject each other's schema variants, requiring an authoritative top-level supervisor to step in and break deadlocks.

Harness Engineering Is Graph Search: The Next Horizon in Applied Agent Architecture

The prevailing assumption in AI engineering has long been that model intelligence solves operational problems: wait for the next frontier checkpoint, increase context windows, and complex software tasks will magically resolve. The empirical reality of dev-primitives exposes the limits of that view. Swapping prompts or upgrading model generations inside flat execution loops delivers marginal gains on repo-scale work, while restructuring the harness around code topology produces nearly 5× improvements. In complex software development, harness engineering is graph search—traversing, bounding, and repairing a network of interdependent AST nodes.

The decision of whether to invest in decentralized dev-primitive architectures comes down to an operational heuristic based on repository depth, cyclomatic complexity, and test harness maturity:

IF (Cyclomatic Coupling > Low) AND (Files Touched > 10) AND (Deterministic Test Suite == Available):
    Deploy Decentralized Primitive Harness (LSP-Bounded Nodes + Frontier Diagnostician)
ELSE:
    Retain Monolithic Terminal Agent (Aider / Devin / OpenHands)

For routine maintenance, localized bug fixes, and exploration across loosely coupled scripts, monolithic CLI agents like Aider or Devin remain the right choice. They impose zero indexing overhead, tolerate messy environments, and navigate arbitrary bash tooling without upfront structural parsing.

However, once you cross the threshold into deep architectural refactors—where single changes propagate across dozens of interdependent modules and cyclomatic complexity is high—monolithic harnesses collapse into context saturation and semantic drift. Investing in graph-coupled dev-primitives becomes essential precisely when a repository possesses a mature, deterministic test suite capable of guiding diagnostic blame. Without solid test suites, decentralized primitives lack the execution feedback required to converge.

Moving forward, the defining advantage in autonomous software engineering will not belong to teams writing clever system prompts or passively feeding multi-million-token contexts into central models. It will belong to operators who treat codebases as distributed multi-actor topologies. When you decompose a codebase into isolated, stateful nodes that respect compiler boundaries and negotiate typed contracts along dependency edges, you stop fighting context degradation. You transform raw inference into an orchestrated, self-healing execution graph.

Resources

• [2610.07832] Harness Engineering for Software Engineering via Modular Executable Dev-Primitives