Agent Harnesses

Mehran Mozaffari·Updated 1 September 2026

13 resources0 related posts

A Harness Is the Difference Between a Model That Can Act and a Model That Acts Reliably

Strip the marketing away and a harness is the software around a language model that turns "a model that could in principle act" into "a system that does act, and whose actions you can constrain." It is the agent loop: assemble context, call the model, parse the action, check it against a permission policy, execute it inside some confinement boundary, feed the observation back, and manage the growing history until the model declares completion or you pull the plug. Anthropic's engineering guide on building agents states the load-bearing fact plainly: agents are typically "just LLMs using tools based on environmental feedback in a loop," and the implementation is often straightforward — which means the differentiator is everything the loop wraps: the agent-computer interface, the guardrails, the sandbox, the stopping conditions.

The definition matters because three adjacent categories get miscategorized into it constantly. Orchestration frameworks (LangGraph, Microsoft Agent Framework) give you durable state machines and routing but are not themselves harnesses — they schedule model calls; they do not own the terminal your agent types into. Tool libraries and protocols (MCP servers, tool registries) define what an agent can call but say nothing about the loop or the policy around the calls. And model providers' "agent modes" blur the line deliberately: a cloud agent that spins up its own container (Codex Web, Claude Code on the web) is a harness that happens to be rented by the minute. The practical consequence of the confusion is real: teams pick a framework when they needed a harness, then discover that durable execution and checkpointing do nothing about the actual hard problems — what the agent is allowed to touch, what happens when the context window fills, and how a human says no.

The field has also produced its own existence proof that harness design is consequential rather than cosmetic. METR's time-horizon research — the measurement of task length (as timed for human professionals) that agents complete at 50% reliability — shows capability doubling roughly every seven months, with the important caveat that their own measurements show models succeed on nearly 100% of tasks under four minutes but under 10% of tasks over four hours. The gap between those two numbers is not model skill; it is sustained coherence over an evolving context, which is precisely what a harness maintains or fails to maintain. When METR updated the methodology in Time Horizon 1.1 (January 2026), the measurement got a larger task suite but the frame stayed the same: harness and model are evaluated together, and you cannot attribute a score to one without the other.

The Anatomy Is Converging: Loop, Context Manager, Permission Gate, Sandbox

Read enough harness code and the architecture collapses into five components. The model adapter normalizes provider APIs (LiteLLM in mini-swe-agent and smolagents; native clients in the vendor harnesses). The context manager decides what enters the prompt: system instructions, repository memory files like CLAUDE.md and AGENTS.md, tool schemas, and the running history — including whether the history is truncated, summarized, or left linear. The action interface is what the model emits: structured tool calls, or plain shell commands, or executable code. The permission gate sits between the model's decision and the machine: approval prompts, allowlists, or full autonomy. The execution boundary confines what an approved action can reach: nothing at all (raw host), Docker containers, or platform sandboxes (macOS Seatbelt, Linux namespaces — the native sandboxing the Rust terminals ship). Recovery machinery — checkpointing, resumable runs, compaction — wraps the whole thing.

flowchart TD
    TM[Task from human or automation] --> CM[Context manager<br/>instructions, memory files, tool schemas, history]
    CM --> MA[Model adapter<br/>provider API via one client]
    MA --> PS[Parsed action<br/>tool call, shell command, or code]
    PS --> PG{Permission gate<br/>allowlist or approval prompt}
    PG -->|denied| FB[Refusal fed back as observation]
    PG -->|allowed| SB[Execution boundary<br/>host, container, or native sandbox]
    SB --> OB[Observation<br/>stdout, diffs, test output]
    OB --> CM
    FB --> CM
    CM --> RC{History pressure<br/>over threshold}
    RC -->|yes| CP[Compaction or checkpoint<br/>summarize, persist, resume]
    RC -->|no| DONE[Stop condition<br/>task complete or budget spent]
    CP --> MA

Two design axes separate every harness on the market. First, the action interface: SWE-agent's research line spent 2024 building bespoke agent-computer interfaces — purpose-built file viewers and edit tools — and the follow-up mini-swe-agent threw them away, betting that a model given nothing but bash does as well or better. Open Interpreter has taken the same logic to its endpoint: it is literally a Rust fork of Codex whose /harness command swaps between emulated harness personalities (claude-code, kimi-code, qwen-code, deepseek-tui, swe-agent, minimal), because which action interface wraps a low-cost model changes its performance as much as changing the model. Second, statefulness of execution: most terminal harnesses keep one live shell session, while mini-swe-agent executes every action with an independent subprocess.run — its maintainers call the removal of the stateful session "a big deal" for stability, because a shell whose working directory, environment, and virtualenv have silently drifted from what the model believes is a classic source of unreproducible failures.

The permission gate is where harness philosophy becomes visible to users. Claude Code's permission modes, Codex CLI's sandbox and approval flags, and goose's extension-level permissions all implement the same trade: interruption cost per action versus blast radius per compromise. The security-relevant fact is that this gate is the only thing standing between a model that has read attacker-controlled text and your machine — every harness that offers a "just approve everything" escape hatch is offering a mode where prompt injection becomes arbitrary code execution. That is not a hypothetical composition; it is the default configuration many tired operators drift into, which is why the failure-mode section below treats permission fatigue as a first-class bug.

The Terminal Harnesses: Four Products, Four Different Bets

Claude Code is the bet that the harness should be a product surface, not a library: one closed-source engine delivered across terminal, VS Code, JetBrains, desktop app, and web, with repository conventions carried in CLAUDE.md, reusable workflows packaged as skills, and shell hooks that run before or after actions. Its docs pitch composability — pipe logs into it, run it in CI, schedule recurring routines in the cloud — and an Agent SDK for embedding the same tool set in your own agents. The bet's strength is coherence: memory, permissions, and subagents behave identically on every surface. Its cost is the corollary: the harness is only fully itself inside Anthropic's subscription, and the source is not yours to inspect or fork.

Codex CLI is the same bet played open: a Rust coding agent under Apache-2.0, installed by a one-line script or npm, authenticated either by signing into a ChatGPT plan or by an API key. OpenAI open-sourced the harness while keeping the model relationship commercial, which produces the most interesting licence-and-cost shape in the field: the harness is free to fork (Open Interpreter proves it, being exactly that fork), while the supported path bills through a ChatGPT subscription. The repository's own scale — well over a hundred thousand stars at the time I read it — plus its skills directories and devcontainer infrastructure show a harness being run as both product and open-source project simultaneously.

goose is the bet that the harness should be neutral infrastructure. It is Apache-2.0, written in Rust, runs as a desktop app, CLI, and API, speaks to 15-plus model providers including local Ollama, connects to 70-plus extensions through MCP, and — the governance fact that matters — now lives under the Agentic AI Foundation at the Linux Foundation. The position here: if you need a harness that outlives any single vendor's pricing decisions, this is the one whose stewardship is structurally designed for it. The tradeoff is that neutrality is a ceiling as well as a floor; goose does not have a vendor's incentive to tune one model's harness-specific performance to the last percent.

Aider is the specialist's bet: Python, Apache-2.0, 6.8 million installs, and a feature set — the repository map built from tree-sitter, automatic git commits with sensible messages, lint-and-test loops after each edit — aimed at one job, pair programming inside an existing codebase. It is the harness I reach for when the task is surgical edits to a large unfamiliar repository rather than open-ended agentic work. Its longevity (it predates the current harness boom) and its public leaderboards give it an evidence culture the newer entrants lack.

Open Interpreter deserves separate treatment because it is the field's first meta-harness: Apache-2.0, Rust, and explicitly "a fork of OpenAI's Codex with a focus on emulating the agent harness that gets the best performance out of low-cost models." Its /harness switcher and its reimplementation of provider-recommended harnesses (the Kimi Code harness, for instance) make the industry's open secret operational: providers tune harnesses for their own models, and mismatched harness-model pairings leave measurable performance on the table. Its portability stance — shared AGENTS.md, shared .agents/skills directories, ACP compatibility, Codex SDK wire compatibility — is a deliberate anti-lock-in position, and it is the clearest evidence that harnesses are becoming a portable, comparable layer rather than a product feature.

The Research Line Taught the Field the Opposite Lesson Twice

The SWE-bench team's two harnesses bracket the field's intellectual history. SWE-agent (Princeton/Stanford, NeurIPS 2024, MIT) demonstrated that agent-computer interfaces matter: purpose-built tools for viewing and editing files took GPT-4-class models from flailing with raw shells to state-of-the-art SWE-bench runs, and the paper's framing — that tool design deserves the same effort as human-computer interface design — was absorbed by everyone, including Anthropic's guide, which reports spending more time optimizing tools than prompts on its own SWE-bench agent.

Then the same team built mini-swe-agent and reversed the thesis. Version 2 is roughly one hundred lines of Python, carries no tools but bash, does not even use the model's tool-calling interface, keeps a completely linear message history, and executes each action as an independent subprocess — and it scores above 74% on SWE-bench Verified, with the team's own recommendation now being to prefer mini over SWE-agent going forward. The FAQ's motivation section is the honest sentence in the whole field: as models got more capable, most of the specialized scaffolding stopped earning its complexity, and a scaffold that puts "the language model rather than the agent scaffold in the middle of our attention" is both a better baseline and a better fine-tuning target because trajectory and messages are identical. My position: both lessons are true simultaneously — bespoke interfaces mattered when models were weak, and the minimal loop wins now — and any harness evaluation that does not state which regime it is testing in is not interpretable.

OpenHands completes the research-to-product arc, but with a twist worth watching: the repository has repositioned as Agent Canvas, a self-hosted control center (MIT, TypeScript, in beta) that runs its own open agent plus Claude Code, Codex, Gemini, or anything speaking the Agent Client Protocol, across local, Docker, VM, or OpenHands Cloud backends. The architecture has cleanly separated the agent-server SDK from the canvas frontend, and the README's own warnings are refreshingly blunt — running without the sandbox "will have full access to your filesystem." The position: OpenHands is becoming the harness-agnostic cockpit, which is a different and arguably more defensible niche than being yet another agent.

The Framework Layer Kept the Graph and Lost the Cloak

Three frameworks mark the boundaries of what a harness is not. LangGraph (MIT, from LangChain) is a low-level orchestration framework for stateful, long-running agents: durable execution that resumes after failure, human-in-the-loop interrupts, memory abstractions, and a paid observability and deployment platform (LangSmith) attached. It is trusted in production at recognizable companies, and its honesty about being infrastructure — inspired by Pregel and Apache Beam, usable without LangChain — is credible. The position: LangGraph solves the persistence and coordination problems a terminal harness ignores, and does not solve the confinement and permission problems a terminal harness treats as central; confusing the two is the category error this dossier opened with.

AutoGen (Microsoft Research lineage, 60k-plus stars) is the cautionary tale about framework churn. The repository now carries a maintenance-mode banner in orange: no new features, community-managed, with new users pointed to Microsoft Agent Framework — the enterprise successor with stable APIs, multi-provider support, and cross-runtime interop via A2A and MCP, plus a migration guide for existing AutoGen users. The licence situation is itself a small lesson in reading before adopting: GitHub classifies the repository as CC-BY-4.0 (that is the documentation licence), with a separate LICENSE-CODE file carrying the code licence — a dual structure that automated compliance scanners routinely misreport. The position: AutoGen pioneered multi-agent conversation patterns and its AgentChat API remains a fine teaching surface, but starting a new production project on a framework whose own README says "maintenance mode" is choosing migration as a future deliverable.

The permission gate is easier to reason about as a runtime conversation than as a box, because every harness implements the same negotiation between the model's intent, the operator's policy, and the confinement boundary:

sequenceDiagram
    participant H as Human operator
    participant L as Agent loop
    participant M as Model
    participant G as Permission gate
    participant S as Sandbox or host
    L->>M: context plus observation history
    M-->>L: proposed action (edit a tracked file)
    L->>G: check action against policy
    alt action allowlisted
        G->>S: execute without interrupting the human
    else action needs approval
        G->>H: approval prompt with command and diff
        H->>G: approve once or allow for session
        G->>S: execute approved action
    else action denied
        G-->>L: denial returned as observation
    end
    S-->>L: stdout, diff, test results
    L->>M: append observation to history
    Note over H,S: the gate is the only component that can say no<br/>once bypassed it cannot be re-asked mid-action

smolagents (Hugging Face, Apache-2.0) is the framework that agrees with the minimalists: roughly a thousand lines of code, first-class CodeAgents that write their actions as executable Python rather than JSON tool calls, sandboxed execution outsourced to E2B, Modal, Docker, or Blaxel, and model-agnostic plumbing through LiteLLM or local transformers. Its position in the market is pedagogical and practical at once — it is the codebase to read when you want to understand what a harness actually is, and a legitimate production choice when you want CodeAgent semantics without adopting a vendor's product surface.

Project	Steward	Licence	Shape	Model lock-in	Cost shape	My position
Claude Code	Anthropic	Closed source	Terminal, IDE, desktop, web, cloud routines	Anthropic models	Subscription per seat, or API tokens	Best product coherence; you do not own it
Codex CLI	OpenAI	Apache-2.0	Terminal, IDE, app, web	OpenAI via plan or key	Free binary; ChatGPT plan or token metering	The open harness to fork or standardize on
goose	AAIF / Linux Foundation	Apache-2.0	Desktop, CLI, API	15+ providers, local via Ollama	Free; bring any key or subscription	Neutrality as strategy; governance built to outlive vendors
Aider	Paul Gauthier + contributors	Apache-2.0	Terminal pair programmer	Any, best with frontier models	Free; token metering	The precision tool for edits in big repos
mini-swe-agent	Princeton/Stanford team	MIT	CLI and batch research harness	Any via LiteLLM	Free; tokens dominate; ~100 LOC to maintain	The baseline every harness should be measured against
SWE-agent	Princeton/Stanford team	MIT	Research harness, configurable via YAML	Any	Free; superseded by mini in the team's own guidance	Historic importance; use for tool-interface experiments
OpenHands Agent Canvas	OpenHands (All Hands AI)	MIT	Self-hosted control center + agent server SDK	Any, ACP-compatible agents	Free self-host; commercial cloud tier	The harness-agnostic cockpit to watch
Open Interpreter	Open Interpreter team	Apache-2.0	Rust terminal, harness emulator	Low-cost models focus (Kimi, DeepSeek, GLM, Qwen)	Free; tokens; open protocol compatibility	Proof that harness choice is a portable, swappable layer
smolagents	Hugging Face	Apache-2.0	Python library, CodeAgent	Any via LiteLLM/transformers	Free; tokens; sandbox runner fees (E2B/Modal) if used	Read it to learn harnesses; use it for code-action agents
LangGraph	LangChain	MIT	Orchestration framework	Any	Free library; LangSmith platform paid	Right tool for durable state, wrong tool for confinement
AutoGen → MAF	Microsoft	Docs CC-BY-4.0, code licence separate (LICENSE-CODE)	Multi-agent framework	Multi-provider	Free; migration engineering is the real cost	Do not start new production work here; migrate or use MAF

What Actually Changed in the Last Eighteen Months

Five dated shifts altered adoption decisions rather than feeding a news cycle. July 2025: the SWE-bench team announced mini-swe-agent hitting 65% on SWE-bench Verified in about a hundred lines — the moment the minimal-loop thesis got its number (the README now claims above 74%). March 2025 into 2026: METR's time-horizon measurement landed, then Time Horizon 1.1 (January 29, 2026) refreshed the suite; the seven-month doubling became the field's standard capability yardstick, and SWE-bench Verified-based replication showed an even faster doubling under three months. 2025: OpenAI released Codex CLI as Apache-2.0 Rust, and the open-source ecosystem responded within the year — Open Interpreter rebased itself onto Codex as a harness-emulating fork, which is faster convergence than any interoperability committee has managed. 2026: OpenHands completed its repositioning from agent to control center, adopting ACP so Claude Code, Codex, and Gemini agents run in its canvas; the same year, goose moved under the Agentic AI Foundation at the Linux Foundation, making harness governance an actual foundation project rather than a corporate freebie. And December 2024's Anthropic agents essay, whose "simple, composable patterns beat complex frameworks" conclusion, aged into the field's default wisdom — the team's own note that "much of the tooling landscape described in this post has changed" is itself the pattern: harness advice now has a shorter half-life than model releases.

Choosing by Constraint, Not by Preference

Licence first, because it eliminates options faster than features. Every harness in the table above is MIT or Apache-2.0 except two: Claude Code, which is a closed product you license through use, and AutoGen, whose repository mixes a CC-BY-4.0 documentation licence with a separate code licence — read LICENSE-CODE before your compliance team's scanner reads it for you. If your organization requires auditable, forkable control planes (regulated industries, on-prem everything), the Apache/MIT terminal harnesses with self-hosted backends — Codex CLI, goose, OpenHands Canvas — are the only candidates, and vendor cloud surfaces are out regardless of quality.

Cost shape second, because per-seat subscriptions and token metering behave differently at scale. A five-engineer team on subscription-billed harnesses (Claude Code via a Claude plan, Codex via ChatGPT plans) has predictable monthly costs and ceiling friction on heavy days; the same team on bring-your-own-key harnesses pays tokens that scale with task length — and METR's own data implies the token bill per task is growing as models attempt longer work. The underdiscussed number is engineering time: running SWE-bench-style batch evaluations on mini-swe-agent is cheap because nothing needs maintaining; running an equivalent matrix through LangGraph plus LangSmith means paying platform fees and carrying orchestration expertise; migrating off AutoGen to Microsoft Agent Framework is a real project with a real guide and real hours. Sandboxing runners are a third cost line smolagents users know well: E2B and Modal bill per sandbox-second, which turns "we ran the agent in a fresh container each time" into a line item.

Data sensitivity and confinement third. Terminal harnesses on a raw host give the model your whole machine unless you configure otherwise — OpenHands's own README warns that the no-sandbox option means full filesystem access, and mini-swe-agent's environment docs enumerate Docker, Podman, Singularity/Apptainer, and bubblewrap precisely because research code needs reproducible confinement. If the agent will read untrusted content (issues, web pages, dependencies), the harness must have a real execution boundary and a permission gate you do not override out of impatience; if it only ever sees your own code and writes to a container, the calculus loosens. Match the harness to the threat model, not to the demo.

Where It Breaks

Context exhaustion and compaction failures. Long tasks fill windows; harnesses summarize or truncate; the model loses constraints it was given hours earlier. Trigger: any multi-hour task with a large repository — the regime where METR measures under-10% success. Symptom: repeated edits that undo earlier work, re-reading files already read. Mitigations are harness-specific (Claude Code auto-compaction and memory files, mini's deliberately linear history that makes the loss visible rather than hidden), but no harness has made the problem disappear, because it is not a plumbing problem — it is the capability frontier itself.

Permission fatigue flipping into full access. The gate works until it is annoying; the operator enables "always allow" or runs with the equivalent of Codex's danger-full-access mode; from then on the permission gate is decorative. Trigger: high-frequency approval prompts during a long refactoring session. Consequence chain worth internalizing: with full access, a model that has ingested attacker-controlled text (a malicious README, a poisoned dependency page, a hostile issue comment) can be instructed to exfiltrate secrets or run destructive commands, and nothing in the harness will interrupt it.

Stateful-session drift. Harnesses that keep one long-lived shell accumulate divergence between the session's actual state (working directory, activated virtualenv, exported variables, background processes) and the model's belief about it. Trigger: long sessions with directory hopping and environment activation. This is precisely the failure mini-swe-agent's independent-subprocess design exists to kill, at the cost of losing shell state entirely — a trade the maintainers consider obviously correct and some interactive users still find sterile.

Sandbox escape hatches on the platforms you actually run. Confinement quality is uneven across operating systems; native sandboxing is strongest on macOS and Linux, Windows paths historically lag, and container-based confinement (Docker on macOS) has its own file-watching and performance friction. Trigger: teams standardizing on the platform the harness team tested least. The OpenHands repository carrying a dedicated Windows README is the tell.

Benchmark-harness coupling. Harness-specific tool interfaces inflate benchmark scores without transferring to your task. Trigger: choosing a harness because it topped a leaderboard run with its own tuned interface and model pairing. The SWE-agent→mini arc is the controlled experiment: when the interface was stripped out and the score held, the interface's contribution was revealed to be regime-dependent. Demand harness-held-constant evidence before attributing gains to the model, and model-held-constant evidence before attributing them to the harness — Open Interpreter's harness switcher exists because vendors themselves tune harnesses per model, so public numbers rarely isolate either.

Framework churn. AutoGen's maintenance-mode banner is the live example; earlier generations of agent frameworks vanished entirely. Trigger: production dependence on a framework whose roadmap belongs to a corporate strategy rather than a foundation. The goose/AAIF move is the industry's structural answer — governance designed to survive the founding vendor's loss of interest — and it is a legitimate selection criterion, not trivia.

Open Questions

The field cannot currently answer which matters more, harness or model, in the regime buyers care about: interactive, hours-long work on real repositories. The mini-swe-agent evidence says scaffolding complexity stopped paying once models crossed some capability threshold; Open Interpreter's whole product says harness-model fit still swings outcomes for the low-cost models most teams actually deploy at volume. Both can be true — harness choice matters more below the frontier and less at it — but nobody has published the crossover point.

Interoperability is consolidating around three standards — MCP for tools, ACP for editor-to-agent connections, AGENTS.md for repository conventions — and whether that triad holds or one absorbs the others is unresolved. The strategic stake is real: a harness that speaks only proprietary surface conventions (skills directories, memory file formats) is a lock-in vector no matter how open its source license is, and Open Interpreter's portability manifesto is the opening argument in that fight.

Finally, the evaluation question METR frames but does not close: time horizons are measured with specific harnesses, and the doubling trend conflates model improvement with harness improvement. If the harness contribution were separately measured — same model, minimal versus product harness, across the capability range — the field would know how much of the plotted progress is scaffolding and how much is the model. That ablation exists only in fragments (mini's README claims, DataCurve's DeepSWE comparisons where mini beat Claude Code and Codex), and the fragments disagree in interesting ways. Until someone runs it properly, every harness purchase decision is an act of informed faith.