Turning Production Coding Agents into RL Environments: The Mechanics, Failure Modes, and Economics of Capture Proxies

The Distribution Shift Trap in Agentic RL

Synthetic Gym sandboxes like SWE-gym and InterCode have long been the default testbeds for code-agent research. They standardize interactions into tidy env.step(action) -> (observation, reward, done) loops that plug neatly into standard reinforcement learning libraries. But in my experience, running production agents against real repositories looks nothing like a Gym environment. Real developer tooling—whether Anthropic’s Claude Code, OpenCode, or Hermes—operates within an intricate web of client-side scaffolding: dynamic terminal multiplexing, progress bars, interactive subshell timeouts, specialized diff formats, client-side history truncation, and elaborate system prompts.

When a model is trained exclusively inside an artificial Gym wrapper, it learns to optimize for that wrapper's stylized abstractions. The moment you transplant that policy into a production CLI harness, performance degrades drastically. The agent fails not because its code-synthesis reasoning is deficient, but because of severe train-test distribution shift. As I've explored before, the deterministic scaffolding around an LLM is the real architecture of production execution (see Beyond Prompt Coaxing: Why the Deterministic Harness Is the Real Architecture of Production AI Agents). If your policy never encounters the exact ANSI escapes, multi-file inspection patterns, and prompt structures native to your deployment harness, policy optimization ends up optimizing against an irrelevant artifact.

The operational answer is to treat unmodified production harnesses directly as the RL environment. Rather than writing brittle synthetic shims that attempt to approximate developer CLIs, you run the actual CLI binaries against real codebases and test runners. Training directly on the target harness forces the model to adapt to the authentic conversational topology, multi-turn error cascades, and context management strategies it will navigate in production.

The empirical upside of this alignment is striking. Benchmark evaluations on open-weight models like the LFM2.5 series demonstrate that training directly through unmodified multi-harness workflows can drive pass@1 on coding suites from roughly 42% up to 54%. Just as importantly, training against the real harness unlocks reward shaping that curbs tool thrashing. Because rollouts flow through the real tool loops, teams can penalize excessive exploratory steps without modifying harness source code. Published evaluations show this approach drives a 31% reduction in tool calls on problems the model already knows how to solve, teaching the policy to stop chasing circular bash commands and execute concise, targeted repairs.

Inside the Capture Proxy: Protocol Emulation, Token Extraction, and Logprob Interception

Plugging an unmodified CLI harness into an RL loop presents an immediate architectural hurdle: production agents like Claude Code, OpenCode, or Hermes expect to talk to remote inference APIs, while policy gradient algorithms require access to raw token IDs and sampling distributions. Rewriting the harness to expose Python tensors breaks the primary design requirement of maintaining zero upstream changes.

The architectural solution is protocol interception via a transparent capture proxy, typified by the OpenEnv and rllm-model-gateway patterns. Instead of altering the agent’s code, you redirect its network target by overriding environment variables like ANTHROPIC_BASE_URL or OPENAI_BASE_URL to point to a local proxy gateway. The agent CLI operates as an unmodified black box, executing bash commands, invoking tools, and parsing responses, completely unaware that its commercial API endpoint has been intercepted.

Behind this facade, the proxy conducts the heavy translation work necessary for policy gradient methods like Generalized Reinforcement Policy Optimization (GRPO) and PPO. When the harness issues a multi-turn completion or chat request, the proxy translates the incoming API schema (such as Anthropic Messages) into the native chat template of the underlying open-weight model. It then dispatches the request to a co-located vLLM engine. Crucially, the proxy intercepts and preserves the exact rollout telemetry that commercial APIs discard: raw prompt token IDs, completion token IDs, and the per-token log probabilities ($p_{\text{old}}$) under the current rollout policy $\pi_{\theta_{\text{old}}}$.

As the agent iterates through multi-turn bash runs and tool invocations, the proxy streams generated tokens back to the CLI to preserve responsive client-side timeouts, while silently buffering the step-by-step interaction graph. Once the harness concludes its task run and executes the validation test suite, the proxy pairs the final pass/fail outcome and tool-step counts with the collected trajectory buffers. It then packages the completed episode into an RL rollout batch and pushes it to an asynchronous trainer, such as Hugging Face TRL, to compute policy gradient updates against the advantage baseline.

sequenceDiagram
    autonumber
    participant CLI as Unmodified Agent CLI<br/>(Claude Code / Hermes)
    participant Proxy as Capture Proxy<br/>(OpenEnv Gateway)
    participant vLLM as vLLM Inference Engine
    participant Sandbox as Local Sandbox / Tools
    participant TRL as TRL Trainer<br/>(Async GRPO)

    Note over CLI,Sandbox: Multi-turn interaction loop
    CLI->>Proxy: POST /v1/messages (Anthropic API schema)
    Proxy->>vLLM: Forward token generation request
    vLLM-->>Proxy: Return tokens + raw logprobs (p_old)
    Proxy-->>CLI: Emulate Anthropic SSE stream
    CLI->>Sandbox: Execute bash command / tool call
    Sandbox-->>CLI: Return execution stdout / diff
    Note over CLI,Proxy: Repeats across N turns until task done
    CLI->>Sandbox: Run unit test verification
    Sandbox-->>CLI: Task evaluation result
    Proxy->>Proxy: Stitch complete trajectory & rewards
    Proxy->>TRL: Push trajectory batch (tokens, p_old, reward)
    TRL->>TRL: Execute GRPO policy gradient update

This decoupling dramatically shifts the operational lifecycle of agent training. When an upstream team updates a harness—revising prompt schemas, introducing sub-agent workflows, or adjusting error retry heuristics—the training infrastructure remains untouched. The capture proxy absorbs the protocol changes, preserving environment fidelity without requiring a single code change to the trainer.

Where Trajectory Reconstruction Breaks: Context Compaction and Tokenization Drift

While the capture proxy pattern cleanly isolates the harness from the trainer, treating an interactive CLI agent as a clean Markov Decision Process (MDP) rollout environment exposes severe operational fault lines. The most fragile failure mode stems from the tension between client-side context compaction and downstream trajectory reconstruction.

Real-world coding harnesses are aggressive optimizers of context window budgets. When an agent runs a verbose shell command that dumps thousands of lines of build logs, the harness rarely feeds that raw output indefinitely into future prompts. Instead, harnesses like Claude Code silently prune older terminal outputs, summarize prior conversation turns, or strip redundant tool histories between step $t$ and step $t+1$.

For an online RL trainer, this client-side pruning breaks basic Markovian trajectory assumptions. A standard policy gradient forward pass assumes that turn $t+1$ represents a strict append onto the prompt prefix evaluated at turn $t$. When the harness mutates historical context under the hood, the token sequence the trainer evaluates during policy updates no longer matches the rollout context that the inference engine actually conditioned on. Loss masking applied across historical prompt tokens misaligns, and calculating policy loss against the stitched sequence computes gradients on phantom transitions.

This divergence directly corrupts the importance sampling ratio central to GRPO and PPO:

$$\rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{old}}(a_t \mid s_t)}$$

If the proxy naively stitches turns together without accounting for client compaction, the trainer evaluates the numerator $\pi_\theta(a_t \mid s'_t)$ over a modified context $s't$, while the denominator $\pi{\text{old}}(a_t \mid s_t)$ reflects the original uncompacted prompt. This ratio drift triggers numerical instability, invalidating advantage weighting and frequently driving gradient norms to infinity.

An equally pernicious failure mode occurs at the tokenization boundary during schema translation. When an agent harness emits tool calls using Anthropic's XML tags or JSON blocks, the proxy must map that payload into the target model's native format (such as Qwen's <|im_start|> or Llama's <|tool_call|>), sample tokens via vLLM, and serialize them back into Anthropic Server-Sent Events. Because tokenizers are sensitive to whitespace and subword merges at tag boundaries, small translation discrepancies cause the raw generated token IDs to diverge from how the trainer's tokenizer parses the reconstructed dialogue string.

In GRPO, where rollouts are evaluated against a group-relative baseline without a separate critic network to absorb variance, even a single-token boundary mismatch causes importance sampling ratios to produce NaN values. Without strict proxy-level token logging that captures raw IDs directly from the inference engine rather than re-tokenizing string logs post-hoc, schema translation drift will quietly collapse policy training.

Architectural Tradeoffs: Capture Proxies vs. White-Box Harnesses vs. Gym Sandboxes

Choosing how to wire an agent harness into an RL loop forces an immediate operational choice between internal state visibility and harness agility. In my view, teams attempting agentic RL repeatedly pick the wrong abstraction layer because they treat the agent loop as either a simple Gym environment or an monolithic internal script.

White-box integrations—such as modifying OpenHands or SWE-agent directly to pass PyTorch tensors into an active TRL loop—offer maximum internal visibility. You get zero-copy, in-memory tensor access and can inspect every scratchpad buffer, dynamic few-shot retrieval pool, and intermediate parser state. If you want to modify internal reasoning loops directly, in-process hooks remain powerful (as I explored in Hacking the Agent Loop: An Operator's Guide to In-Process Claude Code Mods). But the architectural debt is brutal: you are permanently wedded to that specific codebase. The moment Anthropic updates Claude Code’s terminal interaction model or a new open-source CLI emerges, your training harness requires an invasive rewrite.

Synthetic Gym wrappers (like SWE-gym or InterCode) sit at the opposite extreme. They offer standardized Python interfaces (step(action) -> observation), but they achieve simplicity by discarding the messy, stateful realities of real developer tools. The resulting distribution shift leaves models incapable of operating within the ANSI-escaped, context-compacting realities of actual developer shells.

The transparent capture proxy (exemplified by OpenEnv and rllm-model-gateway) occupies the pragmatic middle. Its greatest strength is harness agnosticism. Because the harness interacts solely through standardized HTTP API schemas, you can train a single policy across Claude Code, OpenCode, and Hermes concurrently. When an upstream team modifies their CLI, your RL pipeline does not change.

Solution / Framework	Paradigm	Distribution Fidelity	Integration Friction	Internal State Visibility
SWE-gym / InterCode	Synthetic Gym	Low (synthetic bash loops; ignores CLI scaffolding)	High (harness logic must be ported to Gym interface)	High (full direct access to environment variables and tensors)
OpenHands / SWE-agent + TRL	White-Box Integration	Moderate to High (limited to the host harness design)	High (deeply coupled to harness-specific Python code)	Complete (in-memory access to prompts, scratchpads, and tensors)
Hermes / Atropos Pipelines	Offline Distillation	High (runs on native CLI logs)	Low (asynchronous post-hoc log parsing)	Moderate (limited to serialized disk logs; no on-policy steering)
rLLM / OpenEnv Gateway	Transparent Capture Proxy	Absolute (unmodified production harnesses and CLIs)	Zero (repoint target API base URL via environment variable)	Low (black-box; observes only API network payloads)

The architectural price you pay for the proxy pattern is twofold: network serialization overhead and state opacity. Serializing multi-turn exchanges across HTTP sockets introduces latency compared to in-memory tensor passing. More critically, the proxy treats the agent harness as a black box. If Claude Code implements hidden retry loops for malformed JSON, dynamic few-shot retrieval, or client-side caching, the proxy only sees the final API request that crosses the socket boundary. The policy can inadvertently learn to rely on the harness’s hidden correction heuristics rather than mastering robust single-step tool execution.

Async GRPO in the Wild: Policy Staleness, Sandbox Leaks, and Flaky Baselines

Running Generalized Reinforcement Policy Optimization (GRPO) on interactive coding agents creates operational friction that never appears in single-turn reasoning tasks. In mathematical reasoning rollouts, generation completes in seconds. In a coding harness, a single rollout episode can span twenty tool turns, multiple file diffs, compilation cycles, and test execution—frequently stretching wall-clock time between 60 and 300 seconds.

This latency profile creates an acute policy staleness problem. In asynchronous GRPO, the training engine continuously updates the policy weights $\pi_\theta$ while rollout workers are deep in execution. If an agent takes four minutes to compile dependencies and run tests, the policy version that generated those initial rollout actions ($\pi_{\theta_{\text{old}}}$) may lag several gradient updates behind the active trainer. Feeding these lagging trajectories into the policy gradient loss destabilizes the importance sampling ratio:

$$\rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{old}}(a_t \mid s_t)}$$

When the parameter divergence between $\theta$ and $\theta_{\text{old}}$ grows too wide, importance weights explode or collapse to zero, inducing catastrophic policy drift. In my setups, I enforce an explicit staleness threshold: any trajectory whose rollout parameters lag behind the trainer by more than $K$ gradient updates is discarded before advantage computation.

stateDiagram-v2
    [*] --> ProvisioningSandbox: Allocate Task
    
    state ProvisioningSandbox {
        [*] --> SpawnMicroVM: Create clean rootfs
        SpawnMicroVM --> MountWorkspace: Inject repo & tests
        MountWorkspace --> [*]
    }
    
    ProvisioningSandbox --> ActiveTurnLoop: Start Agent CLI
    
    state ActiveTurnLoop {
        [*] --> InterceptAPI: Intercept HTTP Call
        InterceptAPI --> ContextCheck: Inspect payload length
        ContextCheck --> TruncateBuffer: Context > Threshold
        ContextCheck --> DispatchInference: Context <= Threshold
        TruncateBuffer --> DispatchInference: Server-side truncation
        DispatchInference --> EmulateStream: Stream tokens to CLI
        EmulateStream --> ToolExecution: Execute shell / edit
        ToolExecution --> CheckCompletion: Task finished?
        CheckCompletion --> InterceptAPI: Next step (Turn < Max)
        CheckCompletion --> [*]: Complete / Terminated
    }
    
    ActiveTurnLoop --> TaskVerification: Run Test Suite
    
    state TaskVerification {
        [*] --> RunTests: Execute test runner
        RunTests --> ScoreOutcome: Compute verification reward
        ScoreOutcome --> [*]
    }
    
    TaskVerification --> StalenessFilter: Trajectory Complete
    
    state StalenessFilter {
        [*] --> CompareVersions: Check rollout delta K
        CompareVersions --> DiscardTrajectory: Delta > Max Stale Limit
        CompareVersions --> AdmitTrajectory: Delta <= Max Stale Limit
        DiscardTrajectory --> [*]
        AdmitTrajectory --> [*]
    }
    
    DiscardTrajectory --> CleanupSandbox
    AdmitTrajectory --> PushToTrainerBatch: Buffer tokens, p_old & reward
    PushToTrainerBatch --> CleanupSandbox
    
    state CleanupSandbox {
        [*] --> DestroyMicroVM: Wipe container & filesystem
        DestroyMicroVM --> [*]
    }
    
    CleanupSandbox --> [*]

The second failure mode stems from sandbox cross-talk. GRPO relies on group-relative baselines, sampling $G$ parallel rollouts (typically 4 to 8) per problem prompt to normalize rewards without a separate critic model:

$$A_i = \frac{R_i - \text{mean}({R_1, \dots, R_G})}{\text{std}({R_1, \dots, R_G})}$$

Because coding harnesses execute arbitrary shell operations—modifying configuration files, checking out git branches, installing conflicting Python wheels—reusing Docker containers or worker directories causes state leakage. A failed build from Rollout A can leave locked sockets, poisoned caches, or modified dependencies that cause Rollout B to fail entirely unrelated to its code changes. To protect the integrity of group advantage calculations, rollouts must run inside strictly isolated, ephemeral microVMs or container-per-rollout environments destroyed immediately upon task termination.

Finally, timeout-driven test flakiness corrupts group baseline calculations. If local environment load causes a valid solution to hit a 60-second test runner timeout, that trajectory is scored as zero. Within a small group of four rollouts, a single false negative artfully inflates the relative advantage of mediocre, incomplete rollouts that terminated early without timing out. To prevent the policy from optimizing for fast failure over thorough repair, unverified terminations must incur steep penalties, and rollout timeouts must be tracked separately from genuine assertion failures.

Reward Shaping Pitfalls: Tool Churn Penalties and the Premature Quitting Trap

One of the most compelling arguments for routing RL rollouts through an unmodified harness is the ability to shape intermediate interaction economics. In unconstrained rollouts, coding agents notoriously suffer from tool churn: running redundant ls checks, invoking circular grep searches, or executing repeated bash invocations on problems the base model already possesses the reasoning to solve. Introducing an efficiency penalty across tool invocations directly attacks this behavioral rot, with published benchmarks demonstrating that targeted step penalization can drive a 31% reduction in tool calls while preserving resolution accuracy.

Yet implementing this reward without inducing catastrophic reward hacking requires surgical precision. The primary pitfall I encounter in agentic RL is the premature quitting trap. In a naive implementation, an engineer adds a flat negative penalty for every interaction step:

$$R_{\text{total}} = R_{\text{verify}} - \lambda \cdot N_{\text{steps}}$$

On paper, this penalizes verbosity. In practice, the policy gradient exploits the arithmetic. Suppose passing the full integration test suite yields $+1.0$, while a failure yields $0.0$, and each tool turn incurs a penalty of $-0.05$. If an agent attempts an edit on turn two and encounters an unexpected compiler error, it faces an immediate expected-value dilemma. Persisting through five more turns of debugging, test running, and file rewriting risks an episode score of $-0.35$ if the fix fails. Conversely, executing an immediate exit command caps the loss at $-0.10$.

The policy quickly learns that early surrender minimizes negative step penalties far more reliably than attempting complex multi-step repairs. The agent stops exploring, aborts upon the first syntax error, or outputs an empty patch claiming the task is impossible.

To eliminate this pathological behavior, efficiency penalties must be strictly gated behind successful outcome verification:

$$R = \begin{cases} R_{\text{success}} - \lambda \cdot N_{\text{steps}} & \text{if unit tests pass} \ R_{\text{failure}} & \text{if unit tests fail} \end{cases}$$

Under a test-gated formulation, an agent receives zero efficiency credit for being brief if the underlying codebase remains broken. If the verification suite fails, the trajectory incurs a uniform failure penalty regardless of whether it took two turns or twenty. The model is only rewarded for minimizing tool calls after it has proven correctness. This decouples the incentive to economize context from the incentive to attempt recovery, forcing the policy to explore deep diagnostic paths while pruning only the truly redundant loops.

Engineering Blueprints: Practical Harness-RL Implementations

Moving harness-based RL from conceptual benchmark experiments into robust applied pipelines requires orchestrating three discrete systems: ephemeral sandbox runners, transparent capture gateways, and asynchronous policy gradient workers. When architecting these environments, the objective is to eliminate state contamination and protocol drift while preserving the native execution flow of developer tooling.

For organizations looking to optimize their own internal developer tools, an Automated Tool Efficiency Distillation pipeline offers an immediate production blueprint. In this setup, an unmodified internal developer CLI is redirected to a local rllm-model-gateway instance via the OPENAI_BASE_URL environment variable. The gateway streams rollout generation requests directly into a co-located vLLM engine, while the CLI runs locally against git repositories to execute automated unit test suites. The trajectory buffers—pairing captured token IDs and log probabilities with repo test outputs—stream directly into Hugging Face TRL to compute GRPO updates. To prevent the model from learning to exit immediately after encountering a single shell error to duck step penalties, the training loop must strictly gate length penalties behind verified unit test passes.

When teams need a base model that resists overfitting to one vendor's prompting conventions, the architecture can be extended into a Heterogeneous Multi-Harness Policy Generalizer. Here, a distributed worker pool runs containerized instances across diverse harnesses—such as Claude Code, OpenCode, and Hermes—against SWE-bench problem sets. Trajectories route through a central schema-normalizing capture proxy that dispatches inference across a unified Ray-managed vLLM cluster. The critical operational hazard to monitor here is tokenization boundary drift: mapping disparate Anthropic XML blocks and proprietary tool schemas into a shared open-weight chat template frequently misaligns whitespace and special tokens, which corrupts the rollout log probabilities and destabilizes policy updates.

Finally, for continuous code repair, teams can construct an Ephemeral MicroVM Regression Auto-Repair Trainer. This pattern captures failing production integration tests and spins up an unmodified coding agent inside an isolated Firecracker microVM for each rollout trajectory. The capture proxy records exact token IDs, per-token probabilities, and process exit codes, forwarding completed episodes into an asynchronous GRPO queue governed by strict parameter staleness bounds. Because GRPO relies on group-relative baselines across parallel rollouts, running multiple sibling trajectories within shared or reused containers causes state leakage: one rollout's left-behind build caches or modified lock files contaminate sibling environments, corrupting advantage normalizations through spurious cache hits. Enforcing disposable, container-per-rollout isolation ensures that every measured step reflects true policy reasoning.

Resources

• GitHub - huggingface/trl: Train transformer language models with reinforcement learning. · GitHub
• GitHub - rllm-org/rllm: Democratizing Reinforcement Learning for LLMs · GitHub