Code Mode vs Function Calling for Multi-Step Agents

The Context Collapse: Why Exposing Fifty API Tools Hits an Architectural Wall

I have watched agent orchestration evolve through two distinct scaling ceilings, and we are now crashing head-on into the third. In the early days of function calling, the mechanics were deceptively straightforward: you defined three or four tools as JSON Schema declarations and attached them to the model's inference payload. The model inspected the parameters, emitted a structured JSON payload, and your host harness executed the RPC call.

Once an enterprise application demands twenty, thirty, or fifty external capabilities—querying CRMs, updating issue trackers, pulling analytics, parsing file trees—the static function-calling pattern breaks down. First comes prompt bloat: registering dozens of verbose schemas consumes thousands of input tokens on every interaction turn before the user even finishes typing. More damaging is schema distraction. When an LLM faces an expansive list of candidate tools with overlapping parameter namespaces, selection accuracy drops precipitously. The model hallucinates argument keys, picks adjacent but incorrect tools, or fails to activate any tool at all.

To bypass this ceiling, the ecosystem turned to dynamic tool retrieval: indexing schemas in a vector store and injecting only the top candidate schemas via RAG on each turn. That solved initial prompt bloat, but introduced an equally painful operational bottleneck: sequential multi-hop latency. In real-world operator tasks, tools rarely run in isolation. If an agent needs to retrieve a cohort of users, fetch telemetry for each user, and push an alert for outliers, dynamic retrieval must re-evaluate relevance at every individual step. The agent spends hundreds of tokens and multiple round trips merely negotiating which tool schema to fetch next.

This friction makes collapsing the entire operational surface into a single code execution primitive the obvious architectural progression. Instead of juggling dozens of discrete JSON-RPC tools, we provide the agent with a single environment: an execution sandbox populated with typed client libraries.

sequenceDiagram
    autonumber
    actor User
    participant Model as LLM
    participant Harness as Agent Harness
    participant APIs as External Services
    participant REPL as Code Sandbox

    rect rgb(245, 245, 245)
    note over Model, APIs: Classic Multi-Turn JSON Function Calling (3 Round Trips)
    User->>Model: Execute user report workflow
    Model->>Harness: Tool Call 1: fetch_users()
    Harness->>APIs: HTTP Request 1
    APIs-->>Harness: HTTP Response (Raw JSON)
    Harness-->>Model: Prompt Update with Tool 1 Output
    Model->>Harness: Tool Call 2: fetch_metrics(user_id)
    Harness->>APIs: HTTP Request 2
    APIs-->>Harness: HTTP Response (Raw JSON)
    Harness-->>Model: Prompt Update with Tool 2 Output
    Model->>Harness: Tool Call 3: dispatch_alert(summary)
    Harness->>APIs: HTTP Request 3
    APIs-->>Harness: HTTP Response (Ack)
    Harness-->>Model: Prompt Update with Tool 3 Output
    Model-->>User: Final Task Completion
    end

    rect rgb(235, 245, 235)
    note over Model, REPL: Single-Turn Code Mode Execution (1 Round Trip)
    User->>Model: Execute user report workflow
    Model->>REPL: Run script (fetch, join, filter, dispatch)
    REPL->>APIs: Batch API calls executed in runtime
    APIs-->>REPL: In-memory JSON payloads
    REPL-->>Model: stdout: execution summary and status
    Model-->>User: Final Task Completion
    end

Context Efficiency and Token Density: Schema Bloat Against Typed Header Stubs

When you inspect how modern frontier models process structured schemas versus source code, the efficiency gap is staggering. Standard JSON Schema declarations are notoriously verbose. A typical function definition requires nested objects for parameters, explicit type declarations, nested property dictionaries, and an array of required field strings. Expressing fifty enterprise tools in JSON Schema routinely burns 4,000 to 8,000 prompt tokens purely on static scaffolding before a single conversational turn begins.

Exposing those exact same capabilities through TypeScript declaration files (.d.ts) or Python type stubs (.pyi) exploits the native pretraining distribution of modern LLMs. Models trained extensively on code repositories parse interface signatures, docstrings, and type unions with far greater density than deeply nested JSON. In practice, a typed interface stub conveys identical parameter constraints, optionality, and method documentation using 30% to 50% fewer tokens. More importantly, code-native types enforce stricter compositional awareness: an LLM understands enums, mapped types, and structured returns far more reliably when expressed as native code than when navigating repetitive schema boilerplate.

Paradigm / Architecture	Token Footprint (Tool Definitions)	Type-Checking & Parameter Validation Mechanics	Context Payload & Memory Behavior
JSON Schema RPC (OpenAI / Anthropic Native)	High (verbose nested dictionaries, property definitions, and required-field arrays)	Validated by host harness against JSON Schema specifications before execution	Full serialized API response payloads are dumped directly into conversation prompt context
Anthropic Model Context Protocol (MCP)	Moderate (dynamic server-negotiated tool schemas and prompt resource discovery)	Protocol-level parameter schema validation across client-server RPC boundaries	Responses stream back across protocol transport, but raw payloads still occupy prompt context
Typed Sandbox Stubs (Code Mode)	Lowest (concise TypeScript .d.ts or Python .pyi class and interface stubs)	Native language type checker or runtime interpreter verification within sandbox	Raw payloads stay resident in sandbox RAM or local disk; only filtered stdout reaches context

The more catastrophic failure of traditional function calling occurs on the return path. When a standard JSON-RPC tool executes an endpoint—say, querying an analytics warehouse or fetching thousands of customer records—the host application serializes that entire response payload and injects it back into the prompt context for the next inference turn. Dumping megabytes of raw JSON into the context window causes immediate context poisoning, consumes tens of thousands of tokens, and rapidly pushes the model toward attention degradation.

Under Code Mode, the payload boundary changes entirely. External API responses remain resident in sandbox memory. When an agent invokes a client library, the resulting JSON or tabular data deserializes directly into a runtime variable, disk file, or DataFrame. The agent uses standard language primitives—filtering arrays, calculating aggregates, or reshaping objects—and prints only the final distillation or completion status to standard output. By ensuring that raw intermediate data never crosses the network boundary into the prompt context, the token density of typed declarations radically shifts the economics of agent memory.

Latency and Control Flow: Single-Turn Interpreters Against Multi-Hop Inference Loops

The fundamental limitation of multi-tool agent architectures is not simply token usage; it is the latency tax of treating an LLM as an interpreted state machine. In traditional function calling, every procedural step requires a round-trip network hop to the inference endpoint. Consider a commonplace task: paginating across five pages of CRM records, filtering users who upgraded in the last week, querying an internal billing endpoint for each match, and dispatching a consolidated report.

In a classic tool-use architecture, this routine becomes an agonizing multi-turn dance. The model emits a call for page one, waits for host execution, reads the output, emits page two, and repeats. If fifteen matching users are identified, the model may execute fifteen separate serial tool calls to fetch billing data. Each intermediate step incurs prompt evaluation, generation latency, and network transport. A workflow involving twenty discrete API calls easily stretches into several minutes of clock time, compounding inference costs and multiplying points of failure.

In Code Mode, that entire operational sequence collapses into a single execution block. The model does not step through conversational turns to execute control flow. Instead, it writes a short script utilizing standard loops, list comprehensions, conditional branches, and error-handling blocks. The execution happens at native interpreter speeds inside the sandbox, completing in milliseconds what previously required dozens of inference turns.

In my experience evaluating production agent traces, imperative code remains the most expressive orchestration language ever invented. We spent years attempting to prompt models into following state-machine graphs or configuring complex declarative pipelines, yet models reason about control flow with far higher fidelity when writing raw Python or TypeScript. By offloading iteration, pagination, and data joining to an in-sandbox runtime, we eliminate conversational round trips entirely. As I've explored before regarding how Agent Memory Was an Indexing Problem All Along, shifting cognitive work from prompt-level gymnastics to structured execution environments is how we make autonomous systems practical in production.

Attacking the Silver Bullet: The Material Does Not Show Reliable Rollbacks on Partial Script Failure

While consolidating orchestration into a single execution sandbox slashes token overhead and eliminates latency hops, treating Code Mode as an unconditional replacement for structured function calling ignores a catastrophic fault line: distributed state mutability. In traditional tool architectures, the orchestrating harness stands between every discrete action. If a step fails, the host framework intercepts the failure before triggering subsequent operations, maintaining clear transactional boundaries and recording exact state transitions in the application database.

Under Code Mode, that mediation disappears. When an agent writes and runs an imperative script containing multiple external side effects—for instance, billing a customer via an SDK, recording the transaction in an internal database, and dispatching an email notification—it assumes happy-path completion. If that script crashes midway due to an unhandled socket timeout or database connection reset, external systems enter an indeterminate state. The payment gateway processed the charge, the database update never occurred, and the receipt was never queued.

The immediate operational breakdown is how models react to tracebacks. When the Python or Node runtime returns an unhandled exception, frontier models consistently struggle to deduce which line mutations committed permanently versus which operations never ran. Because the model lacks native saga orchestration, two-phase commit primitives, or compensational rollback hooks, its default corrective reflex is to re-execute the entire script from the beginning or blindly patch the failing statement without unwinding preceding side effects. In financial or mission-critical workflows, this behavior introduces duplicate charges, orphaned records, and divergent application state.

Unlike isolated RPC invocations where the host can guarantee idempotency tokens or wrap operations in database transactions, raw imperative scripts possess no inherent transactional rollback mechanism. When an agent writes ad-hoc control flow, it is effectively generating uncompiled, untracked distributed transactions on the fly. If you intend to run mutating actions within an interpreter, you cannot rely on the model to manually invent compensating transactions when an exception strikes. The harness or client SDKs must enforce strict idempotency keys and provide dry-run verification primitives directly inside the imported interfaces, preventing partial failures from corrupting downstream systems.

Threat Modeling the Sandbox: Egress Filtering, Privilege Boundaries, and Blind stdout Leaks

Deploying a single-tool execution runtime fundamentally transforms your agent's threat profile from an API parameter validation challenge into an untrusted code virtualization problem. In traditional tool setups, the model never directly executes system calls; it merely emits parameters validated by schema checkers before touching application logic. Once you give an agent a full REPL or container sandbox, any untrusted content entering the context window—scraped web pages, user-submitted documents, or customer records—becomes a potential code execution exploit vector.

The most acute vulnerability is secret exfiltration paired with unconstrained network egress. Naive sandbox architectures often inject production credentials and authorization tokens directly into environment variables. If an indirect prompt injection instructs the model to inspect os.environ or dump runtime configuration, the agent can exfiltrate every sensitive key. Worse, if the sandbox network namespace lacks strict egress filtering, an injected payload can issue outbound HTTP requests to attacker-controlled endpoints or probe local cloud instance metadata endpoints like 169.254.169.254 to steal ephemeral cluster role credentials. Production environments cannot rely on raw Docker containers; they require microVM virtualization using technologies like Firecracker or gVisor, coupled with strict eBPF or firewall rules blocking internal metadata endpoints and unauthorized host communications.

Operational hygiene breaks down inside the execution kernel as well. Teams opting for persistent, long-lived REPL sessions to avoid container boot latency inevitably confront kernel contamination. As an agent loops through multi-turn tasks, global variables get overwritten, open file descriptors leak, and child processes linger. A variable defined in turn one silently pollutes the scope of turn four, producing non-reproducible runtime bugs that degrade operational reliability.

Simultaneously, you must defend against the blind inspection trap. Models accustomed to standard development workflows routinely issue unconstrained data queries, printing raw responses directly to standard output. Dumping a twenty-megabyte JSON response or an unindexed table scan instantly obliterates the context window, causing catastrophic token waste and truncation crashes on the subsequent inference turn. Unclamped standard output streams completely dismantle the token-efficiency advantages of Code Mode. Production harnesses must implement rigorous stream clamping, capping stdout and stderr at strict thresholds like four kilobytes, spooling excess output to temporary disk storage, and instructing the model to inspect the resulting file programmatically.

The Production Boundary: Isolation Runtimes, Credential Proxies, and an Outside Verification Test

Taking Code Mode from an experimental harness to production requires treating the interpreter as an untrusted virtualization boundary. In my view, running arbitrary model scripts inside raw application processes or unconstrained Docker containers is operational negligence. A robust architecture demands sub-millisecond microVMs or hardened isolation layers—such as Firecracker microVMs, gVisor sandboxes, or Deno isolates configured with strict permission flags. At the operating system level, you must impose kernel-enforced cgroup boundaries: hard memory ceilings capped at 256MB to 512MB to prevent memory exhaustion from taking down runner nodes, paired with rigid wall-clock execution cutoffs of 15 to 30 seconds that terminate runaway loops before they consume host compute.

Credential isolation must be absolute. I never inject raw API keys, production tokens, or cloud credentials into the sandbox environment variables. Instead, all client SDKs inside the sandbox should communicate through a local Unix domain socket or a loopback HTTP proxy residing on the host. This proxy inspects each outbound call, validates the target destination against a strict host whitelist, and injects authentication headers out-of-band. To prevent the context-flooding failure mode, the runner must clamp standard output and standard error at four kilobytes; any excess stream is diverted to a local /tmp/run_output.txt file, returning a truncated preview that forces the model to inspect large datasets programmatically rather than dumping them into the prompt.

Before committing to this pattern, run an outside verification test: pit a standard multi-turn function-calling harness against a typed Python REPL sandbox across an identical 5-step mock ETL task (fetching a paginated user list, querying usage telemetry per user, calculating an aggregate tier score, writing to a staging table, and producing an executive summary). Code Mode slashes token overhead by over seventy percent while dropping execution clock time by an order of magnitude on the happy path. However, when you inject a simulated network timeout on step four, standard tool calling recovers cleanly through host-managed retries, whereas Code Mode requires explicit idempotency keys and structured compensation hooks inside the client SDKs to prevent duplicate staging writes.

quadrantChart
    title Tool Architecture Placement
    x-axis Low Orchestration Flexibility --> High Orchestration Flexibility
    y-axis Low Auditability and Blast Radius Control --> High Auditability and Blast Radius Control
    quadrant-1 Constrained Governance
    quadrant-2 Structured Control
    quadrant-3 High Risk Fragility
    quadrant-4 Freeform Execution
    Standard Function Calling: [0.25, 0.78]
    Model Context Protocol: [0.42, 0.88]
    Declarative Graphs: [0.35, 0.92]
    Code Mode: [0.92, 0.32]

Architectural Selection: Where Code Mode Wins and Where Structured RPC Must Remain

The debate over agent tooling often gets framed as an all-or-nothing displacement, but watching these systems run in production points toward a very different outcome. We are not watching function calling disappear; we are watching it retreat to its proper domain. When an agent is asked to perform batch transformations, cross-API joins, pagination, or dynamic log filtering, forcing every step through an LLM inference round trip is an anti-pattern that burns tokens and multiplies latency. For these data-heavy workloads, imperative scripts operating within typed execution environments represent the only sustainable design.

Conversely, treating a code interpreter as the universal interface for every operational action creates severe governance risks. In regulated enterprise environments, security compliance requires granular role-based access control, deterministic perimeter checks, and tamper-proof audit trails. A security team can easily inspect and authorize a discrete JSON call targeting a specific billing endpoint. By contrast, validating that an arbitrary block of Python or Node.js code will not trigger unintended side effects before execution requires deep abstract syntax tree analysis that is rarely foolproof.

Dimension	Standard Function Calling (JSON Schema)	Anthropic Model Context Protocol (MCP)	Declarative Graphs (LangGraph / DSPy)	Code Mode (REPL Sandbox)
Orchestration Latency	High (1 inference turn per individual action)	High (1 inference turn per individual action)	Medium (Pre-compiled graph, LLM at decision nodes)	Lowest (Multi-step operations run in a single script)
Token Efficiency	Poor (Repeated schema injection and full payloads)	Moderate (Dynamic resource and schema discovery)	High (Only active node inputs enter prompt)	Highest (Typed stubs and in-memory variable retention)
Payload Isolation	Poor (Full serialized responses dumped into context)	Poor (Raw JSON streams into prompt context)	Good (State managed via checkpointed references)	Best (Raw payloads remain in RAM or local disk)
Blast Radius Safety	High (Strict schema validation and host controls)	High (Granular protocol-level permission gates)	Highest (Hardcoded structural guardrails)	Lowest / Complex (Requires microVMs and egress firewalls)
Error Recovery	Clean (Host catches discrete failures at boundary)	Clean (Protocol standardizes error payloads)	Systematic (Explicit fallback routes and retries)	Messy (Unwinding partial script mutations is difficult)
Infrastructure Overhead	Lowest (Stateless serverless host execution)	Low (Direct stdio or SSE process transport)	Low-Medium (Standard application worker fleets)	High (Dedicated microVM sandboxes per session)

Because of these distinct trade-offs, the stable enterprise architecture collapses into a bifurcated model. Exploratory, compound, and read-heavy operations belong inside a single typed execution sandbox where the model can join, paginate, and transform data at native runtime speeds. Meanwhile, high-consequence side effects—modifying payment configurations, dropping tables, or dispatching customer emails—must remain behind structured protocol boundaries like MCP or guarded function calling, where human approvals and strict perimeter policies remain non-negotiable.

Resources

(no official sources were available to link)