Inside SPACE: Scaling Firecracker MicroVM Sandboxes for Production Agent Runtimes

The Architectural Shift From Shared Container Runtimes to MicroVM Sandboxing

When running autonomous agents in production, executing arbitrary generated code is an inescapable requirement. But the standard enterprise playbook—packaging workloads into standard Docker containers managed by Kubernetes—breaks down the moment multi-tenant agent execution enters the picture. Linux cgroups and namespaces were built to partition resources among cooperative processes sharing a single host kernel. They were never designed to serve as an adversarial security boundary against arbitrary, untrusted execution loops.

In an agentic loop, code generation is probabilistic. An agent might attempt low-level socket manipulation, mount-table inspection, kernel exploit probing, or memory exhaustion attacks. If multiple untrusted tasks share the host kernel, a single unpatched local privilege escalation vulnerability or kernel panic compromises the entire physical node. Furthermore, shared runtime environments make persistent file modifications hazardous: dirty states bleed across executions unless containers are ruthlessly destroyed and reconstructed, incurring significant lifecycle overhead.

Traditional Containers (Shared Kernel):
[ Agent Process A ]   [ Agent Process B ]
        │                     │
  (Namespaces)          (Namespaces)
        └──────────┬──────────┘
             [ Host Kernel ]  <-- Single point of compromise
             [ Bare Metal  ]

MicroVM Sandboxing (Hardware Isolation):
[ Agent Process A ]   [ Agent Process B ]
        │                     │
  [ Guest Kernel ]      [ Guest Kernel ]
  [ Isolated Rootfs]    [ Isolated Rootfs]
        │                     │
    (KVM / VMM)           (KVM / VMM)
        └──────────┬──────────┘
             [ Host Kernel ]
             [ Bare Metal  ]

This structural vulnerability is why Perplexity chose AWS Firecracker microVMs over container runtimes when architecting SPACE (Sandboxed Platform for Agentic Code Execution). By anchoring isolation in Linux Kernel-based Virtual Machine (KVM) hardware virtualization, SPACE assigns every sandbox its own unshared guest Linux kernel and private writable root filesystem. If untrusted code panics the guest kernel or consumes raw filesystem inodes, the blast radius is strictly contained within that microVM's virtualized boundary. The hypervisor host remains untouched.

To make microVMs practical for interactive agent loops, SPACE pairs Firecracker with an immutable template architecture. Rather than executing slow, cold operating system boots from scratch, sandboxes spawn from pre-built, immutable root filesystem templates. Ephemeral overlay filesystems give the microVM immediate write capability without dirtying the underlying template. This addresses the tension I regularly see in execution infrastructure: maintaining hypervisor-level security boundaries without accepting the multi-second startup penalties that destroy responsiveness in durable harness loops.

---

Control Plane vs Data Plane: Decoupling Lifecycle Polling From Live Execution

High-scale sandbox architectures inevitably fail when they conflate VM lifecycle management with process streaming I/O. If an agent holds an HTTP connection open to both spin up a virtual machine and stream back real-time standard output, intermediate proxies timeout, gateway connection pools exhaust, and lifecycle state machines lock up.

SPACE avoids this failure mode by enforcing a strict structural bifurcation between an asynchronous HTTP control plane and a streaming data plane.

sequenceDiagram
    participant Client
    participant ControlPlane as Control Plane (HTTP)
    participant DataPlane as Data Plane (SSE)
    participant MicroVM as Firecracker Guest

    Client->>ControlPlane: POST /v1/sandboxes (request instance)
    ControlPlane-->>Client: 202 Accepted (operation_id)
    loop Lifecycle Polling
        Client->>ControlPlane: GET /v1/operations/{operation_id}
        ControlPlane-->>Client: Status (pending)
    end
    Client->>ControlPlane: GET /v1/operations/{operation_id}
    ControlPlane-->>Client: Status (ready, sandbox_id)

    Client->>ControlPlane: POST /v1/sandboxes/{sandbox_id}/commands
    ControlPlane-->>Client: 201 Created (session_token, command_id)
    Note over Client,MicroVM: Process remains gated and unstarted until attachment

    Client->>DataPlane: GET /v1/sessions/attach (Bearer session_token)
    DataPlane->>MicroVM: Start process and stream stdio descriptors
    MicroVM-->>DataPlane: stdout and stderr chunks
    DataPlane-->>Client: SSE stream (output frames)
    MicroVM-->>DataPlane: Process exit
    DataPlane-->>Client: SSE event (process_exited, exit_code)

The control plane handles VM states—creation, pausing, suspension, waking, and deletion. Because microVM operations are inherently non-instantaneous, lifecycle mutations are completely asynchronous: requests immediately return an operation identifier that the client polls to verify state convergence. This prevents control-plane worker starvation during burst traffic.

However, the architecture introduces a critical behavioral detail that operators must understand: the two-step command execution trap. In SPACE, sending POST /v1/sandboxes/{id}/commands does not start the command inside the guest microVM. Instead, the control plane merely reserves the command specification and returns a scoped session attachment token. The process itself remains gated in an unstarted state until an active client establishes a streaming Server-Sent Events (SSE) connection to the data plane using that token.

If a caller creates a command via a decoupled worker and fails to immediately initiate the data-plane attachment stream, the execution freezes indefinitely, leaking allocated sessions and command slots without generating an exit code.

Furthermore, the data plane is strictly ephemeral. SPACE does not maintain an internal circular buffer to replay missed terminal output. If an ingress gateway drops the TCP connection mid-execution, those streamed stdout and stderr bytes are permanently gone. The platform only retains the final exit code for up to 15 minutes post-termination. For production agent runners, client applications must treat the data plane as an unbuffered pipe, multiplexing SSE chunks directly into long-term persistence stores while explicitly monitoring for the terminating exit event frame.

---

Evaluating the Isolation Spectrum: SPACE, E2B, Modal, and Daytona

Evaluating execution platforms for agent workloads requires balancing four conflicting vectors: the hardness of the isolation boundary, startup latency, hardware acceleration capabilities, and infrastructure ownership.

Dimension	SPACE (Perplexity)	E2B	Modal	Daytona
Isolation Primitive	Hardware microVM (AWS Firecracker via KVM)	Hardware microVM (AWS Firecracker via KVM)	User-space syscall kernel (gVisor)	OCI containers / standard Linux namespaces
Cold-Start Mitigation	Node-level warm pools and immutable base templates	Managed warm pools and custom guest daemons	Optimized container snapshotting and daemon reuse	Pre-built container images and dev environment caches
GPU Compute Support	None (CPU POSIX environments only)	None (focused on code interpretation and browser runs)	Native, multi-tenant GPU passthrough and serverless scheduling	Passthrough dependent on host Docker / Kubernetes drivers
Hosting & Control	Self-hosted (Kubernetes via kind/helm or bare metal)	Managed cloud API (with open-source components)	Fully managed proprietary cloud	Self-hosted or managed developer workspace engine

SPACE and E2B share identical foundational security posture: both run Firecracker microVMs over hardware KVM to provide genuine hypervisor-level isolation. However, their operational philosophy diverges sharply. E2B abstracts infrastructure behind a managed SaaS interface, giving developers SDKs with synchronous abstractions. SPACE, by contrast, is engineered from the ground up as a platform to deploy and govern within your own Kubernetes cluster or bare-metal environment via its administration CLI.

Modal takes an entirely different architectural path by pairing container runtimes with Google's gVisor. Rather than running a full virtual machine with a private guest kernel, gVisor intercepts Linux system calls in user space. This drastically reduces cold-start latency and enables native GPU passthrough—making Modal exceptional for agent tasks that require fast local model inference or distributed CUDA compute. But gVisor's syscall translation introduces performance overhead on disk-heavy workloads and does not provide the absolute hardware barrier of KVM virtualization.

Daytona addresses developer workspaces rather than ephemeral agent evaluation loops. Built around OCI containers and development toolchains, it excels at human-in-the-loop dev environments and Git integrations, but its default reliance on standard container namespaces makes it ill-suited for executing arbitrary, hostile agent code without secondary isolation layers like Kata Containers.

For teams running high-throughput, CPU-bound agent tasks within their own security perimeter, SPACE strikes a pragmatic compromise: full hypervisor isolation, direct node-level warm pool management, and zero vendor lock-in, provided you are prepared to manage bare-metal hosts with hardware virtualization enabled.

Lifecycle Primitives, Warm Pools, and the Memory Cost of Ephemeral States

In high-scale agent infrastructure, raw virtualization speed is only half the operational equation; lifecycle state management determines whether your host hardware economics survive. SPACE structures its microVM fleet around three primary operating states—Running, Paused, and Suspended—instantiated from immutable root filesystem templates.

Spawning a guest microVM cold from a base template requires initializing Firecracker vCPU threads, attaching disk overlays, and booting a dedicated Linux guest kernel. While this cold start takes only a few hundred milliseconds, serializing that delay across multi-step agent tool loops destroys interactive user experiences. SPACE mitigates this startup latency by maintaining node-level warm pools: pools of pre-initialized microVMs running on hypervisors, already booted and waiting for assignment.

stateDiagram-v2
    [*] --> ImmutableTemplate: Build rootfs
    ImmutableTemplate --> WarmPool: Pre-allocate microVM
    WarmPool --> Running: Acquire and attach session
    Running --> Paused: Pause vCPUs (preserve RAM)
    Paused --> Running: Resume command
    Running --> Suspended: Serialize state to disk
    Suspended --> Running: Wake microVM
    Running --> Terminated: Explicit DELETE
    Paused --> Terminated: Explicit DELETE
    Suspended --> Terminated: Explicit DELETE
    Running --> Terminated: Dead-man reaper (TTL expiration)
    Paused --> Terminated: Dead-man reaper (TTL expiration)
    Suspended --> Terminated: Dead-man reaper (TTL expiration)
    Terminated --> [*]: Reclaim KVM resources

However, warm pools introduce a severe memory footprint tax that operators must calculate carefully. Unlike standard container engines where Linux cgroups allow soft limits and dynamic page sharing across idle processes, KVM-backed microVMs require rigid host memory reservations. If you provision an execution template with 1 GB of guest RAM and maintain a modest warm pool of 64 standby instances per bare-metal node, that host sacrifices 64 GB of physical memory strictly to hold unallocated standby environments.

Managing running versus paused and suspended instances requires balancing this exact trade-off. In the Paused state, the hypervisor halts the microVM’s vCPU scheduling while keeping its memory space resident in host RAM. This allows virtually instantaneous response when an agent issues subsequent commands within an ongoing conversation. But if memory pressure mounts on the node, instances must shift to Suspended, where memory pages are written out and reclaimed, accepting a longer wake phase when the agent resumes execution.

The most dangerous operational failure mode here is the accumulation of zombie microVMs. If an upstream orchestrator crashes or drops an agent loop without dispatching an explicit DELETE call, paused and suspended sandboxes linger on disk overlays and control-plane state tables indefinitely. Because orphaned microVMs rapidly exhaust node storage and hypervisor memory bounds, production deployments cannot rely solely on polite client teardowns. They require an aggressive dead-man reaper running as a background controller, forcefully terminating any sandbox whose activity lease exceeds its maximum time-to-live threshold.

---

Where SPACE Breaks in Production: Ephemeral Streams, Exit Windows, and Ingress Traps

Deploying SPACE v0.8.0 into high-volume production reveals several acute architectural boundaries that platform engineers must remediate at the routing and client layers: unbuffered streaming data paths, tight exit-code retention windows, and reverse proxy timeouts.

The primary operational trap lies in the data plane’s lack of stream replay buffers. When an agent attaches to a command session via Server-Sent Events (SSE), standard output and standard error stream live from the microVM guest. SPACE does not persist an internal ring buffer or append logs to a database. If an intermediate network hiccup, client restart, or gateway blip drops the TCP connection mid-execution, every byte streamed during that disconnect is permanently lost. There is no offset-based reconnection mechanism. Any resilient agent implementation must terminate the SSE connection inside an internal telemetry proxy that writes output chunks into a durable log aggregator (such as ClickHouse or an object store) in real time before relaying frames to downstream consumers.

Compounding this streaming fragility is SPACE’s 15-minute exit code retention cutoff. Once a command terminates inside the microVM, the control plane preserves the process status, duration, and terminal exit code for only a fifteen-minute window. If your job orchestrator experiences queue backpressure, worker restarts, or polling retry delays that exceed this cutoff, the final outcome of that process becomes unrecoverable.

Furthermore, client parsers cannot assume that a terminated TCP connection implies process completion. Network drops, hypervisor panics, or guest out-of-memory kills will abruptly close the socket without warning. Robust callers must explicitly await and parse the terminating process_exited JSON event frame. Treating a dropped connection as an implicit completion masks fatal execution failures as silent successes.

At the network edge, SPACE’s split architecture creates friction with standard API gateways. Reverse proxies like Envoy, Kong, or NGINX enforce default connection timeouts—frequently set to 30 or 60 seconds. Because long-running agent commands may produce periods of terminal silence during compilation or package installation, intermediate gateways regularly sever the underlying SSE connection. Edge proxies routing data-plane traffic must be explicitly configured with infinite read timeouts, streaming pass-through (disabling response chunk buffering), and strict header isolation to prevent leaking scoped session attachment tokens across boundary redirects.

Finally, platform engineers must recognize that advanced filesystem state management in v0.8.0 remains experimental. While marketing discussions often treat microVM snapshotting, checkpointing, and branch rollbacks as table stakes for exploration loops, SPACE explicitly gates snapshots and file-watching sessions as incubating features. They are excluded from the default stability contract and disabled unless specifically configured per tenant. If your architecture relies on branching dirty environments to explore speculative code paths—a design challenge I examined when reviewing durable workflows over ephemeral scripts—you cannot depend on out-of-the-box snapshot restores in SPACE today; you must model state isolation by spinning up clean sandboxes from immutable base templates.

Production Deployments: Bare Metal, KVM Device Mapping, and Ingress Architecture

Hosting SPACE in a production Kubernetes environment is fundamentally different from managing standard container workloads. Because Firecracker executes code within genuine hardware-assisted microVMs rather than standard Linux namespaces, the underlying nodes must expose Kernel-based Virtual Machine access (/dev/kvm).

This requirement immediately invalidates standard, unprivileged managed Kubernetes worker pools. If you deploy worker nodes on AWS, Azure, or GCP using generic virtualized instances without nested virtualization enabled, Firecracker’s virtual machine monitor fails on initialization. In cloud environments, platform engineers must provision dedicated bare-metal instances—such as AWS c5.metal or i3en.metal—or compute shapes that explicitly permit nested KVM virtualization. At the Kubernetes cluster configuration layer, the node daemonset managing the Firecracker microVM lifecycle requires elevated host privileges to mount /dev/kvm directly into runtime pods alongside local network tap devices.

At the network edge, SPACE’s strict division between its control and data planes requires specialized reverse proxy tuning. Standard ingress controllers—whether Envoy, Kong, or NGINX—are optimized for short-lived REST transactions. By default, they enforce 30- to 60-second read timeouts and buffer HTTP response bodies to optimize downstream packet delivery. For SPACE’s data plane, this default posture is fatal. Long-running compilation commands or multi-step script executions often produce extended intervals of terminal silence. If intermediate proxies lack streaming keep-alives, they sever the underlying Server-Sent Events (SSE) connection prematurely.

Edge ingress routing must configure dedicated path matching for data-plane endpoints like /v1/sessions/attach. These routes require infinite read timeouts, disabled response chunk buffering (proxy_buffering off), and explicit HTTP keep-alive pings. Furthermore, ingress gateways must strictly isolate authentication scopes. Because control-plane operations rely on administrative Bearer tokens while data-plane streams require ephemeral, session-scoped attachment tokens, proxies must never forward administrative credentials across streaming redirects or data routes.

Finally, managing guest environments demands operational rigor around space-admin-cli. Because sandboxes are instantiated from immutable root filesystem templates, operators cannot patch system utilities, install security updates, or reconfigure guest runtime dependencies in-place. Base images are rigid artifacts. Production teams must treat system templates like container base images in an automated CI/CD pipeline: building root filesystems via space-admin-cli, executing automated vulnerability scans, and distributing verified rootfs images across node-local NVMe caches before updating template catalog pointers in the control plane.

---

Practical Implementations for Resilient Agentic Sandboxes

Translating SPACE from an open-source hypervisor orchestrator into a dependable runtime requires wrapping its core primitives in external resilience services. Because the platform intentionally avoids building heavy state management, persistence buffers, or autoscaling controllers into its microVM daemon, platform engineers must address these boundaries within their client and middleware tiers.

For architectures handling asynchronous, compute-intensive execution, implementing an Autonomous Data Analysis Worker with Durable S3 Log Mirroring provides an essential blueprint for handling unbuffered streaming. When an orchestrator provisions a Python data analysis microVM through the asynchronous HTTP control plane, it acquires a session token and connects an external ingestion proxy directly to the SSE data plane. Because SPACE maintains no stream replay buffer, this proxy multiplexes incoming stdout and stderr chunks in real time into an S3 bucket or high-throughput log store. Simultaneously, the worker monitors the stream for the terminating process_exited event payload rather than relying on socket closure, ensuring that unexpected guest panics or intermediate proxy drops are not mistakenly categorized as clean terminations.

When supporting interactive user-facing workflows, such as a Sub-Second Code Assistant with Dynamic Warm-Pool Autoscaling, startup latency becomes the governing operational metric. Spawning cold microVMs introduces multi-hundred-millisecond boot delays that degrade interactive pair-programming sessions. By provisioning bare-metal Kubernetes node groups managed via space-admin-cli and layering a custom warm-pool controller on top of SPACE's administrative API, teams can adjust standby microVM pool depth dynamically based on incoming queue volume. However, because KVM microVMs reserve physical RAM rather than sharing pages like standard Linux containers, over-provisioning warm instances quickly induces hypervisor host memory starvation. The controller must balance pool size against hard host memory cgroups, combined with an automated reaper service that destroys zombie microVMs when upstream orchestrators abandon idle environments.

Finally, organizations executing untrusted or adversarial user code can deploy a Multi-Tenant Security Testing Sandbox with Egress Inspection. Here, untrusted third-party scripts execute inside dedicated Firecracker microVMs booted from immutable system templates, completely isolating the host kernel from privilege escalation attacks. Each sandbox receives an isolated virtual network interface that funnels traffic through an external filtering proxy enforcing strict domain allowlists. When implementing this pattern, operators must navigate two core architectural boundaries in SPACE: the hypervisor's lack of GPU virtualization prevents running local machine learning inference workloads within the microVM, while its incubating snapshot APIs mean teams cannot rely on instant state rollbacks, necessitating clean sandbox teardowns after each test run.

Resources

• GitHub - perplexityai/SPACE: Sandboxed Platform for Agentic Code Execution: pluggable-runtime sandboxes for running agent and user code in isolation. · GitHub
• Introduction - SPACE Guide
• SPACE API