Why I Stopped Building Separate Stacks for Chat and Background Agents: The Durable Rails Architecture

The False Economy of Split-Brain Agent Infrastructure

When engineering organizations begin operationalizing agentic workflows, their instinct is almost always to bifurcate infrastructure. They establish one stack for synchronous, user-facing conversational assistants—typically FastAPI, WebSockets, in-memory state, and Redis—and a completely detached asynchronous architecture for background workflows, running on Celery workers, background queues, or scheduled batch jobs. On the whiteboard, this separation appears rational. Live conversational turns prioritize immediate token streaming and tight latency budgets, whereas multi-hour ledger reconciliations, invoice matching, and autonomous month-end financial closes require durable retries and background resiliency.

In practice, this dual-stack design is a false economy that accrues severe maintenance debt. When you run parallel execution tracks, you are forced to build every foundational capability twice. Tool registries must be duplicated across both frameworks. Identity management, RBAC boundaries, and resource entitlements inevitably drift out of sync between the live assistant and the background worker. Model evaluation, rate-limiting logic, fallback routing between model providers, and token spend telemetry end up implemented in two divergent codebases. Worse still, real-world product workflows constantly blur the boundary: an interactive chat user frequently requests a complex financial reconciliation that must transition into an unattended multi-hour background task, while an autonomous background job frequently requires interactive steering or approval from an operator when an anomaly occurs.

Bridging these two distinct paradigms with bespoke synchronization logic results in fragile glue code and fractured audit trails. I have found that standardizing on a single, unified execution harness anchored in durable distributed execution fundamentally simplifies enterprise governance. Rather than treating real-time chat and long-running background tasks as distinct paradigms, the harness treats every agentic operation—regardless of duration or surface—as an execution graph running atop the same runtime rails.

The upfront cost of adopting a durable execution platform is non-trivial, requiring strict determinism and deliberate lifecycle management. Yet consolidating onto a shared set of rails guarantees that every tool call, permission check, and context transformation is defined, versioned, and audited in exactly one place. As I explored when analyzing state machines, checkpoints, and durable execution, mission-critical agent reliability demands an execution foundation that can survive arbitrary worker crashes, API rate limits, and network partitions without dropping context. A unified harness provides that foundation across the entire organization, replacing ad-hoc distributed coordination with verifiable durability.

The Three-Tier Persistence Core: Temporal, PostgreSQL, and Object Storage

Operating a unified agent runtime requires dividing persistence responsibilities across specialized storage tiers. When an agent executes over dozens or hundreds of conversational turns, attempting to use a single database or orchestration state store for every persistence need leads directly to operational collapse. The architecture succeeds by establishing a clean three-tier division of labor across Temporal, PostgreSQL, and object storage.

sequenceDiagram
    actor User as User Client
    participant PG as PostgreSQL (Message Ledger)
    participant Temporal as Temporal (Workflow Orchestrator)
    participant S3 as S3 (Claim-Check Store)
    participant Worker as Worker Activity
    participant Model as Foundation Model API

    User->>PG: Insert user turn & interaction metadata
    User->>Temporal: Dispatch execution turn
    Temporal->>Worker: Schedule execution activity
    Worker->>PG: Fetch message tree & permissions
    Worker->>S3: Upload full context window & large attachments
    Worker-->>Temporal: Return claim-check URI (s3://...)
    Temporal->>Worker: Schedule model inference activity with URI
    Worker->>S3: Retrieve context window via claim-check URI
    Worker->>Model: Invoke model with tools schema & context
    Model-->>Worker: Stream tokens & emit tool call
    Worker->>S3: Store raw model output & intermediate tool results
    Worker->>PG: Record assistant turn in relational message tree
    Worker-->>Temporal: Complete activity with output URI pointer
    Temporal-->>User: Acknowledge execution completion

Temporal serves strictly as the orchestrator for execution flow and deterministic state recovery. It does not act as an application datastore. Instead, Temporal maintains the append-only event history required to guarantee replay safety. If an underlying worker node crashes midway through an agentic loop, or if a downstream model provider throttles requests under heavy load, Temporal orchestrates exponential backoffs, idempotent activity replays, and step continuation without losing execution state.

PostgreSQL functions as the operational message ledger and relational metadata index. It stores the conversational message trees, session metadata, parent-child subagent hierarchies, and tool access audits. Structuring this layer in PostgreSQL allows interactive web clients and administrative consoles to query conversation threads, inspect intermediate steps, and update message trees with high concurrency and low latency, completely isolated from Temporal's internal workflow state.

Object storage operates as the claim-check repository for heavy artifacts. Iterative agent loops generate massive context windows, dense OCR extractions, and multi-megabyte CSV files. Pushing these payloads directly through workflow orchestration parameters is a catastrophic failure mode. Orchestration engines rely on gRPC interfaces, which enforce strict maximum message size limits—typically capped at 4MB. Furthermore, writing large conversational transcripts directly into workflow history causes rapid history bloat, pushing executions against memory caps and degrading replay performance.

By applying the claim-check pattern, the harness ensures that worker activities write heavy prompts, raw model responses, and tool outputs directly to object storage, passing only lightweight, deterministic URI references through the orchestration engine. This guarantees deterministic replay performance while keeping workflow histories compact.

Decoupling Ephemeral Sandboxes from Long-Lived Session State

A critical vulnerability in agent system design is the conflation of conversation state with code execution environments. When agents perform autonomous accounting, financial close reconciliations, or tabular data extraction, they must routinely write and execute arbitrary Python, bash, or SQL scripts to calculate ledger adjustments, parse malformed invoices, or aggregate transaction histories.

Tying an agent's persistent memory or session lifecycle directly to the container executing that code is an anti-pattern. If a generated script encounters an unexpected memory leak, triggers an unhandled segmentation fault, enters an infinite loop, or gets evicted by the host node, tightly coupled architectures suffer fatal session termination. As I noted when dissecting autonomous workspaces in cloud storage, compute environments must remain strictly disposable artifacts that can be torn down and recreated without threatening the integrity of the underlying session.

In a decoupled harness, sandboxes are treated as disposable worker resources managed through isolated, ephemeral container environments. When an agent decides to generate and execute code, the orchestration layer dispatches an execution activity to an isolated sandbox instance. The sandbox receives only the explicit input parameters and tabular data slices required for that specific computation. It possesses zero persistent local authority over the broader session state.

Before a sandbox container terminates, any generated data outputs, transformed spreadsheets, or execution logs are deterministically flushed back to object storage, and the resulting structured outputs are committed to the message ledger in PostgreSQL. If a container hangs, strict cgroup CPU allocations, memory limits, and hard execution timeouts terminate the process safely. The failure is returned to the orchestrator as a standard, recoverable activity error, allowing the agent to analyze the stack trace and adjust its approach without corrupting the broader conversation tree.

To eliminate the cold-start latency penalties that would otherwise render real-time interactive chat unusable, the execution tier maintains a pool of pre-warmed sandbox containers. These warm runners sit idle with baseline runtimes loaded, locked down behind strict network egress restrictions to prevent unauthorized data exfiltration. When a code execution turn arrives, the harness immediately routes the workload to a warm container within milliseconds, providing responsive interactive execution while preserving absolute isolation between the disposable runtime and durable session state.

The Chat Latency Tax: Running Synchronous Dialogue over Durable Queues

The primary argument leveled against unifying conversational chat and background workloads on durable distributed queues is latency. In a naive implementation, that criticism is entirely valid. When you run an interactive chat turn through Temporal, you pay an architectural tax: task queue scheduling hops, worker polling intervals, state persistence transactions, and activity dispatch round-trips. In an in-memory WebSocket handler backed by Redis, token generation starts within tens of milliseconds. Running that same turn across an orchestration engine introduces 100 to 300 milliseconds of distributed coordination latency before the first token can stream.

The deeper threat to conversational performance, however, is queue contention and head-of-line blocking. If interactive chat turns share a task queue or worker pool with long-running accounting workflows—such as multi-megabyte CSV reconciliations, tabular ledger audits, or batch document extractions—the system quickly starves. Heavy, compute-bound activities monopolize worker slots, leaving conversational requests waiting on queue polls. In live dialogue, even a one-second pause destroys the user experience.

I address this trade-off by strictly isolating execution tiers while maintaining shared domain logic. The workflow runtime remains unified—sharing tool registries, model wrappers, and authorization logic—but physical worker fleets and task queues are partitioned. Interactive conversational turns run against a high-concurrency, low-latency queue served by pre-warmed, lightweight worker nodes configured with aggressive timeout thresholds. Heavy computational reconciliation runs are routed to separate, memory-optimized worker pools mapped to dedicated asynchronous queues. Furthermore, token streaming should bypass workflow state entirely: worker activities stream tokens out-of-band directly to the client via WebSockets or Server-Sent Events, persisting the final compiled turn to PostgreSQL and Temporal only upon completion. This keeps real-time responsiveness intact without forfeiting durable state tracking.

Architectural Paradigm	Latency (Interactive Turns)	Fault Tolerance & Durability	Operational Overhead
Unified Durable Orchestration (Temporal + Postgres + Sandboxes)	Moderate: 100–300ms scheduling overhead; mitigated by dedicated queues and out-of-band token streaming.	Exceptional: Deterministic replay, automatic resumption, zero state loss across worker reboots or outages.	High: Requires distributed systems expertise, workflow determinism discipline, and multi-tier storage management.
Graph Frameworks (LangGraph)	Low: Minimal runtime overhead with direct memory or checkpointer lookups.	Moderate: Relies on database snapshot checkpoints; in-flight process failures require manual recovery loops.	Moderate: Accessible Python/TypeScript SDKs; requires external orchestration to scale to long background jobs.
Custom Redis / Celery Stacks	Very Low: Optimized for sub-50ms conversational streaming and ad-hoc job queues.	Poor to Moderate: Prone to dropped tasks, uncoordinated worker restarts, and fragmented execution state.	High (Accrued): Inevitably reinvents a fragile, bespoke workflow engine over time to bridge sync and async boundaries.
Managed Provider APIs (e.g., Bedrock / Assistants)	Low to Moderate: Optimized by the provider for raw token-generation turnaround.	High (Opaque): Managed by vendor infrastructure, but internal failure modes and retries are black boxes.	Minimal: Low initial operational burden; introduces severe vendor lock-in, rigid tool limits, and zero VPC data control.

Where Event-Sourced Replay Collides with Agentic Loops

Durable execution engines achieve fault tolerance through deterministic event sourcing. When a worker crashes, encounters an activity timeout, or migrates between nodes, the workflow re-executes its code from the beginning, reconstructing its exact in-memory state by replaying every past event recorded in its append-only ledger. In conventional microservice workflows with a handful of static steps, this model is bulletproof. In iterative, multi-turn agent loops, it creates a severe distributed systems failure mode: event history bloat.

An autonomous agent does not execute a predictable sequence of five activities. A single reconciliation task might involve evaluating a prompt, dispatching an LLM call, parsing malformed structured output, querying a sandbox runner, validating ledger balance sheets, and dispatching a subagent. Over dozens of iterations, that single task generates thousands of discrete workflow events—timer registrations, activity task scheduled markers, completion records, and external message receipts.

Temporal enforces a hard ceiling of 50,000 events or 50MB payload per execution history, with non-configurable warnings beginning at 10,000 events. As event histories expand into the thousands, replay latency degrades exponentially because every restart requires sequentially processing the accumulated event backlog. If an iterative agent loop hits the 50,000-event hard cap, the orchestrator terminates the workflow unrecoverably.

stateDiagram-v2
    [*] --> ActiveExecution
    ActiveExecution --> ToolInvocation: Tool Call Dispatched
    ToolInvocation --> ActiveExecution: Activity Result Returned
    ActiveExecution --> HistoryThresholdCheck: Turn Completed
    HistoryThresholdCheck --> ActiveExecution: Event Count < 2000
    HistoryThresholdCheck --> StateCheckpointing: Event Count >= 2000
    StateCheckpointing --> ContinueAsNew: Context Snapshot Flushed to S3 & PG
    ContinueAsNew --> ActiveExecution: Reset Execution History
    ActiveExecution --> Completed: Task Goal Achieved
    Completed --> [*]

To prevent this collapse, I treat execution reset via ContinueAsNew as a mandatory architectural boundary. Instead of allowing a workflow run to accumulate unbounded history, the agent evaluates its event counter after every completed turn. When the execution approaches a safe threshold—typically between 2,000 and 5,000 events—the workflow serializes its consolidated state, flushes full conversation trees and large artifacts to S3, commits the relational index to PostgreSQL, and calls ContinueAsNew. This resets the event history to zero while launching a clean workflow instance that carries forward only the deterministic S3 pointer to its active state.

A compounding risk in long-running agent workflows is non-deterministic replay violations. Because Temporal reconstructs state by re-running workflow code against past events, any mid-flight alteration to workflow orchestration logic—such as tweaking an internal condition, altering an activity sequence, or re-ordering a loop—causes the replay engine to detect an execution divergence. The runtime immediately halts the workflow with a non-deterministic policy error.

For teams building autonomous agents that execute over days or weeks, code versioning discipline is non-negotiable. Workflow definitions must be treated as immutable. Any behavioral change to agent orchestration must be gated behind SDK versioning APIs (such as workflow.GetVersion()) or deployed onto isolated task queue deployments. Without strict version hygiene and proactive execution rollovers, an event-sourced harness will systematically break the very autonomous workflows it was built to protect.

Idempotency Gates and Context Compaction in Production

In distributed systems, the default response to transient failure is an automated retry. When Temporal encounters an activity timeout, worker process death, or network partition, its runtime replays the failed activity according to its configured backoff policy. For read-only actions—such as querying an account balance or reading a transaction status—retries are harmless. In transactional, ledger-facing tools, unconstrained retries are catastrophic.

If an agent invokes a tool to post a journal adjustment, disburse a vendor payout, or modify a chart of accounts, the network connection may drop after the downstream ERP commits the write but before the worker receives an acknowledgment. If that activity retries blindly, the agent creates duplicate ledger entries and balance distortions. I prevent duplicate side effects by making tool dispatching strictly idempotent. Every mutating activity must require a deterministic idempotency key computed from the execution tuple: hash(workflow_id, run_id, activity_name, step_index). Downstream service adaptors register this key before execution, enforcing check-then-write or two-phase commit semantics. If a retried activity arrives with an existing key, the service returns the cached response rather than re-executing the mutation.

The second operational hazard in deep agentic execution is context degradation. When an autonomous agent operates across dozens of turns, its context window rapidly swells with tool outputs, tabular diffs, and execution logs. Naive sliding-window truncations inevitably drop essential system instructions or critical transaction boundaries, causing "instruction drift" where the agent loses its operating parameters and hallucinates offset balances.

I address this by enforcing a three-tiered context retention model that separates immutable rules from transient computation artifacts.

Context Tier	Retention Rules	Eviction Triggers	Token Budget Impact
System Anchors	Immutable & Permanent: Core identity, user objective, schema definitions, accounting invariants, and safety boundaries are pinned to every prompt assembly.	Never evicted; remains static throughout the entire multi-turn workflow lifecycle.	Predictable, fixed baseline overhead (~1,500–3,500 tokens) across all turns.
Semantic Rollup Summaries	Structured Distillation: Periodic rollups that distill completed intermediate steps, confirmed balance checks, and verified decisions into concise state summaries.	Generated whenever active context utilization exceeds 65% of the model's operational window.	Replaces thousands of verbose multi-turn tool traces with compact structured summaries (~500–1,000 tokens).
Ephemeral Tool Outputs	Disposable Scratchpad: Raw tabular extracts, sandbox execution logs, terminal output, and intermediate validation payloads.	Evicted immediately after the calling turn completes, with full payloads persisted directly to object storage.	Elastic, zero-accumulation overhead; transient spikes during execution (~2,000–10,000 tokens) drop to zero on turn completion.

By pruning transient scratchpad dumps while strictly anchoring initial system rules and accounting constraints, the agent maintains precise instruction adherence across 100-turn executions without context window exhaustion.

Production Blueprints: Implementing Durable Agent Workflows

Operating a unified agent runtime across an organization is fundamentally an exercise in distributed systems discipline rather than prompt design. Abstracting agent logic into high-level agentic libraries often obscures worker lifecycle management, queue topology, and persistence mechanics. For engineering teams moving beyond fragile prototype loops, durability requires treating task queue topology, activity payload claim-checking, and resilient tool dispatching as foundational infrastructure.

This architectural pattern directly unlocks an Autonomous Multi-Entity Ledger Reconciliation Engine. In this setup, long-running accounting agents ingest mismatched banking feeds, execute algorithmic reconciliation scripts inside isolated Firecracker microVM sandboxes, and post verified balancing adjustments directly to enterprise ERPs. Orchestrating this pipeline with Temporal workflows provides the multi-day durability required for month-end close cycles, while PostgreSQL maintains transaction graph indexing and S3 stores multi-megabyte ledger snapshots. The principal failure mode to guard against is duplicate journal creation when automated activity retries fire against external financial endpoints; every ERP mutation activity must demand a deterministic idempotency key derived from the workflow ID, run ID, and transaction identifier before executing any write.

The identical execution rails can power an Interactive Operations Copilot with Ephemeral Code Verification. Finance and operations operators engage in low-latency synchronous chat to analyze live revenue numbers, generate complex data transformations, and visualize balance sheets on demand. To deliver responsive token streaming while preserving durability, the architecture utilizes dedicated low-latency task queues and pre-warmed container pools, with workers streaming tokens directly to the client over WebSockets while recording state transitions out-of-band. The primary operational trap is head-of-line blocking: if long-running batch reconciliation jobs share worker capacity with live dialogue turns, interactive requests stall behind multi-minute computations. Strict worker pool and queue segregation eliminates this contention entirely.

Finally, an asynchronous Durable Invoice Parsing and Human Approval Pipeline demonstrates how this design orchestrates long-lived human-in-the-loop governance. The workflow orchestrates multi-page document OCR, leverages vision models to extract line items against purchase orders, and safely enters an asynchronous sleep state for days while waiting for departmental budget owners to post approval webhooks. High-resolution source scans reside securely in S3, referenced through lightweight claim-check URIs during extraction activities. Here, the critical implementation trap is context degradation across multi-stage review loops. When automated compaction summarizes lengthy discussions between human reviewers and the system, the platform must guarantee that initial invoice line-item extraction anchors are preserved unmodified so the agent evaluates final ledger postings against verified source data rather than degraded conversational summaries.

Resources

• What is a Temporal Worker? | Temporal Documentation