Herdr Keeps Coding Agents Running Across Lids, Reboots, and SSH Hops: How the Client-Server Split Actually Works

The Problem Herdr Solves: Agent Sessions Die Where Terminals Die

I came to the Herdr repo (herdrdev/herdr, Rust, Apache-2.0, version 0.8.2 at the docs I read) through a practitioner's setup walkthrough: a MacBook Pro doing the actual agent work, driven remotely through Tailscale and Termius, with an Oracle VPS, a Raspberry Pi 5, a Dell XPS, and an Omarchy box rotating through the fleet. The specific hardware list is trivia. The pattern underneath is not: once you run more than two coding agents, your bottleneck stops being model quality and becomes session custody. Which agent is waiting on me? Which one is quietly stuck? What happens to a Claude Code session when my laptop lid closes?

The standard answers are all bad in their own way. Run the agent in your terminal and it dies with the terminal. Run it in tmux and it survives, but nothing tells you it has been sitting on an approval prompt for forty minutes — tmux shows you a green pane and a blinking cursor, and distinguishing "working" from "blocked waiting for permission" requires a human to read the scrollback. Wrap agents in a Python orchestrator and you get lifecycle management but lose the interactive terminal, which is where these agents actually live — they are TUI applications, not daemons.

Herdr's design bet is that you should not have to choose. It is a background server that owns real terminal processes, plus attached clients that render them. Close the lid, drop the network, restart the machine — the server keeps the panes alive and restores session shape on restart. And because the server can see every pane's process tree and screen content, it can classify each agent's state: blocked, working, done, idle, or unknown. That last mile — knowing an agent needs you — is the feature that tmux never had and that agent orchestration frameworks keep rebuilding badly at a higher layer.

The install is deliberately boring: curl -fsSL https://herdr.dev/install.sh | sh, or brew install herdr, or mise, or a PowerShell one-liner on Windows. One Rust binary, no Electron. It runs inside whatever terminal emulator you already use, which means adoption cost is roughly "learn one prefix key."

Workspaces, Tabs, Panes: The Object Model

The docs lay out a four-level object model, and it maps cleanly onto how multi-agent work actually organizes itself:

• Workspace — the top-level project container, one per repo or investigation. Its sidebar state rolls up from the agents inside it, so the project list itself becomes an attention dashboard.
• Tab — a layout within a workspace; the docs suggest separating views like agents, logs, server, review.
• Pane — a real terminal. Herdr renders its output, forwards input, and preserves it across client detach. Panes split right or down.
• Session — a persistent server namespace. herdr session attach work and herdr session attach side-project give you fully separate pane trees, sockets, and runtime state under one config file.

The agent states are the part worth memorizing, because everything else in the product — the sidebar rollup, the notification sounds, the socket API's "wait until blocked" primitive — is downstream of this classification:

State	Meaning	What I do with it
blocked	Agent needs input, approval, or a decision	Drop everything; this is the only state that gates wall-clock progress
working	Actively running	Leave it alone; batch-check others
done	Finished, not yet reviewed	Queue for a review pass
idle	Finished or waiting, already seen	Ignore until something changes
unknown	Herdr cannot confidently classify	Treat as blocked until proven otherwise

Detection comes from three sources: foreground process inspection, screen manifests, and optional integrations. That last phrase matters — the manifest approach means Herdr recognizes agents by what they draw on screen, which is how it runs what you already run. The README names Claude Code, Codex, Cursor, OpenCode, and Grok explicitly, with "and the rest" carrying the tail. Herdr does not wrap or re-implement these agents; it owns their terminals. That is the correct architectural humility: agent vendors change their TUIs every month, and a tool that tried to own the agent loop would be in a permanent rewrite war. A tool that owns the terminal and pattern-matches on state is insulated from all of it.

One design choice that surprised me: Herdr is mouse-native. Click panes, drag split borders, right-click menus — all first-class. Keyboard people get tmux-style prefix bindings (ctrl+b default) layered on top, and if you hate mouse capture entirely, [ui] mouse_capture = false in config turns it off. The three input modes — terminal mode (keys go to the focused pane), prefix mode (one Herdr action after the prefix), and navigate mode (persistent workspace navigation) — are exactly tmux's mental model, so migration friction is near zero for anyone who has used a multiplexer before.

The Server Owns the Panes; Your Terminal Is Disposable

Architecturally, Herdr splits into a background server and one or more attached clients. The server owns panes and process state; the client is just a terminal UI. Detach with ctrl+b q and the agents keep running; reattach with herdr. herdr server stop is the deliberate kill switch. After a full server stop, a restart restores the saved session shape.

This is the same fundamental design as tmux, but with two additions that change what the persistence is for. First, state classification persists with the panes, so reattachment is not just "your scrollback is back" but "here is the one pane that needs you." Second, session shape restoration means the workspace layout — which agents, in which tabs, in which splits — is itself durable state. When your working context is "seven agents across three workspaces," the layout is the working set, and restoring it is not a convenience, it is the difference between resuming in thirty seconds and spending twenty minutes rebuilding splits and re-attaching to the right repos.

flowchart TD
    subgraph SERVER["herdr background server (owns panes and process state)"]
        W1[Workspace: repo A] --> T1[Tab: agents]
        W1 --> T2[Tab: logs]
        T1 --> P1[Pane: Claude Code<br/>state: working]
        T1 --> P2[Pane: Codex<br/>state: blocked]
        T2 --> P3[Pane: dev server<br/>plain process]
        W2[Workspace: side project] --> P4[Pane: OpenCode<br/>state: done]
        STATE[Agent state classifier<br/>foreground process + screen manifests]
        P1 & P2 & P3 & P4 --> STATE
    end
    subgraph CLIENTS["attached clients (disposable)"]
        C1[Laptop terminal UI]
        C2[SSH thin client<br/>herdr --remote]
        C3[CLI / socket API<br/>scripts and other agents]
    end
    C1 -->|render + input| SERVER
    C2 -->|SSH stream| SERVER
    C3 -->|cli + socket| SERVER

There is an escape hatch, herdr --no-session, which skips the server/client split entirely for debugging or compatibility. The docs are explicit that persistent session mode is the default and the point; the flag exists so the fallback never requires uninstalling your habits.

Remote Attach: Two Modes, and Why the Thin Client Wins

Remote access has two documented paths, and the difference between them is more consequential than it looks.

The tmux-style path: SSH into the server, run herdr there. Everything runs on the remote host, your SSH session is just the render pipe. Fully supported, zero surprises, and subject to the classic failure mode — a flaky network makes your UI janky even though the agents never noticed, and you get none of your local terminal's desktop integrations.

The thin-client path: herdr --remote workbox from your local machine. Your local Herdr binary connects over SSH, starts or attaches the remote server, and streams the UI back — but rendering and input handling happen locally. Three consequences, all documented:

1. Local desktop features bridge into the remote session. Image clipboard paste works across the boundary: the local client copies the image to a remote temp file and pastes that path. Anyone who has tried to paste a screenshot into a remote agent session knows exactly how much plumbing that saves.
2. Keybindings stay local. Your local muscle memory applies even when the remote server has different config. The local bindings are snapshotted at attach time — detach and reattach after editing them. Custom command keybindings are deliberately not sent, because those commands would execute on the remote host. That is a security boundary drawn in the right place.
3. Binary lifecycle is managed. Herdr checks the remote platform, prefers a matching herdr already on the remote PATH, then probes Homebrew, mise, and Nix profile paths. If nothing matches, an interactive run prompts to install to ~/.local/bin/herdr; a non-interactive run fails instead of modifying the host. Local builds can ship their own binary via HERDR_REMOTE_BINARY.

Platform support is specific: Linux, macOS, and Windows local clients, attaching to Linux or macOS hosts on x86_64 and aarch64. Windows is not supported as the remote host — relevant if your fleet looks like the one that led me here, which mixed Apple silicon Macs, an x86 Dell, a Raspberry Pi (aarch64 Linux, fine), and an Oracle VPS. Name your targets once in SSH config and herdr --remote workbox picks up host, user, and port from there. herdr --remote workbox --session agents attaches to a named remote session.

Under the hood, remote attach generates a temporary SSH config that includes your own config first, then appends fallback keepalive settings — your explicit keepalive config wins — plus a per-attach control socket for connection reuse on Linux and macOS (Windows OpenSSH gets neither). Authentication is your normal OpenSSH auth; the docs correctly point out that passphrase-protected keys in non-interactive contexts need ssh-agent loaded first, and that debugging any remote auth failure starts with plain ssh workbox before blaming Herdr. There is also an experimental --handoff flag for live handoff to a supported running remote server instead of the default restart flow — I would treat that as opt-in-only until it graduates.

sequenceDiagram
    participant L as Local client (laptop / phone-class device)
    participant S as OpenSSH (config host "workbox")
    participant R as Remote herdr server
    participant A as Agent pane (Claude Code)
    L->>S: herdr --remote workbox
    S->>R: connect via control socket, keepalives active
    R->>R: check remote platform + existing herdr on PATH
    alt no matching binary
        R-->>L: prompt to install to ~/.local/bin (interactive only)
    end
    L->>R: attach session (local keybinding snapshot)
    R->>A: agent keeps running regardless of client state
    Note over L,A: laptop lid closes — client dies, server and agent unaffected
    L->>R: reattach later, state classifier reports blocked pane
    L->>A: answer approval, agent resumes

That last sequence is the whole value proposition compressed: the agent's liveness is decoupled from every client that has ever looked at it, and reattachment is answered with state, not scrollback.

Direct Attach and the Observer/Controller Protocol

Beyond the full workspace UI, Herdr exposes direct attach — one server-owned terminal streamed into your current terminal. herdr agent attach reviewer attaches by agent name; herdr terminal attach term_abc123 by terminal ID; --takeover replaces an existing input owner. Ownership rules are explicit: only one writable direct-attach client owns input and resize for a terminal at a time. That single-writer rule is exactly what you want when a human and a script might both reach for the same pane.

The more interesting surface is the session observer/controller pair, which is Herdr's answer to programmatic access without pretending terminals are APIs:

herdr terminal session observe w1:p1 --cols 120 --rows 40
herdr terminal session control w1:p1 --takeover --cols 120 --rows 40

The observer prints newline-delimited JSON terminal.frame records containing base64-encoded ANSI bytes, then a terminal.closed record. Multiple observers can watch one terminal without taking input, resize, scroll, or ownership. The controller is the writable counterpart: it emits the same frames and reads NDJSON commands on stdin — terminal.input for text or base64 bytes, terminal.resize, terminal.scroll, terminal.release — again with exactly one controller owning input and resize.

This is a thoughtfully narrow protocol. It exposes what a terminal shows and what a keyboard can type — nothing else. That makes it trivially bridgeable (the docs mention third-party bridges as the intended consumer) while refusing to become a general remote-code-execution API. If you want agents driving agents, that lives one layer up:

Agents Driving Herdr: The Multi-Agent Pattern

Herdr ships a CLI and a socket API, and the README's agent-native pitch is specific: agents can spawn panes, prompt each other, and wait until another agent is genuinely blocked. That last primitive is the one I find most valuable, because it encodes the correct coordination semantics. Naive multi-agent setups poll on timers or sleep for fixed durations, which either wastes cycles or races ahead of unfinished work. "Block until pane X enters blocked state" is event-driven handoff built on the same state classifier the human UI uses — one source of truth for "is it my turn," shared by humans and machines.

There is a documented agent skill for this (herdr.dev/docs/agent-skill/), and the community has built an MCP server on top for orchestrating multi-agent workflows inside Herdr. The repo itself practices what it preaches — it carries its own .agents/skills directory, including a pre-release audit skill and triage instructions, and an AGENTS.md that instructs AI contributors to read before touching anything. Small thing, but a codebase being navigable by the agents it hosts is a clear tell of where the maintainers' heads are at. (The repo also ships request and done sounds as assets — the audible version of the state machine, which sounds gimmicky until you have six panes and one pair of eyes.)

The fleet pattern from the setup that pointed me here — one always-on agent host, thin clients everywhere, Tailscale as the network fabric, Termius as the mobile SSH client — composes with these pieces without any of them being Herdr features. Herdr contributes the persistent server and the remote thin client; the network and the mobile terminal are commodity. The reason the combination works is that Herdr's remote attach is stateless from the client's perspective: a phone-class device that connects for ninety seconds, answers the one blocked pane, and vanishes loses nothing, because nothing lived on the client.

How It Compares When You Actually Choose

Dimension	Herdr 0.8.x	tmux	Zellij	Web orchestrators (agent dashboards)
Process persistence across disconnect	Server owns panes; restore on restart	Server owns panes	Server owns panes	Usually containers, not terminals
Agent state awareness	Native: blocked/working/done/idle per pane	None	None	Varies; usually requires agent SDK integration
Runs unmodified agent TUIs	Yes — owns their terminals	Yes	Yes	Often re-implements or embeds
Programmatic access	CLI + socket API + NDJSON observer/controller	send-keys, barely structured	Layout files, actions	Rich APIs, but your agents leave the terminal
Remote thin client	First-class, with binary management and clipboard bridging	SSH + run tmux inside	SSH + run Zellij inside	Browser; needs exposed endpoint
Input model	Mouse-native + tmux-style prefix	Keyboard only	Keyboard + mouse	Mouse only
Footprint	Single Rust binary, no Electron	Single C binary	Single Rust binary	Server + frontend + auth

The honest comparison line: tmux plus a discipline of naming sessions can get you 70 percent of the persistence. What it cannot get you is the state classifier and the wait-until-blocked primitive, and in my experience that 30 percent is where all the attention cost lives. Conversely, if you are already running agents through a platform SDK with webhooks and dashboards, Herdr is not competing with that — it is the layer underneath for the interactive sessions the SDK does not own.

Where I Would Be Careful

• The state classifier is pattern-matching, not mind-reading. unknown exists as a state for a reason. Before wiring automation to "wait until blocked," test it against your specific agent version's idle and approval screens; a false negative (agent blocked, classified working) stalls the pipeline exactly as hard as no orchestrator at all.
• Remote attach manages remote binaries. The interactive install prompt is convenient and also a mutation of a host. For fleet machines you care about, pre-install a pinned herdr on the remote PATH and let the version-match path win; for anything non-interactive, know that it fails closed rather than modifying the host — the correct default, but it will surprise you once in CI.
• Windows is a client, not a host. Remote hosts are Linux or macOS only, and Windows clients miss the per-attach control socket. Mixed fleets work; Windows servers do not.
• Single-writer is load-bearing. Direct attach and controller mode allow exactly one input owner. If a long-running script holds a controller and you attach interactively, you need --takeover — fine, but make sure the script's exit path releases, or your next attach becomes a takeover war.
• Keybinding snapshots are attach-time. Edit local bindings while a remote attach is live and nothing changes until you detach and reattach. Harmless, but it will look like a bug the first time.

None of these are disqualifying. They are the rough edges of a tool that picked a sharp architectural bet — the pane is the unit of scheduling, the terminal is the universal agent API — and is executing it with unusual discipline for something at 0.8.x. Thirty-three thousand GitHub stars say a lot of people hit the same session-custody wall I did and went looking for the same answer.

Resources

• herdrdev/herdr on GitHub (README, Apache-2.0)
• Herdr docs: Concepts
• Herdr docs: Persistence and remote access
• Herdr docs home (version 0.8.2)