The Boss Sign-Off Gate: What agency-agents-zh's Orchestrator Actually Gets Right and Wrong About Human Checkpoints

276 Markdown Files and a Separate Orchestrator: Two Products Wearing One Name

The first thing to get straight about agency-agents-zh is that it does nothing. It's 276 markdown files — 213 translated from the English msitarzewski/agency-agents upstream, plus 63 written for the China market specifically: Xiaohongshu and Douyin content ops, WeChat and Feishu/DingTalk workflows, gov-to-B and medical compliance, industrial HMI and mechanical design, even livestock recordkeeping. Each file is a persona document — identity, key rules, a workflow process, deliverables, success metrics — the same shape as any well-written system prompt. There's no inference call anywhere in the repo, no scheduler, no state. Open the repo and you're looking at a folder of instructions, full stop.

Getting those instructions in front of a model is a copy-and-paste problem with sixteen different paste targets. Claude Code and GitHub Copilot read plain markdown agent files natively, so those are a direct drop into ~/.claude/agents/ or ~/.github/agents/ — no tooling required. Everything else goes through scripts/convert.sh --tool <name>, which reformats the same source content into whatever shape a given assistant expects. Cursor gets .mdc rule files with alwaysApply: false, meaning the assistant is supposed to read each file's description field and decide at runtime whether a given rule is relevant to the current prompt. That's a retrieval mechanism built entirely out of string matching, and it breaks exactly as you'd expect: Trae's own documentation, converting the same library, warns that installing all 268 rules dilutes the description matching until almost none of them fire reliably, and recommends hand-picking 10–20 frequently used ones instead. A Discord-based integration ("Hermes Agent" mode) hits a harder wall — Discord's command JSON has an 8,000-character cap, so a full install simply doesn't fit, and agents have to be installed by --category instead.

None of that is where "demo to production" gets interesting, though. Everything this piece is actually about — auto-composing a workflow from a one-line request, running it as a DAG, pausing for a human sign-off mid-run — lives in a separate project entirely: a desktop app called Agency Orchestrator. The persona library is just raw material it consumes. Treating the two as one product, which the shared branding invites, is the wrong mental model — one is a content pack, the other is the runtime that turns that content into something resembling a process.

Where the Checkpoint Actually Lives: Compose, DAG, and the Boss Sign-Off Gate

ao compose "<one-line request>" --run is the on-ramp. You describe what you want in a sentence, the orchestrator selects specialists out of its pool of 267 roles, and it writes a workflow.yaml DAG connecting them — this is genuinely the "demo" end of the spectrum, minutes from prompt to a working pipeline. The DAG engine reads that file, figures out which steps have no dependency on each other and runs them concurrently — a tech_review and a design_review firing in parallel is the example given in the docs — and pipes each step's output into the next via {{variable}} templating. The visual canvas editor rejects circular dependencies at validation time, which matters more than it sounds like, because I'd treat a DAG assembled by an LLM as something that needs a validation pass regardless of how the docs describe it.

Where this stops being a toy is a real, shipped template: 一人公司-做投研.yaml ("one-person company, doing investment research"). It contains this node, verbatim:

- id: boss_signoff
  type: approval
  prompt: |
    风控官已提交风险清单：
    {{risk_report}}
    重大结论出具前须老板签字。确认继续生成最终研究报告吗？
  depends_on: [risk]

That's a literal mid-DAG pause, not a decorative field. The risk-analysis step's output gets substituted directly into the approval prompt via {{risk_report}}, and the step that produces the final deliverable is itself gated with depends_on: [boss_signoff] — nothing downstream executes until a human responds. Closed issues #23 and #25 back this up from the implementation side: human_input and a web-based real-time intervention path (a pause dialog that writes back to the running process's stdin) were both built as genuine mid-run interrupts, the kind that block execution and wait, not a status field a UI merely displays after the fact.

flowchart TD
    A["ao compose one-line request"] --> B["select roles from pool of 267"]
    B --> C["generate workflow.yaml"]
    C --> D["risk step"]
    D -->|"{{risk_report}}"| E["boss_signoff: type=approval, depends_on=[risk]"]
    D --> F["tech_review"]
    D --> G["design_review"]
    F -.parallel.-> G
    E -->|"paused, awaiting manual approval"| E
    E -->|"approved"| H["final_report: depends_on=[boss_signoff]"]

The parallel tech_review/design_review branch and the boss_signoff gate are independent parts of the same DAG — parallelism is an optimization for steps with no shared dependency, the approval gate is a deliberate serialization point that the DAG author inserted on purpose. That distinction is the whole design: default to running fast wherever nothing depends on human judgment, and force a stop wherever something does.

The Goldilocks Model-Tier Problem Nobody Mentions in the Demo

Every orchestration demo implicitly sells the same premise: chaining specialized agents beats asking one model to do the whole job. The maintainer actually tested that premise instead of asserting it — a blinded, position-debiased eval (eval/run-eval.ts, written up in EVAL_FINDINGS.md) comparing multi-agent DAG output against single-shot prompting on identical tasks, run across three model tiers. The results don't support a uniform claim, and the maintainer says so directly.

Tier	Model	Multi-agent record	Stated mechanism
Weak	llama3 8B (Ollama)	Lost 3 of 4 templates, 0/3 across repeats	Errors and drift compound at each handoff
Mid (tool's default)	DeepSeek	Won 3 of 4, one template 3/3 high-confidence	Handoffs add real correction/specialization value
Strong	Claude (both sides)	~1 win, 3 losses — roughly tied-to-behind	Single-shot already near ceiling; orchestration overhead doesn't pay for itself

The pattern is a Goldilocks curve, not a monotonic improvement. On the weak end, a DAG makes things worse, not just slower — each handoff is a fresh opportunity for the next agent to inherit and amplify an upstream mistake, and with a small model there's nothing correcting for it. On the strong end, the single model was already producing something close to the best available output, so paying for coordination overhead bought nothing. The value only shows up in the middle, on a model good enough to execute a narrow role well but not so capable that specialization is redundant — which happens to be exactly the tier the tool defaults to.

This is the part I'd want a team evaluating this tool to sit with before they get excited about the approval gates. It's tempting to read "human checkpoint" as a general safety net — the idea that even if an agent screws something up, a person catches it before it ships. That's not quite what's happening here. The checkpoint is downstream of a process whose net value is itself conditional on model choice. If you swap DeepSeek for whatever your org already has a contract with — a much weaker local model to cut cost, or a much stronger frontier model because it's "better" — you're not just changing latency or cost, you're potentially inverting whether the orchestration step was worth running at all, and no approval gate downstream tells you that's happened. A human signing off on a risk report generated by a degraded pipeline is still signing off on a degraded pipeline. The gate catches specific bad outputs it's shown; it doesn't audit whether the architecture generating those outputs is even net-positive for the model you actually plugged in. That's a re-test-on-every-model-swap discipline, not a one-time setup decision, and it's the kind of thing that's invisible in a demo run on whatever model happened to be configured when the demo was recorded.

Depends_on Confusion: Why Generated DAGs Need a Reviewer Before They Need an Approver

The most instructive bug in this whole project isn't in the checkpoint mechanism — it's upstream of it, in the step that writes the DAG in the first place. ao compose asks an LLM to turn a one-line request into a workflow.yaml, and one recurring failure mode is the model confusing a step's output variable name with its step id inside depends_on. The result is an error like "step X depends on nonexistent step Y," where Y is a variable name that appears nowhere else in the file. You can't grep your way out of that. There's nothing to search for, because the thing the error is complaining about was never a real identifier to begin with.

What makes this worth dwelling on is the fix history, not just the bug. Issue #94, filed against an early v0.2.9, got a maintainer response that this class was already handled by the validation chain. It wasn't. Issue #103, against desktop v0.4.1, shows the same failure recurring, and the maintainer's own follow-up walks it back: the earlier claim was inaccurate. The three-stage repair chain — a heuristic pass that fills in missing dependency edges, a pass that fixes variable references, and an LLM re-fix that reads the raw error text — had a specific hole: when the bad reference can't be parsed into a name, all three stages silently no-op and the user gets the raw, undebuggable error anyway. That's not a rare edge case failing to get caught; it's the exact shape of input the bug produces, sliding past three layers built to catch it.

The eventual fix, autoFixDependsOnIds, is appropriately conservative — it only rewrites when a match is unambiguous, and refuses outright on ties, cycles, or self-dependency rather than guessing. It's gated behind npm 0.14.0 and desktop 0.4.3, which means anyone on an earlier point release is still going to hit the original failure, fix or no fix.

I'd take two things from this. First, ao run --resume last --from <step-id> and --feedback "<note>" exist precisely because re-running a whole DAG to fix one bad step is wasteful — that's the right instinct, and --team locking a saved roster is what turns a one-off composition into something repeatable. Second, and more important: a successful ao compose tells you the YAML parsed, not that the graph is sound. Validation here is a chain of heuristics and LLM re-fixes over LLM-generated text, with a documented history of specific inputs slipping through all three stages at once. That's a strong argument for putting an approval gate right after DAG generation, before any step executes — not just at the end, where you're reviewing output built on a structure nobody actually checked.

Where the Checkpoint Leaks: Resume Gaps, Rendering Gaps, and Silent Rewrites

The boss_signoff gate is real, but the guarantees around it are thinner than the gate itself. Three documented edges are worth knowing before you build a process around it.

The first is a resume gap specific to binary output. --resume is built to restart a workflow from a given step without re-running everything before it, which is the right feature for a checkpoint pattern — approve, then resume, rather than approve and re-execute from scratch. But for image-generation steps specifically, the maintainer's own handoff notes flag a known v1 limitation: when --resume skips a step that produced an image, the new run directory doesn't contain the old image's bytes. The run's markdown still references a file path, but that path points at the previous run's directory, which may already be cleaned up. Anything piped through {{variable}} text substitution survives a resume cleanly. Anything that landed on disk as a file does not, automatically.

The second is a viewing gap at the moment it matters most. The live SSE stream — the view a human would presumably watch while a run is in progress — doesn't render images as they're generated mid-run. They only show up after the run finishes and the history view re-fetches with path rewriting applied. That means a reviewer sitting at an approval gate, watching the run live, may be asked to sign off on a step whose actual visual output isn't visible to them yet. The approval prompt can render correctly and still not show the thing being approved.

The third is a trust problem rather than a technical one. The maintainer's own audit notes list "saving silently rewrote the user's YAML without telling them" as one of six defects found in a single pass — since fixed by switching to an explicit fill-in-and-tell-you behavior, but notable as a pattern, not just an isolated bug. An approval gate is only as trustworthy as the assumption that the file you're looking at is the file that's actually going to run. A tool that has, at least once, mutated that file underneath the user without telling them is a tool where that assumption needs to be checked rather than assumed.

None of these breaks the checkpoint mechanism outright — the pause-and-wait behavior at boss_signoff is genuinely built, not cosmetic. But checkpoint integrity is a property of the whole system around the gate, not just the gate itself: whether artifacts survive a resume, whether the reviewer can see what they're reviewing, and whether the file on disk matches what's on screen. All three are described by the maintainer as still-settling, not solved.

stateDiagram-v2
    [*] --> Composed
    Composed --> Validated
    Validated --> Running
    Running --> Running : parallel steps (tech_review, design_review)
    Running --> Paused
    Paused: Paused at boss_signoff (approval)
    Paused --> Approved
    Paused --> Rejected
    Approved --> Resumed
    Resumed --> Completed
    Rejected --> Feedback
    Feedback --> ReRunStep : re-run from step
    ReRunStep --> Running
    note right of Resumed
        Steps holding image/file artifacts:
        new run directory can lose the prior
        step's binary output; text piped via
        {{variables}} survives, files may not
    end note

Agency Orchestrator Against the Checkpoint Field: LangGraph, Dify, Temporal, and the Rest

Set next to the tools that already own this problem, Agency Orchestrator's approval/human_input nodes are real but modest. LangGraph's interrupt() pauses execution before a node and persists state keyed to a thread_id, with replay-from-any-checkpoint debugging as a first-class feature — it's become close to default vocabulary for this pattern, but it's a Python/JS graph you write, not a YAML file a non-developer edits. CrewAI's human_input=True flag is the closest conceptual cousin to boss_signoff — a role-based crew pausing for review — but it's still a Python SDK dependency, not a standalone app. Microsoft Agent Framework's RequestPort and ctx.request_info() give typed suspend/resume with middleware and telemetry as first-class citizens, which is what production-hardened looks like when a vendor's support org is behind it — at the cost of enterprise/.NET-Python lock-in that a solo operator has no reason to take on. Temporal sits above all of them on durability: signals and wait_condition() over an event-sourced history mean a workflow survives a crash and resumes correctly, no exceptions, for workflows that persist for months — a meaningfully higher bar than Agency Orchestrator's timestamped ao-output/ directories, which are fine for a resumed CLI run and unproven for recovery mid-step. That durability costs a server and cluster to run. n8n, by contrast, has no approval primitive at all — human-in-the-loop is assembled by hand from a Wait node and a notification node, which works but means "pause for approval" isn't something the tool gives you, it's something you build. Dify is the real comparison point: its Human Input node ships versioned, with Approve/Reject/Escalate buttons and state that persists for weeks, which is more documented and further along than anything Agency Orchestrator has shown for its own gate.

Tool	Checkpoint/approval primitive	Durability model	Code required?	Distribution cost
Agency Orchestrator	approval/human_input YAML node types	Timestamped ao-output/ directories	No — YAML + desktop app	Zero-server; 7 of 11 backends reuse an existing CLI login
LangGraph	interrupt(), thread_id-keyed state, replay-from-checkpoint	Persisted graph state, developer-managed store	Yes — Python/JS	Library; you host and run it
CrewAI	human_input=True task flag	In-process, developer-managed	Yes — Python SDK	Library; you host and run it
Microsoft Agent Framework	RequestPort / ctx.request_info() typed suspend-resume	Session-based state, enterprise-managed	Yes — enterprise stack	Vendor platform, .NET/Python lock-in
Temporal	Signals / wait_condition()	Event-sourced history, crash-survivable, months-long	Yes — workflow code	Needs a server/cluster
n8n	None — assembled from Wait + notification nodes	Workflow-run state	No — visual builder	Self-hosted or cloud; 400+ integrations
Dify	Human Input node (Approve/Reject/Escalate)	Weeks-long persisted state	No — visual builder	Self-hosted or cloud, versioned release

Where that leaves Agency Orchestrator is a narrow but real position: ahead of n8n, which doesn't even offer a first-class gate; roughly parallel in ambition to Dify, but behind it in documented maturity — Dify's node is shipped and versioned, this one is a README claim backed by one working template. It isn't trying to compete with Temporal or Microsoft Agent Framework's durability class at all, and it shouldn't be judged as if it were. What actually distinguishes it is distribution: zero server, no API key required for 7 of its 11 backends because it reuses a CLI login you already have — Claude Code, Gemini CLI, Copilot, Codex CLI, OpenClaw, Hermes, Ollama. That's a genuinely different cost structure than anything else on this list, and it's the reason I'd consider this tool at all despite everything in the two sections above it.

The Failure Mode That Matters More Than the DAG: Silent Empty Output

Everything above assumes a workflow either runs or throws an error you can see. Issue #99 is the case that breaks that assumption, and it's the one I'd worry about most in an unattended deployment. Route a step through Azure OpenAI or another reasoning-model provider, and the connector's default token cap — 4096, inherited from a non-reasoning-model era — gets consumed entirely by the model's internal reasoning before a single visible token comes out. The API returns finish_reason: "length" with empty content. No exception, no retry trigger, nothing an error handler can catch. The step "succeeds" and produces nothing.

What makes this worth dwelling on is where it was hiding. The reporter traced the same root cause — a token cap sized for a pre-reasoning-model world — to four separate places in the codebase: the connector's default, the routing logic for non-Azure reasoning models, the web UI's "test connection" endpoint, and the hardcoded default baked into ao compose's generated YAML. Each needed its own patch, shipped across two releases (0.13.0, then 0.14.0), because fixing the concept in one place didn't fix the four places it had been copied into.

The maintainer's handoff notes describe the same shape of bug recurring around provider config generally. ANTHROPIC_BASE_URL was overloaded to mean two unrelated things — a relay endpoint for direct API calls, and the OAuth login state for the Claude Code CLI subscription — and six distinct bugs trace back to that one conflated variable, including one config silently rerouting an unrelated CLI subscription. One relay provider returns HTTP 200 with a 404 encoded in the response body, which defeats any liveness probe that trusts the status code and had to be special-cased. Another provider's /v1/models metadata claims it doesn't support image generation when the endpoint actually works fine — the tool can't even trust vendor-reported capabilities, so it falls back to probing the real endpoint directly.

None of this is exotic. It's the ordinary mess of eleven providers with inconsistent auth schemes and inconsistent reasoning-token accounting. But the failure mode it produces — success with nothing behind it — is categorically worse than a crash inside a cron-driven, --notify-backed deployment. A crash gets noticed. A scheduled run that completes cleanly and pushes "done" to a DingTalk group, with an empty or truncated report attached, gets ignored until someone actually reads the output — which, for a report nobody was already suspicious of, might be a while. Design any unattended run around this tool with that specifically in mind: the thing to alert on isn't failure, it's plausible-looking success with nothing inside it.

Running This Unattended: Version Skew, EACCES, and Orphaned Jobs

A separate class of problem shows up once you move this off a laptop and onto something that's supposed to keep running. The v0.18.0 report is the clean example: ao web's key-saving path calls mkdir inside the global npm package directory — /opt/homebrew/lib/node_modules/agency-orchestrator/.local in the reported case — and hits EACCES. That works fine on a demo machine where npm i -g landed under a user-owned prefix. It breaks the moment Node is provisioned the way any IT department actually provisions it: Homebrew or a system package manager, root-owned global directories. A tool that writes runtime state into its own install path is making an assumption about the install path that doesn't hold outside a personal laptop.

Then there's a propagation asymmetry that matters specifically for cron-and-forget setups. Provider and sponsor config changes reach every installed user within six hours, pushed via a remotely fetched manifest. Actual code fixes — the token-cap patches, the depends_on fix — only reach users who explicitly run npm i -g agency-orchestrator@latest or pull a new desktop build. Deploy this via Docker on a NAS with a cron schedule, as the docs recommend, and you get config drift for free while every bug fix sits there until someone owns the upgrade cadence deliberately. As of the fetched handoff doc, that skew was already visible in the numbers: the GitHub agent library was at 276 agents on 1.4.0, while the published npm package most people actually install was still on 1.2.7 with 267, pending a publishing config change. What the README claims and what npm install delivers on a given day aren't guaranteed to match.

Two more things I'd want on record before trusting this in production. Video-generation steps carry an explicit cost risk the maintainer flagged directly: the local process can die while the vendor-side job keeps running and billing continues, which is why polling failures are built to never abort the task, resolution and duration presets are never guessed across vendors, and endpoint support is probed rather than assumed. And CI was broken for over a month — npm test chaining files with && meant one failing file silently no-op'd everything after it — without anyone noticing. That's less a bug than a maturity marker: the project's own quality gate wasn't watching itself during exactly the period these features shipped.

Where I'd Actually Put This Pattern to Work

None of the above argues against using this — it argues for using it deliberately, with the specific failure modes above already priced in. Three shapes come to mind, all buildable this week with pieces already covered.

The most obvious fit is a content-ops pipeline for exactly the vertical work the persona library is stacked with — Xiaohongshu, Douyin, WeChat copy that currently gets written, reviewed, and published as three separate manual steps. Chain the relevant persona files through ao compose into a DAG that ends in an approval node before publish, with --notify pushing the draft and a short risk summary to a review group ahead of that gate — the same shape as boss_signoff, just aimed at a content calendar instead of an investment memo. Two things I wouldn't skip: run it on a DeepSeek-class provider, not whatever local model happens to be cheapest, because the eval data says multi-agent composition loses to a single well-prompted call below a certain tier, and it's not a minor loss. And read the generated workflow.yaml before trusting it — given how long the depends_on confusion bug survived three layers of auto-repair, "compose succeeded" is not the same claim as "the graph is correct."

Second is a compliance or risk digest built directly on the 一人公司-做投研.yaml shape rather than adapted from it: a risk-analysis step feeding a boss_signoff-style approval, gating a final report step. The payoff isn't the gate itself, which is straightforward — it's --resume --from <step> --feedback "<note>", which lets a reviewer push a targeted revision into one step instead of re-running the whole pipeline every time something needs a correction. That's the difference between a checkpoint that's actually cheap to iterate against and one that technically exists but nobody uses because re-running costs too much. The one thing to verify before depending on it: if any upstream step produces an image or other file artifact, confirm it survives a --resume first — that's a documented gap, not a hypothetical, and it's the kind of thing that fails quietly in exactly the way you don't want a compliance record to fail.

Third, and this one's aimed straight at the failure mode from the section above: any scheduled, --notify-driven report needs a validation step inserted between generation and notification that rejects trivially short or empty output before it gets pushed anywhere. Cron plus Docker, per the documented deployment path, plus one small check that a report actually has content — cheap insurance against the exact silent-empty-output pattern reasoning-model providers produce. Pin the orchestrator version explicitly when you set this up. Provider config drifts toward you automatically; the fix for the bug you're protecting against does not.

Resources

• Agency Orchestrator — 一句话，让多个 AI 角色自动协作
• jnMetaCode/agency-agents-zh on GitHub
• jnMetaCode/agency-orchestrator on GitHub