Tax Agent Bench: What GLM-5.3's Third-Place Finish Actually Tells Us About Agentic Statutory Research

What Tax Agent Bench Measures That Standard QA Benchmarks Can't

Most LLM evaluations in the legal domain are static QA sets. You feed a model a question, it produces an answer, a rubric decides if it's right. The problem is that this bears almost no resemblance to how tax research actually happens. No professional sits down with a fully-formed query and produces a single response. The workflow is iterative by nature: you start with a client fact pattern, you form an initial hypothesis about which code sections might apply, you search, you find something that contradicts your hypothesis, you reformulate, you dig into the legislative history, you check whether a revenue ruling has been superseded, and only then do you draft.

Tax Agent Bench captures this. It drops models into an agentic loop with tool-calling capabilities, requiring them to execute multi-step lookups, run full-text statutory searches, and retrieve over tax code corpora and administrative guidance before assembling an answer. This isn't testing memory or pattern matching. It's testing whether a model can drive a research workflow from ambiguity to a defensible conclusion.

The test set itself is 193 private, curated corporate tax research problems. Private matters here—leakage is the eternal enemy of legal benchmarks, and the questions are designed to require genuine investigation rather than retrieval from training data. The five domains cover the real spectrum of professional work: multi-step statutory interpretations, IRS guidance synthesis, controversy and precedent analysis, formal tax memo drafting, and tax forms and filings compliance.

Scoring is where it gets brutal. Written outputs are evaluated against expert tax practitioner rubrics, and ground-truth verification places strict penalties on hallucinated authorities. A model can nail the legal reasoning and still fail if it cites a Treasury Regulation subsection that doesn't exist. Citations must be precise, verifiable, and traceable to actual statutory sections, regulatory subsections, and applicable precedent. This is the single most consequential design decision in the benchmark—it forces models to prove their work, not just sound plausible.

The difficulty ceiling speaks volumes. As of the September 2026 leaderboard, no evaluated model has exceeded 80% overall accuracy. Claude Fable 5.1 tops the chart at 77.64%, Claude Opus 5 sits at 75.06%, and GLM-5.3 holds third at 73.09%. That eight-point gap between the leader and the 80% mark isn't noise. It's the unresolved problem at the heart of professional legal AI: compound retrieval errors, subtle regulatory misinterpretations, and the sheer density of the tax code's internal cross-references are still beyond frontier models operating without human oversight. The benchmark reveals that professional-grade legal research for LLMs remains an open problem, not a solved one.

The Agentic Loop Mechanics: How GLM-5.3 Actually Navigates the Tax Code

The interesting thing about Tax Agent Bench is that it doesn't just measure whether a model knows tax law. It measures whether a model can operate within the tax research environment—issuing tool calls, parsing results, deciding what to retrieve next, and assembling a coherent memo from fragments of retrieved authority. This is a different skill set entirely from static QA performance, and it's why the leaderboard looks the way it does.

The loop unfolds like this: a user query arrives at the dispatcher, which in GLM-5.3's case is the generalist model itself. From there, a tool router decides what kind of lookup is needed. The branches are what you'd expect from an actual tax research workflow: statutory search against the Internal Revenue Code and Treasury Regulations, precedent and guidance search across Revenue Rulings, Revenue Procedures, and Private Letter Rulings, and full-text judicial search to find circuit court decisions and Tax Court opinions. Results get pulled from these corpora, assembled into context, and fed back to the model. The model then drafts its memo. That memo goes to an expert rubric that scores it for accuracy, completeness, and citation fidelity—with a penalty for hallucinated authorities.

flowchart TD
    Query[User Tax Query / Fact Pattern] --> Dispatcher[GLM-5.3 Dispatcher]
    Dispatcher --> Router{Tool Router}
    
    Router -->|Statutory Search| IRCDB[IRC & Treas. Reg. DB]
    Router -->|Precedent & Guidance Search| GuidDB[Rev. Rulings / PLR DB]
    Router -->|Full-Text Judicial Search| CaseDB[Judicial Case Law DB]
    
    IRCDB --> Context[Context Assembly & Retrieval]
    GuidDB --> Context
    CaseDB --> Context
    
    Runtime["All models remain below 80% overall accuracy"]
    Runtime -.-> Context
    
    Context --> Draft[Draft Memo / Finding]
    Draft --> Rubric[Expert Rubric Scoring]
    
    Rubric -->|Verification & Re-retrieval| Dispatcher
    Rubric -->|Penalty for Hallucinated Citations| Penalty[Hallucination Check]

GLM-5.3's first-place finish in Rule & Source Lookup at 81.29% tells us something specific about where this model's strengths lie. It's not just that it can find things. It's that it excels at issuing precise follow-up queries to locate obscure IRS guidance. Think about what that requires: a model must read the first batch of results, recognize that the primary authority it found doesn't fully answer the question, and formulate a new, more targeted search to dig into Private Letter Rulings or Revenue Procedures that aren't obviously connected to the surface-level query. That's a reasoning loop, not a retrieval step.

This is where the contrast with domain-fine-tuned vertical models becomes sharp. Thomson Reuters, LexisNexis, and Bloomberg Law rely on curated editorial tagging—KeyCite, Shepard's, CCH topic indices—where retrieval queries map onto human-annotated tax taxonomies. It's excellent for citation fidelity out of the box, but it's brittle when facts fall outside pre-structured categories. The generalist approach treats the tax code as raw corpora and explores it dynamically. The trade-off is real: generalists adapt to novel statutory interpretations that vertical models haven't been manually indexed for, but they also face drift in nested cross-references. GLM-5.3 might succeed at finding IRC §382 in the primary lookup but then lose track of the constructive ownership rules defined under §318 across consecutive retrieval steps. That's precisely the kind of multicascade failure that keeps overall accuracy below the 80% threshold.

Where GLM-5.3 Wins: Retrieval and Controversy Analysis

GLM-5.3 doesn't top the overall leaderboard—Claude Fable 5.1 and Claude Opus 5 both outscore it. But looking at the category breakdown reveals a fascinating split: the generalist from Zhipu AI leads the benchmark in exactly the subdomains that reward flexible exploration and autonomous source discovery.

Rule & Source Lookup at 81.29% is first among all 18 models evaluated. What does that category test? It tests whether a model can locate obscure IRS administrative guidance—Private Letter Rulings, Revenue Procedures, and statutory subsections that aren't top-of-mind for even experienced practitioners. PLRs, by their nature, are narrow. They're issued to specific taxpayers about specific factual situations, and they're not binding precedent. But they're highly informative for understanding how the IRS applies the law to particular facts. Finding one requires knowing it exists, knowing roughly where it might be indexed, and being willing to chase it down through multiple searches.

The second category where GLM-5.3 takes first is Controversy & Precedence Analysis at 71.08%. This is about synthesizing conflicting authorities—identifying where an IRS administrative position appears to conflict with judicial precedent, or where circuit courts are split on a particular question. It requires a model to hold multiple authorities in tension simultaneously, reason about their relative weight, and produce an analysis that acknowledges the conflict rather than papering over it.

Why would a generalist outperform specialized systems here? I'd argue it's because these categories reward flexibility in query formulation and hierarchical reasoning about authority. At 81.29%, the model is issued a statutory problem, and it has to reason about whether the revenue ruling it found might conflict with a Tax Court opinion it hasn't yet retrieved. That kind of hypothesis-driven exploration is easier for a model with broad general knowledge and packed tool-calling instincts than for a fine-tuned system operating on a rigid rule graph.

Category	GLM-5.3 (3rd overall)	Claude Fable 5.1 (1st)	Claude Opus 5 (2nd)
Overall Accuracy	73.09%	77.64%	75.06%
Rule & Source Lookup	81.29% (1st)	not disclosed	not disclosed
Controversy & Precedence Analysis	71.08% (1st)	not disclosed	not disclosed
Cost per Task	~$1.80	not disclosed	not disclosed

The cost dimension deserves attention. At approximately $1.80 per test task, GLM-5.3 delivers agentic tax research at a fraction of what enterprise legal research subscriptions cost. Proprietary per-seat pricing from legaltech vendors—even before factoring in the human analyst hours—routinely surpasses that figure for a single research session. This is a significant structural shift. It means that a generalist model, deployed as a tool in the research loop, can undercut specialized enterprise products on unit economics while competing on accuracy in the subdomains where retrieval flexibility matters most.

That said, the gaps matter too. GLM-5.3's 73.09% overall accuracy falls behind Claude Fable 5.1's 77.64%, and that four-point difference appears to come from drafting and formal compliance structuring. Finding the right authorities isn't the hard part for GLM-5.3—assembling them into a defensible tax memo with precise, verifiable citations is where it lags. The industry-wide ceiling below 80% reinforces that no generalist has fully solved the compound failure problem of multi-step statutory research.

The Failure Modes That Keep GLM-5.3 Below 80% Overall

The gap between GLM-5.3's 81.29% on Rule & Source Lookup and its 73.09% overall accuracy isn't a single weak spot—it's a cluster of specific, structural failure modes that compound across the agentic loop. The closer I look at where this model breaks, the more it looks like the problems are architectural, not intellectual.

Statutory sub-clause drift is the first one that jumps out. The Internal Revenue Code is a machine built on cross-references. IRC §382 limitation calculations don't stand alone—they point to ownership shift definitions buried in §318, which in turn invoke constructive ownership rules scattered across dozens of other sections. When GLM-5.3 runs a multi-hop tool-calling loop, it often succeeds at the primary lookup—finding §382, pulling its text, citing it correctly—but then loses the thread when the next retrieval step requires tracking a nested definition from a different title. Instead of continuing the chain, it falls back on semantic generalizations. The model generates a plausible paraphrase of what "ownership shift" might mean rather than retrieving the actual §318 constructive ownership rules that the statute explicitly incorporates. This isn't a knowledge gap; it's a state-tracking gap.

Authority hierarchy and temporal invalidation is the second, and arguably more dangerous, failure mode. Tax law enforces a strict precedence structure: the IRC statute outranks Treasury Regulations, which outrank Revenue Rulings, which outrank Private Letter Rulings and memorandum decisions. A PLR is, by definition, non-precedential—it applies only to the specific taxpayer who requested it. Yet generalist LLMs don't carry an inherent graph of these relationships. Without hard-coded Shepardizing tool hooks—checks that verify whether an authority has been superseded, overruled, or otherwise invalidated—the model frequently cites superseded regulations, proposed regulations that never finalized, or PLRs as binding authority against conflicting revenue rulings. In the benchmark's strict rubric, this is a fatal error, not a partial one.

Effective date blindness compounds on top of that. Corporate tax questions routinely pivot on temporal specifics—TCJA sunset provisions, CARES Act NOL carryback exceptions, grandfathering clauses that depend on the tax year in question. GLM-5.3 tends to treat the retrieved corpus as an unversioned snapshot. It finds the regulation, pulled the right text, but never verifies that the text actually applies to the tax year in the fact pattern. The model reasons as if the statute is a timeless object, when in reality it's a moving target with transitional rules at every amendment.

Numeric overconfidence is the most insidious of the four. Memo drafting often requires reconciling computations—adjusted basis, earnings and profits, boot distribution allocations under §351 and §356. GLM-5.3 handles the legal citations correctly but then drifts in the arithmetic. Token-level math inside dense legal prose is error-prone; the model produces memos that are legally sound and mathematically inconsistent.

These aren't independent—they compound. A 4-5 step agentic loop with a 90% per-step success rate still yields roughly 59% end-to-end accuracy. And that's why GLM-5.3's retrieval strength alone isn't sufficient for final outputs. It leads in finding authorities, but drafting and formal compliance structuring require sustained integrity across all four failure modes simultaneously. The lags behind Claude Fable 5.1 in memo quality aren't about retrieval at all—they're about what happens after the authorities are in context.

Why No Model Has Breached 80%: The Compounding Error Problem

The 80% ceiling on Tax Agent Bench isn't the result of any single model's weakness. Some simple arithmetic on agentic loop mechanics reveals why it's structural.

Consider what a single research task requires: a model must execute 4-5 sequential steps—formulating an initial hypothesis, issuing a statutory search, interpreting the retrieved text, identifying a conflicting authority, checking precedent, and assembling them into a memo. If each step succeeds at 90%—which is genuinely optimistic for tool-calling loops on dense statutory corpora—the math works out to roughly 59% end-to-end accuracy. That's worse than what we're actually seeing from top models.

The benchmark's strict rubric makes this harsher. The ground-truth verification penalizes any hallucinated citation. One fabricated Treasury Regulation subsection—even if the overall legal analysis is sound—can zero out an entire memo. In production terms, this is a binary all-or-nothing structure: you can be 95% right, but that remaining 5% is a citation that doesn't exist, and the entire output fails professional review.

Specialized legaltech systems avoid this problem through a fundamentally different architecture. Thomson Reuters, LexisNexis, and Bloomberg Law don't rely on an LLM navigating raw corpora. They use curated editorial tagging—KeyCite, Shepard's, CCH topic indices—where retrieval queries map onto human-annotated tax taxonomies. The citation graph is pre-built and deterministic. A model can't cite a superseded regulation because the system won't retrieve one. The trade-off is rigidity: these systems falter when a fact pattern falls outside pre-structured categories or requires novel statutory interpretation. But they also don't face the compounding error problem, because the verification isn't left to the model.

The generalist agentic approach is fundamentally different. It treats the tax code as raw text and hands the model the responsibility for finding, verifying, and weighing authorities. That flexibility is what lets GLM-5.3 excel at source discovery and controversial analysis. But it also means every single step in the loop is a potential point of failure, and the rubric design ensures those failures are fatal.

The 80% barrier, then, isn't a measure of intelligence. It's a measure of how brittle agentic loops are when the ground truth verification is strict. The engineering challenge isn't building a smarter model—it's building a production architecture that absorbs the failure modes at each step. That's why I'd argue the barrier isn't about model capability at all; it's about the absence of deterministic guardrails on top of the agentic loop.

Production Guardrails: How to Actually Deploy a Generalist Tax Agent

The core lesson from Tax Agent Bench is that GLM-5.3's retrieval strength is real, but it cannot be deployed raw. The structural failure modes—drift, temporal blindness, numeric overconfidence—aren't model bugs; they're architectural gaps. Closing them requires a production layer that doesn't trust the model to be right, but instead verifies its work at every step.

The first thing I'd build is a version-aware retrieval layer. Plain embedding search over tax law is a non-starter. What you need is a structured metadata index that tracks tax year applicability, document type hierarchy, and supersession links for every authority in the corpus. When the model issues a statutory search, the retrieval system filters by effective date and authority type before returning results. A superseded 1986 Revenue Ruling never enters the context window, so the model never gets the chance to cite it as binding. This is the closest you can get to deterministic Shepardizing in an agentic loop, and it's the single highest-leverage improvement you can make.

sequenceDiagram
    participant User as User
    participant GLM as Agent Orchestrator (GLM-5.3)
    participant Router as Tool Router
    participant IRC as Statutory Search (Versioned IRC DB)
    participant PLR as Precedent Search (Rev. Rulings / PLR DB)
    participant Validator as Temporal & Authority Validator
    participant Filter as Context Filter (Chunk Pruning / Token Budgeter)
    participant Memo as Draft Memo
    participant Linter as Citation & Math Linter
    participant HR as Human Review (CPA / Tax Counsel)
    participant Output as Final Output

    User->>GLM: Tax query / fact pattern
    GLM->>Router: Route to statutory & precedent search
    Router->>IRC: Search statute & Treasury Regulations
    Router->>PLR: Search Revenue Rulings & PLRs
    IRC->>Validator: Return candidate authorities
    PLR->>Validator: Return candidate authorities
    Validator->>Validator: Check hierarchy & effective dates
    Validator-->>GLM: Reject superseded / non-applicable authorities (retrieval failure)
    Validator->>Filter: Pass valid authorities
    Filter->>Filter: Prune chunks & enforce token budget
    Filter->>GLM: Return pruned context
    GLM->>Memo: Draft tax memo
    Memo->>Linter: Verify citations & computed values
    Linter->>HR: Verify valid citations, correct calc
    HR->>Output: Approve and release final memo

Second, you need a deterministic citation verifier. This isn't optional—it's the safety net that catches the hallucinated Treasury Regulation subsections that the strict rubric penalizes. Build a parser that extracts every cited section (§), regulation subsection, and case reporter citation, then checks each against a known-valid corpus. Any citation that doesn't resolve to a real authority gets flagged for correction or rejection before the output reaches a human. This catches the "syntactic plausibility" hallucinations that generalist models are prone to.

Third, route all arithmetic through a calculator sandbox. The failure mode there is token-level math drift inside legal prose. A Python sandbox that computes basis adjustments, E&P, and boot allocations deterministically—then inserts the numbers into the final memo—completely removes that failure class. The model should never be the one performing statutory arithmetic.

Fourth, implement chunk pruning and token budgeting. Production environments with uncurated search indices risk dumping tens of thousands of tokens of Treasury Regulations into context every turn. That's a 3–5x cost surge and a latency spike. Prune retrieved sections to the specific subsections that matter, and enforce a hard token ceiling per turn.

Finally, human-in-the-loop review is non-negotiable. The benchmark shows no model above 80%. That means one in five outputs has a compound error that an expert rubric would catch. Framing these as research drafts for CPA or tax counsel review, not as unsupervised client advice, is the only responsible deployment path.

Resources

• vals.ai