Tax Agent Bench: What 391 Expert-Written Tax Questions Reveal About Agentic Legal Reasoning

The evaluation problem tax research poses that generic benchmarks can't touch

Corporate tax research is a uniquely hostile evaluation target, and not for the reasons you'd expect. It's not that the domain knowledge is esoteric—it's that the output form and failure modes are fundamentally incompatible with how most benchmarks measure capability.

Think about what MMLU-Pro or GSM8K actually test. They ask a model to produce a single answer—a letter, a number, a string—that can be checked against ground truth. The reasoning path is invisible, or at best inferred from the final answer. That's fine for physics facts or arithmetic. It's catastrophic for tax research, where a model can arrive at the correct conclusion through a series of hallucinated citations, invented statutory cross-references, and wrong-year temporal reasoning, and still look like it succeeded on a final-answer metric.

The real problem is that there's no compiler for tax law. In software engineering, you can run the agent's code and see whether the tests pass. The environment itself enforces correctness on intermediate steps. Tax law has no equivalent. The Internal Revenue Code is not executable. A model can cite a nonexistent Treasury Regulation, apply a sunset provision that was already repealed, or misallocate basis across a multi-tiered partnership structure, and the final memo will still read plausibly. The only way to know whether the reasoning was sound is to have someone—or something—check every step along the way.

That's the core difficulty Tax Agent Bench is trying to solve. It doesn't just ask for the final tax answer. It asks for the research process to be verifiable, and it constructs ground truth by having tax lawyers and CPAs author rubrics that break each task into granular sub-checks. This is expensive. It's slow. And it's the only honest way to evaluate whether an agent can actually do the work, rather than just pattern-match to a plausible-looking conclusion.

There's also the malpractice dimension. Hallucinating a citation in a general-knowledge benchmark is a minor ding. Hallucinating a revenue ruling in an actual tax memo is a professional liability issue. Any benchmark that treats citation fidelity as a soft error is not measuring the thing that matters in practice. That's why Tax Agent Bench's rubric design treats a fabricated IRC section as a hard failure independent of whether the final conclusion happens to be correct—because in the real world, that's exactly how it would be treated.

sequenceDiagram
    participant User as Tax Researcher
    participant Agent as LLM Agent
    participant Tools as Retrieval Tools
    participant Rubric as Expert Rubric

    User->>Agent: Provide fact pattern and tax question
    Agent->>Tools: Initiate statutory lookup (IRC, Treasury Regs)
    Tools->>Agent: Return relevant provisions and dates
    Agent->>Tools: Search IRS guidance and court precedent
    Tools->>Agent: Return authoritative sources, hierarchy noted
    Agent->>Agent: Perform temporal analysis (year-specific thresholds)
    Agent->>Agent: Calculate intermediate tax liability/basis
    Agent->>Rubric: Submit research memorandum with citations
    Rubric->>Rubric: Run 3-89 sub-checks on steps and citations
    Rubric->>User: Return granular score with failure points

Inside the harness: how the agentic loop, retrieval tools, and memo synthesis fit together

The harness is where Tax Agent Bench diverges most sharply from static benchmarks. The model isn't handed a prompt and asked to reply. It's placed inside an agentic loop with a defined tool surface—lookup, document search, and statutory retrieval—that spans the full hierarchy of US tax authority: the Internal Revenue Code, Treasury Regulations, IRS guidance (revenue rulings, revenue procedures, notices, private letter rulings), and court precedent.

The agent receives a fact pattern—say, a Section 382 ownership change in a year where the TCJA's EBITDA-based interest limitation applies—and must work through it iteratively. It queries the statutory text, retrieves Treasury Regulations, checks whether a particular revenue ruling has been superseded, pulls relevant Tax Court opinions, and synthesizes all of it into a single formal research memorandum. Along the way, it's expected to resolve ambiguous thresholds, track date-dependent law changes, and make judgment calls about which authority governs.

What strikes me about this design is the temporal analysis requirement. The agent can't just retrieve the current version of the statute. It has to understand that the §163(j) limitation was different in 2017 than in 2021, and that the source year in the fact pattern determines which version applies. That's a retrieval problem, a reasoning problem, and a discipline problem all at once. A model that pulls the current text and applies it retroactively will fail the time-specific rubric checks even if its legal theory is directionally correct.

The memo synthesis step is also critical. This isn't a multiple-choice or short-answer task. The agent produces a structured artifact—the kind of output a tax associate would actually deliver. That means the rubric can evaluate not just the conclusion, but whether the memo cites authority properly, identifies areas of ambiguity, and distinguishes between binding precedent and non-binding administrative guidance.

There's a deliberate opacity here that I think is the right call. Vals AI keeps the exact internal tool APIs partially proprietary, both to prevent test set contamination and to stop teams from gaming the harness. The consequence is that you can't perfectly replicate the evaluation environment in-house—you can only approximate it. For a team building tax agents, this means the public leaderboard is a directional read, not a definitive one. Your retrieval index, your tool schemas, and your prompt formatting will all shift the score meaningfully.

flowchart LR
    A[Fact Pattern Input] --> B[Retrieval Tool Layer]
    B --> C[IRC Lookup]
    B --> D[Treasury Reg Search]
    B --> E[IRS Guidance Retrieval]
    B --> F[Case Law Search]
    C --> G[Temporal Analysis]
    D --> G
    E --> G
    F --> G
    G --> H[Precedent Hierarchy Check]
    H --> I[Intermediate Calculation Step]
    I --> J[Synthesis into Research Memorandum]
    J --> K[Rubric Scoring with 3-89 Sub-checks]

The rubric architecture: how 23.6 intermediate checks prevent the right answer via wrong reasoning

The rubric is the heart of this benchmark, and its design philosophy is worth unpacking. Each question carries between 3 and 89 individual sub-checks, averaging around 23.6. That range tells you something important: some tasks are relatively contained—a statutory lookup with a couple of verification points—while others are sprawling research projects that require dozens of independent facts to be correct.

Why so many checks? Because the benchmark is explicitly designed to prevent the "right answer via wrong reasoning" failure mode. The rubric doesn't just verify that the final conclusion matches professional consensus. It verifies the path. Every intermediate factual step, every temporal determination, every numeric calculation is scored independently. If an agent gets the final answer right but misstates the applicable interest deduction limit for the wrong tax year, the rubric catches it. If it fudges an intermediate basis calculation and the error compounds into a correct-looking final number, the sub-checks expose the flaw.

The citation authenticity checks deserve special attention. An agent cannot cite a revenue ruling that doesn't exist, or attribute a holding to a court that never made it, and still score well. The rubric verifies that each citation is authentic, relevant, and correctly positioned in the authority hierarchy. This is the malpractice guardrail, and it's why the benchmark is so demanding: a model that knows the right answer but cites it to the wrong authority fails on a dimension that matters more than the conclusion itself.

Now, the semantic equivalence problem is real, and I think it's the benchmark's most interesting tension. The rubric was authored by tax professionals—real CPAs and lawyers who wrote the checks the way they'd actually evaluate an associate's memo. If a model writes "ordinary income recapture under §1245" instead of "Section 1245 depreciation recapture gain," that's the same legal concept, but a rigid rubric might not recognize it. Strict grading here risks false negatives, punishing an agent for phrasing when the substance is correct. On the other hand, loosening the rubric risks letting sloppy answers slide through on semantic fuzziness. The tension is fundamental, and the benchmark's answer—using expert-authored rubrics with granular sub-checks—is a reasonable middle ground, even if it leaves some grading noise in the mix.

Evaluation Method	Granularity	Cost	Noise	What It Catches
Rubric sub-checks	Very high (3-89 steps per task)	High (expert-authored, expensive to maintain)	Low-moderate (semantic equivalence issues)	Intermediate reasoning, citation authenticity, temporal accuracy, compounding errors
Exact-match	Very low (single string/number)	Minimal (fully automated)	High (brittle, fails on phrasing)	Final answer only, nothing about reasoning path
LLM-as-judge	Moderate (holistic scoring)	Moderate (automated but requires prompt engineering)	Moderate (evaluator bias, drift)	Overall quality, semantic equivalence, but weak on precise numeric/temporal checks
Human expert review	Very high (any depth)	Very high (expensive, slow, not scalable)	Low (when experts agree)	Everything, but only at small scale; impractical for regular regression runs

Where Tax Agent Bench sits versus LegalBench, SWE-bench, and GAIA

The most useful way to position Tax Agent Bench is not as a competitor to any of these benchmarks but as a deliberate point of convergence across them. It takes the agentic tool-use architecture of SWE-bench, the domain depth of LegalBench, and the artifact-production requirement that neither of those fully captures, then adds a rubric layer to substitute for the compiler that SWE-bench gets for free.

Compared to LegalBench, the tradeoff is straightforward. LegalBench spans 162 tasks across the full legal reasoning spectrum—issue-spotting, contract clause classification, statutory interpretation. That breadth is genuinely valuable if you're evaluating general legal competence. But most of those tasks are single-turn. The model reads a prompt, produces an answer, gets scored. There's no iterative retrieval, no multi-step research, no expectation that the model will navigate a hierarchy of authority and synthesize it into a deliverable. Tax Agent Bench gives up that breadth for something narrower but operationally more realistic: an agent that has to work a problem, not just answer it.

Against SWE-bench, the comparison exposes the fundamental asymmetry between code and law. SWE-bench has a compiler, an execution environment, a test suite—deterministic ground truth that the agent's code either satisfies or doesn't. Tax law has none of that. There's no way to "run" a tax memo and check whether it compiles. The rubric is the compiler, and that's both the innovation and the weakness. A rubric is more flexible than a test suite—it can catch a hallucinated citation or a wrong-year phase-out threshold that no test would ever detect—but it's also noisier. Rubrics require interpretation, and interpretation introduces grading variance that a deterministic test suite simply doesn't have.

GAIA is the most interesting comparison because it occupies a similar agentic space with a fundamentally different scoring philosophy. GAIA tasks are multi-step, tool-using, generalist—but they're scored by exact-match on a final string or number. That's a critical constraint. It means GAIA can only be used for tasks that have a single unambiguous answer. Corporate tax research almost never works that way. A question about whether a particular transaction triggers Section 382 might have a defensible answer and an aggressive answer, depending on jurisdiction and filing posture. Tax Agent Bench's rubrics accommodate that ambiguity by scoring the reasoning path and the authority hierarchy, not just the conclusion. The cost is that you can't automate scoring as cheaply—and the benchmark is worse for it in terms of repeatability, but better in terms of what it actually measures.

Where Tax Agent Bench is genuinely better than all three: it's the only one that makes citation fidelity a first-class evaluated property. That's the malpractice guardrail, and it's what makes the benchmark feel like it's measuring professional work rather than academic exercise.

Authority hierarchy, circuit splits, and the rubric's blind spots

The failure mode that worries me most in this benchmark—and the one I'd watch hardest in any real tax agent—is the authority hierarchy inversion. Agents routinely treat a Private Letter Ruling or an IRS FAQ as if it carried the weight of a Treasury Regulation. That's not just a citation error; it's a fundamentally different legal theory that would get an associate pulled aside in a real memo review. The rubric catches this because citation authenticity checks verify not just that the citation exists, but that it's positioned correctly in the hierarchy. A PLR cited as controlling over a Treasury Regulation fails on a dimension that no final-answer metric would ever expose.

The harder problem is what I'd call rubric binary-ness against legitimate ambiguity. Corporate tax has genuine interpretive disputes—circuit splits, minority positions, aggressive-versus-conservative filing postures. A question about whether a transaction triggers Section 382 might have a defensible conservative reading and a defensible aggressive reading. The deterministic rubric was authored around one professional consensus, but if the prompt doesn't explicitly pin down jurisdiction and posture, the rubric effectively chooses a winner. An agent producing a valid alternative position—arguably the mark of a sophisticated tax mind—fails the sub-checks because they were written for a specific standard interpretation. That's a false negative, and it's baked into the deterministic design.

No benchmark in this space is just a benchmark. It's always a test of the entire harness—the retrieval engine, the tool schemas, the prompt formatting, the corpus versioning—as much as the model itself. Tax Agent Bench is no exception, and this is the thing I'd want teams to internalize before reading too much into a leaderboard number. A model evaluated against BM25 over the raw IRC will score drastically differently than one evaluated against a specialized legal hybrid search that Cross-references citators. That's not a subtle distinction; it's the difference between finding a relevant court opinion and finding the court opinion with the right precedential weight.

The rubric scoring mechanism itself introduces noise. With ~23.6 intermediate checks per question evaluated by an LLM-as-a-judge (or regex extractors), semantic equivalence becomes a binary trap: "ordinary income recapture under §1245" vs. "Section 1245 depreciation recapture gain" are the same legal concept, but a rigid sub-check might not recognize them as equivalent. That's a false negative on phrasing, not substance. Overly strict rubrics also punish omissions of implied intermediate premises that a seasoned practitioner would consider understood. The net effect is that benchmark scores carry grading variance that has nothing to do with the model's actual capability.

Then there's the maintenance problem, which I think is the most underappreciated. Tax law is point-in-time dependent and constantly drifting. The TCJA changes, new revenue procedures, sunset provisions, retroactive CARES Act modifications—all of it means the benchmark's corpus and rubrics require ongoing upkeep. A question about §163(j) interest deduction limits that was correctly scored last year might have a different correct answer this year if the EBITDA/EBIT threshold shifted. Whoever maintains this benchmark is committing to perpetual regulatory tracking, which is exactly the kind of tireless work that's impossible to sustain at full fidelity. The public number on a leaderboard is a snapshot of a moving target, not a stable property of any model.

So what should you take away from a Tax Agent Bench score? Not "this model is better than that model." At best, you can read it as "this model, in this harness, with this retrieval index, at this point in time, produced results aligned with this benchmark's specific interpretation of these tax questions." The variance between harnesses likely exceeds the variance between models. That's not a criticism of the benchmark—it's a warning about how you use it.

What I'd build differently: three concrete applications for teams that want to act on this

The benchmark's failure modes map so directly to engineering interventions that I'd want any team building a tax agent to act on them immediately. Here are the three projects I'd start with.

First, a point-in-time tax retrieval system with temporal metadata tagging. The entire temporal analysis failure mode—models retrieving the current statute for a historical tax year—has a concrete engineering fix at the index layer. Tag every IRC section, Treasury Regulation, revenue ruling, and PLR with effective dates and sunset dates. Then make the retriever filter by the tax year extracted from the fact pattern. The catch, and it's a subtle one: some provisions are retroactive. The CARES Act changed things that applied before its passage. If your retrieval layer rigidly filters only to documents effective on or before the question's tax year, you'll silently drop provisions that were in effect but only became visible later. You need a mechanism that surfaces current-but-retroactively-applicable provisions alongside the date-filtered set, so the agent can reason about both rather than defaulting to whichever the index returns. Over-filtering is the failure mode to watch, not under-filtering.

Second, an internal memo-format benchmark that detaches from the public harness. The harness variance problem is real, but it's only a problem if you're comparing apples to oranges. Take 50–100 Tax Agent Bench questions, run them through your own agent harness with your firm's memo template, your retrieval index, and your tool schemas—then compare your scores against the public leaderboard. The score gap is diagnostic, not damning. If you're 15 points below the leaderboard, the question is whether your retrieval index lacks the citator functionality that the benchmark harness has, not whether your model is fundamentally worse. This exercise converts an obscure benchmark score into a team-specific tuning action, and it tells you exactly what part of your stack is the bottleneck.

Third, a tax agent with mandatory calculator and citator gates. The two failure modes that create actual liability risk—numeric compounding errors and hallucinated citations—are both architectural problems, not prompt-engineering problems. Route all numeric sub-problems through a sandboxed Python code interpreter instead of letting the model do arithmetic in natural language. And route all citations through a validation step that checks against the IRC/Reg index before output—checking not just that the section exists, but that it remains good law for the relevant tax year. The agent will fight you on this; it'll want to skip the gates to save tokens. The delegation has to be enforced in the harness, not in the prompt. Prompt-level instruction is a suggestion; harness-level enforcement is a contract. And the citator check has to be temporal: a section that was good law in 2017 might have been repealed or amended by 2021, and "it exists" isn't the same as "it applies."

Resources

(no official sources were available to link)