Tracing the Limits: Where Microsoft Foundry's Agent Governance Actually Holds

The Observability Stack That Enterprise Agents Actually Emit

Most of the conversation about enterprise agent platforms fixates on the dashboards. That is the wrong place to look. A trace viewer is a rendering surface; the interesting engineering is in what gets emitted before anything is rendered, and whether the shape of that emission survives contact with a multi-agent loop. Foundry's real differentiator is not the portal — it is the adoption of OpenTelemetry GenAI semantic conventions as the wire format for agent telemetry. That decision matters because it means a nested subagent handoff is not a bespoke log line that only one vendor's UI can parse. It is a structured span with a defined shape.

Concretely, what a span carries is the whole game. Each agent interaction emits a span capturing discrete latency, token consumption split across input, output, and reasoning tokens, the tool's input and output JSON schemas, and the exception tree when something throws. That last item is undervalued. An exception tree attached to the span tells you which nested call failed, at what depth, with what arguments, rather than leaving you to correlate three disconnected log lines by timestamp and hope.

When you run Semantic Kernel, AutoGen, or LangGraph inside the Foundry Agent Service, the tracing decorators and middleware stream those spans out. Where do they go? Through the instrumentation hook and into Azure Monitor Application Insights, with a parallel rendering in the Foundry portal trace viewer. The pipeline is not a straight pipe from runtime to viewer — there is a collector in the middle, and that is where the leverage lives.

flowchart TD
    A[Agent Runtime<br/>Semantic Kernel / AutoGen / LangGraph] --> B[OTel GenAI Spans<br/>tool calls, nested subagent requests, retrieval subsets]
    B --> C[OpenTelemetry Collector gateway<br/>schema validation]
    C --> D[Foundry Trace Viewer]
    C --> E[Application Insights]
    C -->|sampling decision point| F[PowerBI<br/>burst analysis]
    D --> G[Content Safety<br/>continuous evaluators]

The collector is doing more work than people give it credit for. schema validation at that hop is where malformed tool payloads either get rejected or silently pass through and pollute your downstream queries. And the sampling decision point is the single most consequential configuration in the whole stack — I'll come back to it at length, because getting it wrong is how traces become useless precisely when you need them most.

Now contrast all of this with the naive approach, which is roughly "log everything the agent does and grep it later." For a single-turn chatbot that is fine. For an AutoGen-style multi-agent turn with iterative tool calling, it collapses. Each high-level task fans out into dozens or hundreds of nested operations, and if you are storing full execution context — prompt payloads, full system instructions, raw tool JSON, intermediate reasoning tokens — you are writing megabytes of telemetry per task. That is not a logging strategy, it is a denial-of-service attack on your own ingestion budget. The reason the semantic-conventions approach survives where naive logging does not is that the span structure lets you decide, per span, what is worth keeping. Structure is what makes sampling possible without losing meaning. Without it, sampling is just data loss with extra steps.

The framing I keep coming back to: observability for agents is not about collecting more, it is about collecting the right fields in a schema that lets you reconstruct a failure after the fact. Foundry gets the schema right. Whether your tracing survives production is a separate question, and it is almost entirely about that sampling decision.

Why Telemetry Cardinality Breaks Your Traces at Around 300 Spans

Let me put a number on the collapse. A single AutoGen-style multi-agent turn with tool calling can emit well over a hundred spans. Five agents, each making twenty tool calls, with a couple of retries per failed call: five times twenty times two is two hundred spans for one logical task. Two hundred spans, each carrying a prompt payload, a tool schema, a response body, token counts, and timing. Now run that at a hundred tasks per minute. That is twenty thousand spans a minute, north of twenty-eight million spans a day, before you count the nested subagent handoffs that each carry their own child spans.

Log Analytics ingestion caps are not theoretical at that point. You hit them. And when you hit them, the platform does not politely ask you to reduce volume — it starts sampling.

This is where the distinction between deterministic sampling and head-based sampling earns its keep, and where most teams learn it the hard way. Deterministic sampling means you make the keep/drop decision based on stable attributes, so a given trace ID always makes the same decision and you get reproducible coverage. Head-based sampling makes the decision at the root span, before the trace has finished — and every child span inherits that decision. The root is cheap to evaluate and the overhead is bounded, which is why it is the default. The problem is that at the moment the root span is created, you have no idea whether this trace is going to succeed or fail.

That is the crux. Head-based sampling throws away error paths with exactly the same probability as successful runs. If you are sampling at ten percent to stay under your ingestion ceiling, you are keeping one in ten failures. The one trace you desperately need — the multi-agent run where the third subagent picked the wrong tool, the transaction aborted halfway, and a downstream API timed out — has a ninety percent chance of having been discarded before it even completed. You are left with fragmented traces where the tool-failure spans and the safety evaluation events are simply gone. Reconstructing what happened across a distributed multi-agent failure when the failure spans were sampled out is not debugging, it is archaeology on a site someone else already bulldozed.

The correction I would push for is tail-based sampling: buffer spans, decide at the end of the trace, keep everything that errored or tripped an evaluator, and sample the boring successful ones aggressively. The cost is a collector that holds trace state in memory and a policy that knows what "boring" means. That is real operational work. But the alternative — head-based sampling at the root, deployed because it was the default — is quietly guaranteeing that your most important traces are the ones you never see.

The arithmetic makes the case on its own. Twenty-eight million spans a day at full payload is not a budget you can absorb, so you must sample. The only question is whether your sampling policy is aware of outcomes. Head-based sampling is not. If you take one thing from this section: the sampling decision point in your collector is a correctness concern, not a cost knob.

Guardrail Latency vs. Agent Deadlock: The 150ms Problem

Foundry splits its guardrails into two placements, and the split is the source of a specific pathology. Synchronous input/output filters — Azure AI Content Safety screening prompts before they reach the model and validating responses before they return — run inline. Asynchronous continuous evaluators for groundedness, relevance, and jailbreak detection run on the telemetry after the fact. The async side is cheap and safe. The sync side is where the arithmetic gets ugly.

Put a number on it. Real-time content filtering and DLP checks on every intermediate step add roughly 150 to 500 milliseconds per hop. A multi-turn agent executing eight sequential tool calls pays that tax eight times, and the checks are not additive in a friendly way — they compound against a latency budget the user is already feeling. Eight hops at 150ms is 1.2 seconds of pure guardrail overhead bolted onto a task that already involves multiple model round-trips. At the 500ms end it is four seconds. For an interactive agent that is the difference between "responsive" and "something is wrong with this thing."

Then there is the false-positive pathology, which is worse. Content safety classifiers routinely misclassify benign technical data as adversarial. A SQL query a developer's agent is legitimately executing, a shell script being run as part of a deployment, security logs passed as tool parameters — these get scored as injection attempts or toxicity because they look, at the token level, like the thing the classifier was trained to block. In a single-turn chatbot a false positive is annoying: the user rephrases. In a multi-step transaction, a false positive at hop five means the agent halts mid-flight, and if the tools it already invoked had side effects — a database write, a queued job, a partially created record — you now have a transaction in an invalid intermediate state with no graceful rollback. The guardrail did its job and created a worse problem than the one it prevented.

Dimension	Synchronous inline filtering (Foundry, AWS Bedrock)	Asynchronous post-hoc evaluation (Arize Phoenix, MLflow)
Latency per hop	150–500ms added per intermediate step; compounds across sequential tool calls	Near-zero inline cost; evaluation runs on telemetry after the response ships
False-positive reversibility	Abort is real-time and often irreversible — the halt already happened	Evaluation flags the run; a human or re-run can reverse course before consequences compound
Rollback complexity	High — agent stops mid-transaction, side-effecting tools may have already fired	Low — nothing was blocked, so nothing needs unwinding
Injection detection latency	Immediate; the request never reaches the model	Delayed until the evaluator processes the span; a bad request can already be executing
Debugability of a false positive	Hard — the trace shows a halt with no counterfactual; you cannot see what would have happened	Easier — the full run exists and can be inspected against the evaluator's flagged reasoning

The competitor contrast is instructive. AWS Bedrock Guardrails sits in the same synchronous inline camp as Foundry — it screens prompts and responses in the request path, buying immediate injection detection at the cost of the same latency and false-positive abort profile. Databricks Mosaic AI and the MLflow tracing stack, along with Arize Phoenix, place evaluation post-hoc: they let the run complete, evaluate the spans, and cluster behavioral drift without proxying every hop. That placement trades immediacy for reversibility. You catch an injection later, but a benign SQL query that looks malicious does not kill a live transaction.

My read: you want both, and the art is choosing which checks deserve the inline tax. Prompt-injection screening on the first hop of a sensitive transaction is worth 150ms. Running full toxicity classification on every intermediate tool result, including results your own system generated, is not. The failure mode I would watch for is treating the guardrail set as a monolith and paying the sync price across the board because configuring selective placement was more work than turning everything on.

Token Blowup in Autonomous Loops: Quadratic Auth and the 429 Cascade

Here is the math nobody puts in the architecture diagram. When a long-running agent retains full conversational context — the running history, prior tool calls, their results, intermediate evaluation scores — the input tokens per turn grow linearly with turn count. Each new turn re-sends everything that came before. But total spend is the sum across turns, so cumulative token cost grows quadratically. Turn one might cost a thousand input tokens. Turn ten is re-sending ten thousand. The sum over ten turns is not ten thousand, it is closer to fifty-five thousand. The agent is not getting smarter per turn; it is paying an ever-larger toll to remember what it already decided.

Now add the retry pathology. Suppose a tool schema is mis-specified — a required field misnamed, a type mismatch, an enum that does not include the value the model keeps supplying. The agent does not give up. It reads the validation error, appends it to context, and tries again. Each self-healing re-attempt preserves all prior context plus the previous failure plus the new attempt. Five or six of these and you have burned tens of thousands of tokens on a single tool that was never going to work, and the context is now so long that the model's tool-call formatting gets worse, not better.

sequenceDiagram
    participant A as Agent
    participant T as Tool API
    participant C as Context Store
    participant O as Azure OpenAI Deployment
    A->>T: Turn 1 tool call
    T-->>A: failure
    A->>C: append error, retry Turn 2
    C-->>A: context grows linearly
    A->>T: Turn 2 tool call
    T-->>A: failure
    Note over A,C: token spend grows quadratic
    A->>C: append error, retry ... Turn 6
    C-->>A: full history retained
    A->>O: Turn 6 request
    O-->>A: quota approached
    Note over O: TPM exceeded
    O-->>A: 429
    O-->>O: cascading 429s to concurrent workload on same pool
    Note over A: budget circuit breaker should intercept at turn 3

That last arrow is the part that hurts. Azure OpenAI deployments that share a TPM/RPM pool do not isolate one agent from another. A single rogue autonomous agent in a retry loop can exhaust the per-minute quota for the entire Foundry workspace, and every unrelated production workload on that same deployment starts getting 429 Too Many Requests. One badly specified tool schema takes down your whole tenant's traffic. This is the cascading failure, and Foundry's default per-request timeouts do not prevent it — they stop execution, but the tokens are already spent and the quota is already consumed. Timeout is a spend ceiling on a single call, not a budget ceiling on a trace.

This is exactly the gap the two build patterns below close, and if you are running anything autonomous in production I would treat them as requirements rather than enhancements.

The first is a Foundry-side step and loop circuit breaker. The idea is an interceptor on the telemetry side that tracks cumulative token spend, tool-call count, and repeated-tool-failure patterns per trace ID, then enforces hard cutoffs — abort at eight turns, abort at fifty thousand cumulative tokens, abort when the same tool fails twice with near-identical arguments. It fires by injecting a synthetic failure into the agent loop rather than waiting for a timeout. The wiring runs through an OpenTelemetry Collector custom processor, an Application Insights telemetry query, an Azure Monitor alert rule, and a Foundry Agent Service middleware hook. The trap to watch for is subtle and important: make sure the breaker's own telemetry does not get sampled out, because a breaker that fires silently is worse than no breaker. Route breaker firings to a separate low-volume, high-priority workspace so they survive whatever sampling policy governs everything else.

The second is a PII-scrub gateway ahead of ingestion. This is an OpenTelemetry Collector that processes spans before they hit Application Insights: detect PII via regex or Azure Content Safety, redact or strip prompt and completion payloads, but retain the span structure, tool names, timestamps, and token counts. The judgment call is asymmetric retention — drop full raw payloads for successful runs entirely, keep them for error and failure spans. You get debuggability exactly where you need it and pay the storage cost only for the traces that earn it. This connects to Purview DLP policy hooks and needs a low-volume debug workspace with its own access controls. The gotcha, and it bit me the first time I looked at one of these: Purview frequently fails to inspect nested JSON tool payloads as plain text, so your scrub rules must target structured JSON paths explicitly, not just scan the whole blob as text and hope the classifier finds a field named ssn buried four levels deep.

Both patterns share a theme: instrument the economics of the loop, not just its execution. Foundry gives you the span structure to do it. Whether the budget ceiling actually exists is up to you — and the arithmetic says it will not exist by default.

Session State Chaos: Race Conditions in Long-Running Multi-Agent Memory

Foundry decouples agent compute from state persistence, which is the right architecture and also the source of a nasty class of bugs. The runtime holds the reasoning loop; the session state lives elsewhere — Cosmos DB, Blob Storage, or a managed memory store. That separation is what lets a long-running agent survive a restart, but it means every write to shared session memory is a distributed write, and distributed writes under concurrency are where long-running multi-agent systems go to die.

Picture a fan-out. A coordinator agent dispatches three subagents to research different facets of a task, then a fourth to synthesize. The first three finish at roughly the same moment and each writes its findings into the shared session state. If those writes are read-modify-write on the same document, you have a classic lost-update race: subagent B reads the state, subagent C reads the same state, B writes, C writes, and B's contribution vanishes. The synthesis agent now reasons over an incomplete picture and produces a confident wrong answer — and nothing in the trace flags it, because each individual write succeeded.

The orphaned-lock case is worse in a quieter way. Suppose a subagent calls a downstream API and that call times out. If the agent fails to release the lock it took on a session document before dying, the state machine is now parked in an invalid intermediate state with a held lock. Nothing repairs it. Foundry gives you tracing to observe that the session is wedged, but there is no built-in state-repair path — no reconciliation loop that notices a lock older than its expected lifetime and reclaims it. You build that yourself, or you restart the session and lose the accumulated context. This is the recovery gap I'd call the single most under-planned part of long-running agent design: observability tells you that memory corrupted, not how to heal it.

Databricks Mosaic AI takes a different philosophical position. There, tools, vector indexes, and model endpoints are catalog objects governed through Unity Catalog, so state is derived from the catalog and the SQL layer rather than a mutable shared document. That is far less prone to races — you are not doing read-modify-write on a blob, you are querying governed objects — but it is also more constrained: the state model is whatever the catalog can express. Foundry trades that determinism for flexibility, and the flexibility is real. You just pay for it in concurrency bugs that the platform will show you and not fix.

My position: treat every write to shared session state as a critical section, version your session documents, and build an explicit lock-reclaim sweep. Foundry will not do it for you.

Identity Delegation Through Subagent Chains: The OBO Break Point

This one is subtle enough that teams discover it in an audit, not in a test.

Agents in Foundry run under dedicated managed identities authenticated through Microsoft Entra ID, enforcing RBAC and least-privilege tool execution. That works cleanly for a single agent acting on its own behalf. It gets complicated the moment you want the agent to act with the user's permissions rather than its own — because now you need delegated identity, and delegated identity in a multi-hop subagent chain is where things break.

The mechanism you reach for is On-Behalf-Of token exchange. A user authenticates, the front-end agent receives a token scoped to that user, and OBO lets the agent trade that token for a downstream token so the next service sees the user's identity preserved. At one hop this is well-trodden. The break point is the second hop. Each subagent in the chain has to re-acquire the user's context and exchange its token for the next one, and OBO was designed for a linear delegation chain, not for fan-out. When a coordinator spawns five subagents that each need the user's delegated permissions to reach three separate APIs, you are now designing a token-exchange graph: acquire, cache, refresh across parallel branches, then fan the results back in without losing the user binding on any branch. Entra ID will do this, but the OAuth flow design is no longer a configuration — it is an application.

Here is the concrete failure I'd watch for. Without that token-exchange graph working correctly, the agent quietly falls back to its own managed identity and calls the finance API with service principal privileges. The call succeeds. The data comes back. The agent completes the task. Nothing errors, nothing alerts, and the trace looks clean — because from the system's perspective the request was authorized, just authorized as the agent, not as the user who was supposed to be constrained to their own records. The over-privilege leak is invisible until someone in audit asks why this service principal has been reading records it was never granted user-level access to.

AWS's model is cruder but easier to reason about: each action group in Bedrock Agents invokes a Lambda with an IAM role boundary, so the privilege attached to any invocation is explicit and inspectable at the role level. You give up the user-context granularity — you are not carrying the user's identity through the call — but you also cannot silently drift from user permissions to service permissions, because there were no user permissions in the path to begin with. Foundry's entitlement is more precise when it works and more dangerous when it silently degrades.

My rule: any subagent chain crossing a trust boundary gets an explicit assertion that the caller's identity is the user's delegated identity, checked at every hop, and a hard fail — not a fallback — when the exchange cannot complete.

Sampling for Your Life: Tiered Observability That Survives Production

Everything up to here converges on one build decision: your retention policy is a correctness surface, and "log everything into Log Analytics" is not a conservative default — it is the failure mode that guarantees the traces you need most are the ones you lost.

The tiering I run with is explicit. Raw prompt and completion payloads: kept for one hundred percent of failure spans, one hundred percent of spans where a guardrail fired, and roughly five percent of successful runs via a deterministic sampling bucket. Everything else keeps span structure, tool names, timestamps, token counts, and latency — the metadata — with prompt and tool payloads pruned. That gives you two workspaces with different jobs. A hot workspace for debugging, where full context lives briefly and access is tightly scoped, and a cold retention tier that persists metadata forever and full payloads only on violations. The five percent success sample is not for debugging individual runs; it is for behavioral drift detection and cost attribution, and five percent is plenty for that.

The sampling-placement question is the one people get wrong. Head-based sampling at the generator is cheap and bounded, but as I argued earlier it throws away errors at the same rate as successes — at ten percent you keep one in ten failures, which is exactly backwards. Tail-based sampling at the collector buffers trace state in memory and decides at the end, so error and guardrail-fired traces are retained unconditionally while boring successes are aggressively dropped. The cost is collector memory and policy complexity; the payoff is that your failure coverage is one hundred percent instead of probabilistic.

flowchart TD
    A[Span enters OTel Collector] --> B{Is it a failure span?}
    B -->|Yes| F[Full trace storage<br/>retain all attributes + payloads]
    B -->|No| C{Is it a success span?}
    C -->|guardrail fired| G[Always retained with full context<br/>violation workspace]
    C -->|Yes| D{Sampling bucket roll?}
    D -->|5% bucket| F
    D -->|95% bucket| E[Prune attributes<br/>keep metadata<br/>drop prompt/tool payloads]
    E --> H[Cold retention<br/>metadata only]
    F --> I[Hot debug workspace]
    G --> I
    H --> J[Cold retention workspace<br/>full payloads only on violations]
    C -->|No, indeterminate| E

The counterintuitive part is that pruning is the thing that makes debugging possible. When you store full payloads for everything, ingestion caps force the platform to sample for you — and it samples blind, at the root, unaware of outcomes. When you prune deliberately and spend your full-context budget only on failures and violations, you get complete coverage of the runs that matter and a bounded cost for the rest. The team that says "we'll just debug everything in Log Analytics" is the team whose failures get sampled into fragments. Decide what earns full context, enforce it at the collector, and keep the failure path uncircumcised.

The Real Reception Test: Purview DLP vs. Nested Tool JSON

Purview is the reason a lot of risk officers sign off on Foundry, and it is also the governance control that degrades most quietly in an agent workload. The mechanism is simple enough: Purview classifies text, applies sensitivity labels, and enforces DLP on content it can read. The problem is what agents actually pass around. A custom REST tool returns a nested JSON object — a customer record with embedded arrays of line items, a nested compliance field, a base64 attachment reference four levels down — and that object flows into the agent's reasoning context. Purview's scanning is oriented toward text and documents, not toward arbitrary structured tool responses, so a field that would absolutely carry a sensitivity label if it appeared in a SharePoint document or an email body slides through unlabeled when it arrives as a JSON key nested inside a tool's return value. The agent consumes it and reasons over it. Nothing flagged it.

This is the vanguard guardrail failure in miniature. The prompt-middleware model inspects the composed request — the prompt after the agent has already assembled tool results into context. By that point the restricted data is inside the loop. You are screening at the wrong end of the pipeline.

Contrast Databricks Mosaic AI, where tools, vector indexes, and model endpoints are Unity Catalog objects. Access control is enforced at the query engine and storage layer, so a restricted row or table is never returned to the agent in the first place. The governance sees the data before composition, not after. Foundry's model is post-composition: by the time you screen, the agent already holds the payload. That is not a bug you fix with a better classifier — it is an architectural ordering, and it is why agent-to-agent communication compounds it. When subagent A hands nested tool output to subagent B as message content, the payload crosses a trust boundary as opaque JSON that your prompt-level scanners were never positioned to parse.

What this means in practice: if your agents touch regulated data through custom REST tools, you cannot rely on Purview alone. You enforce schema-level DLP at the tool boundary — validate and label tool responses before they enter context — and you treat every subagent handoff as a place where classification must be re-asserted rather than assumed to have survived.

What 12 Months of Model Auto-Update Does to an Evaluation Baseline

There is an unglamorous failure that does not announce itself with an error, and it deserves more attention than the dramatic ones. When a deployment is pinned to "auto-update to latest default," the backing model version can change underneath you. Reasoning behavior shifts, tool-call formatting shifts, prompt sensitivity shifts — overnight, without a deployment event anyone on your team scheduled. Your pre-production evaluation baselines were computed against the old weights. They are now stale in a way that is invisible, because the eval harness still runs and still produces numbers; the numbers just no longer describe the model you are serving.

The trap is that this looks like drift in your prompts or your tool schemas, so teams burn weeks tuning prompts to recover scores that fell because the model changed, not because anything they controlled changed.

I treat evaluation reruns as CI, not as a quarterly chore. The pattern: a workflow triggers on the Azure OpenAI deployment version UPDATE event, runs the full Foundry built-in evaluator suite — groundedness, relevance, tool selection accuracy, goal completion — against a golden query set, and compares against the prior run's scores. If groundedness or goal-completion regresses past a threshold, the promotion blocks. That is the only version of this that actually catches the change, because it is wired to the event that causes it rather than to a human remembering to re-run evals. The build shape is a GitHub Actions workflow calling the Foundry evaluation API, an Azure Container Apps job as the runner, and App Insights holding the comparison data between runs. The gotcha is that a golden set rots: it reflects the query distribution you had when you wrote it, not the one in production. Add scheduled small batches derived from production span scenarios so the set tracks reality instead of freezing around it.

Now the harder judgment — when the churn is acceptable and when it is catastrophic. The decision is not technical, it is about blast radius. A non-regulatory front-end agent that drafts text or summarizes research can live on latest-default and absorb a behavior shift; worst case a user notices slightly different output, and your eval regression gate catches the genuinely bad case. A financial transaction workflow is the opposite: the model's tool-call formatting and reasoning are load-bearing, a shifted behavior can cause a wrong tool invocation with side effects, and the consequences are not reversible by rephrasing. For anything with irreversible side effects or a compliance surface, you pin the version explicitly, you treat a version bump as a change request with its own eval gate, and you never let "latest" be a deployment property you set once and forgot.

The Countermeasure for Guardrail False Positives: Compensating Transactions

Given that a synchronous guardrail can abort an agent mid-flight at hop five — after a database write has landed, after a job has been queued, after a record has been partially created — the conclusion is uncomfortable but unavoidable: idempotency and compensating transactions have to be architectural constraints, not afterthoughts bolted on when a false positive first bites.

The reason this is non-negotiable in Foundry specifically is the abort profile I described earlier. Inline content filtering and DLP checks run on intermediate steps, and false positives on benign technical payloads — SQL, shell scripts, security logs — are routine. Each of those false positives is a potential mid-transaction halt with side effects already committed. You cannot make the classifier correct. You can make the consequences recoverable.

The design move is a tool wrapper layer. Every tool that touches an external system goes behind a wrapper that enforces three things: the operation is idempotent (a retried call with the same arguments produces the same end state, not a duplicate row), the wrapper registers a compensating action when it performs a side effect, and the compensation is recorded in durable storage keyed by trace ID so it survives an agent crash. When a guardrail aborts the loop, a rollback coordinator reads the registered compensations for that trace and issues them in reverse order. The model never has to understand rollback; the infrastructure handles it.

This is why I think agents destined to run untethered must be modeled as explicit state machines with named transition points, not as an opaque reasoning loop. The state machine gives you the rollback points — you know exactly which states committed side effects and which did not, so the compensation logic is deterministic even though the agent's path through it is not.

Side effect	Native rollback	Compensating transaction	Deferred execution (two-phase commit)
DB write	Ideal — wrap the write in a transaction and roll back on abort. If the agent owns the connection, native rollback is the cleanest option and needs no compensation logic.	Needed when the write lands across a service boundary the agent cannot transactionally span; issue a delete/update that restores prior state, recorded per trace ID.	Use when a write must be coordinated with other tool calls before any commit; stage the write, commit only after all hops succeed, abort cleanly if any guardrail fires.
Email send	None. You cannot unsend.	The only workable option — on abort, send a correction or a "please disregard" follow-up. Requires the original send to be logged with enough context to compose the correction.	Strong fit — stage the message body, hold the send until the transaction commits, discard on abort so nothing is sent at all.
Financial transfer	None in practice — rarely reversible without counterparty cooperation.	Mandatory — a paired reversal transfer, which is why every transfer tool must be idempotent and carry a compensating reversal registered the moment it commits.	Preferred architecture — authorize and reserve at hop N, capture only after all checks pass; a mid-flight abort releases the hold with no money moved.
Data retrieval cache	Trivial — drop the cache entry; retrieval has no external side effect to unwind.	Unnecessary — a failed read leaves nothing to compensate, so the retrieval tool needs no rollback path at all.	Unnecessary — reads are safe to repeat, so deferral adds latency without buying recoverability.

Read the table as a triage. Reads and cache fills need nothing. Transactional writes that the agent owns get native rollback. Anything with an irreversible external effect — an email, a transfer — gets a compensating transaction, always, registered at commit time so it is available even if the agent dies. And anything that spans multiple tool calls with a shared commit point gets two-phase execution, because deferring the commit is the only way to guarantee a guardrail abort leaves the world untouched. The wrapper layer enforces the discipline; the agent just calls tools and never has to know which category it is in.

Resources

(no official sources were available to link)