Engineering the Enterprise MCP Gateway: How Uber Scaled Tool Protocols Across Ten Thousand Microservices

Why the N-by-M Server Sprawl Breaks Down at Ten Thousand Microservices

When developers first experiment with Anthropic’s Model Context Protocol (MCP), the architecture seems refreshingly straightforward: you spin up a dedicated server process in Node or Python, wire it over stdio or HTTP Server-Sent Events (SSE) to a local agent harness like Claude Code, and expose a handful of tool definitions. This standalone model is completely defensible for a solo engineer or a single product surface.

In a distributed enterprise operating over 10,000 internal microservices, that pattern collapses into an unmaintainable operational catastrophe.

If you mandate that each engineering team author, host, containerize, and maintain their own bespoke MCP server wrapper, you introduce an untenable $N \times M$ deployment topology. Every service team must now manage independent runtime sidecars, track protocol SDK upgrades, maintain custom schema translation boilerplate, and configure bespoke network routing. Even worse, standard stdio process pipes cannot traverse enterprise network topologies, while fragmented HTTP/SSE endpoints create an unmonitored security surface that bypasses corporate identity tokens, rate limiters, and circuit breakers.

Beyond operational sprawl, local tool servers assume a point-to-point connection where an agent runtime knows upfront which servers to connect to. In an ecosystem spanning thousands of legacy polyglot services—communicating across HTTP/REST, gRPC, and TChannel—no human developer, and certainly no single agent prompt, can manage thousands of static connection strings.

The structural solution is decoupling tool management into a dual-plane architecture:

1. A Centralized MCP Registry (Control Plane): Serves as the single authoritative catalog for tool schemas, version pinning, ownership metadata, access policies, and operational status.
2. A High-Throughput Proxy Gateway (Data Plane): Operates as a universal protocol bridge. Agent clients connect to a single gateway endpoint speaking standard MCP JSON-RPC, while the gateway dynamically converts requests into native RPC transports, resolves service mesh locations, validates caller claims, and marshals responses back into MCP format.

By decoupling discovery from proxy execution, service teams never write standalone MCP boilerplate. The platform absorbs network transport, distributed tracing, and protocol bridging centrally, turning enterprise microservices into agent tools without increasing the organization's microservice operational footprint.

Harvesting the Mesh: Automated IDL Mining and the Lifecycle of a Tool Contract

The biggest hurdle to establishing a unified tool plane is onboarding friction. Asking thousands of busy infrastructure and product teams to hand-craft JSON Schema descriptors for their endpoints guarantees near-zero adoption. Uber bypassed this by realizing that an enterprise service mesh already possesses an exhaustive source of truth: Interface Definition Language (IDL) repositories containing Protobuf and Thrift definitions.

To harvest these definitions without developer toil, Uber implemented AutoCrawler, an automated background discovery engine orchestrated via Cadence workflows. AutoCrawler continuously inspects internal IDL code repositories across more than 10,000 services. It parses structured method signatures, request parameters, and response types directly from Thrift and Protobuf files.

However, raw IDLs make terrible LLM tools. Engineers routinely name fields using terse abbreviations, cryptic acronyms, and sparse type hints that confuse frontier models during parameter inference. AutoCrawler bridges this semantic gap by piping extracted method contracts through an LLM enrichment pipeline. The model parses surrounding comments, infers intent, fleshes out human- and agent-readable parameter descriptions, and synthesizes standard JSON Schema tool representations.

Crucially, discovery does not imply exposure. Exposing raw production RPCs directly to autonomous agents creates an immediate blast-radius disaster. Uber enforces a strict operational principle: every discovered tool starts disabled by default.

To promote an endpoint from discovered to invocable, service owners must actively submit a declarative configuration change via a pull request. Automated CI/CD pipelines evaluate these PRs against strict security guardrails—flagging risky write or mutation endpoints, requiring PII sanitization filters, and verifying rate-limit boundaries. Only after human peer review and automated policy approvals does the tool enter the active Registry state.

stateDiagram-v2
    [*] --> Discovered: AutoCrawler parses IDL (Protobuf/Thrift)
    Discovered --> SchemaEnrichment: LLM expands parameter names & descriptions
    SchemaEnrichment --> InactiveStaging: Staged in Registry (Disabled by default)
    InactiveStaging --> PRReview: Service owner submits enablement PR
    PRReview --> PolicyCheck: Automated CI checks (Mutations, PII filters)
    PolicyCheck --> PRReview: Violations flagged / changes requested
    PolicyCheck --> Active: Approved & merged into Registry
    Active --> ProxyExecution: Servicing agent tool calls via Gateway
    Active --> Deprecated: IDL removed or service retired
    Deprecated --> [*]

This lifecycle ensures that the tool catalog stays continually synchronized with production contracts while maintaining a hard verification boundary before an autonomous agent can touch a backend endpoint.

The Proxy Data Plane: Translating JSON-RPC Across Heterogeneous Protocols

Once an agent selects an active tool, execution shifts from the catalog to the Proxy Gateway data plane. AI agent runtimes—whether developer terminal agents like Claude Code, IDE CLI tools like aifx, or autonomous orchestration workers like Uber's internal Minion—communicate strictly over standard MCP JSON-RPC protocols. Downstream production backends, however, expect high-performance binary protocols routed through Uber's internal service mesh, Muttley.

The gateway’s primary operational responsibility is real-time protocol translation and context protection.

When an agent issues a tools/call JSON-RPC payload, the gateway intercepts the request and verifies the caller's delegated identity tokens. Rather than allowing the gateway to act as an unconstrained proxy with elevated privileges, it unpacks the engineer’s cryptographic identity alongside the active agent session ID.

Next, the proxy resolves the target service's native protocol and interface definition from the Registry. It dynamically transcodes the JSON-RPC arguments into structured binary payloads—such as Protobuf over gRPC or Thrift over TChannel—and dispatches the invocation across the Muttley mesh. This leverages Uber's battle-tested infrastructure: circuit breakers, dynamic service discovery, connection pooling, and distributed tracing spans are inherited automatically without custom code.

sequenceDiagram
    autonumber
    actor Developer
    participant Agent as Agent (Claude Code / Minion)
    participant Gateway as MCP Proxy Gateway
    participant Registry as MCP Registry
    participant Mesh as Muttley Service Mesh
    participant Backend as Microservice (gRPC / TChannel)

    Developer->>Agent: Prompt requiring internal service invocation
    Agent->>Gateway: POST /mcp (tools/call JSON-RPC payload)
    Gateway->>Registry: Fetch schema contract & target transport
    Registry-->>Gateway: Tool metadata & protocol mapping
    Gateway->>Gateway: Validate auth & serialize to binary RPC
    Gateway->>Mesh: Dispatch binary RPC via mTLS
    Mesh->>Backend: Forward call (gRPC / TChannel)
    Backend-->>Mesh: Return raw enterprise payload
    Mesh-->>Gateway: Forward raw binary payload
    Gateway->>Gateway: Response projection (strip unneeded nested fields)
    Gateway->>Gateway: Serialize filtered data to JSON-RPC
    Gateway-->>Agent: 200 OK (Clean JSON-RPC response)
    Agent-->>Developer: Stream synthesized reasoning

The reverse path holds the most critical reliability optimization: response projection.

Enterprise microservices are engineered for inter-service communication and frequently emit deeply nested objects filled with internal tracking IDs, debug metadata, pagination cursors, and redundant status hashes. Dumping an unfiltered 2 MB enterprise payload into an agent runtime blows past LLM context boundaries, triggers catastrophic token spend, and degrades model reasoning.

Before serializing the backend response into a JSON-RPC response payload, the gateway applies declarative response projections. These projection rules prune extraneous branches, drop unrequested debug metadata, and flatten verbose nesting into compact, semantically dense JSON. By the time the response reaches the agent, it contains only the fields strictly required to answer the prompt, keeping token footprints lean and downstream reasoning coherent.

Context Window Economics: Progressive Tool Discovery and Code Mode Offloading

Frontier models claim multi-million-token context windows, but treating that headroom as a dumping ground for enterprise tool schemas is an architectural blunder. In an ecosystem of 10,000 microservices, serializing even a modest subset of detailed JSON schemas upfront consumes dozens of thousands of tokens before an agent writes a single token of thought. Worse than the direct cost is attention degradation: when an LLM faces hundreds of active schemas, tool selection accuracy plummets, parameter hallucination rises, and the model struggles to differentiate identically named entities across domain boundaries.

Uber resolves this upfront token tax using an approach called Omni MCP, which treats tool discovery as a progressive runtime pattern rather than a static connection initialization. Instead of mounting a catalog dump, the client agent boots with four minimal meta-tools:

• discover_server: locates candidate backend systems by service domain.
• discover_tools: queries tools within a specific namespace matching a semantic task.
• get_tool_schema: fetches the precise, enriched JSON contract for a single selected operation.
• invoke_tool: dispatches execution through the Proxy Gateway once parameter schemas are loaded.

┌─────────────────────────────────────────────────────────────┐
│               Progressive Discovery Sequence                │
└─────────────────────────────────────────────────────────────┘
  1. Boot with meta-tools   ──> discover_server / discover_tools
  2. Targeted resolution    ──> get_tool_schema(target_tool)
  3. Narrow invocation      ──> invoke_tool(args)

By decoupling discovery from schema loading, the agent operates with a lean context footprint, pulling full parameter specifications into active memory only when it commits to invoking a target endpoint.

However, progressive schema loading solves only half the context problem. The return path presents an equally severe failure mode: payload blowup. Production RPCs routinely return dense, heavily nested multi-megabyte payloads containing internal tracking metrics, debug traces, and complete entity graphs. Even with gateway-level response projection pruning extraneous branches, enterprise queries frequently exceed what an agent's reasoning window can digest without truncating earlier conversation turns.

To handle large outputs without polluting the prompt, Uber introduced Code Mode via its developer CLI, aifx. For agents running in terminal or IDE environments (such as Claude Code or custom harnesses discussed in Deconstructing Always-On Agents: The Event Loop, Sandboxed Harnesses, and Context State Engineering), large payloads bypass the conversational LLM message block entirely. The gateway writes raw payloads directly to an ephemeral local sandbox filesystem.

Instead of forcing a 50,000-line JSON object into context, aifx injects a compact file handle reference into the agent prompt:

# Gateway offloads payload to local sandbox
/tmp/aifx-sandbox/session-8492/trip_details_20261012.json [2.4 MB]

The agent then uses standard POSIX shell tools (jq, grep, head) inside its sandbox to inspect specific fields deterministically. This shifts context consumption from an unconstrained $O(N)$ payload dump to an $O(1)$ file pointer, allowing the agent to paginate, slice, and verify nested data without burning token budget.

Governing Autonomous Blast Radius: Identity Delegation and Idempotent Retries

The moment an MCP gateway abstracts thousands of internal RPCs into accessible agent tools, it inherits the classic confused deputy problem. In a traditional service mesh, microservices authenticate caller identities via mutual TLS (mTLS) and service-to-service service accounts. If an agent executes through an enterprise gateway that makes downstream RPC calls using its own ambient service credentials, the gateway acts as an all-powerful deputy, effectively bypassing fine-grained user permissions and audit controls.

To prevent privilege escalation, Uber enforces strict identity delegation across the Muttley service mesh. The gateway strips ambient proxy privileges from the call path and mints cryptographically signed delegation tokens containing a dual assertion:

1. The Human Operator Identity: The verified corporate credentials of the developer directing the agent session.
2. The Autonomous Execution Context: The active session ID, parent task identifier, and the specific agent runtime harness (e.g., Minion or aifx).

┌──────────────────────────────────────────────────────────────────┐
│                   Dual-Assertion Auth Headers                    │
├────────────────────────────────┬─────────────────────────────────┤
│ X-Caller-Identity-User         │ developer@uber.com (mTLS/OIDC)  │
├────────────────────────────────┼─────────────────────────────────┤
│ X-Caller-Identity-Agent-Session│ minion-sess-92f1-48ba           │
├────────────────────────────────┼─────────────────────────────────┤
│ X-Gateway-Attestation-Signature│ <cryptographic-token>           │
└────────────────────────────────┴─────────────────────────────────┘

Downstream services evaluate authorization policies against both headers. If an engineer lacks permission to invoke an endpoint in production, an agent executing on their behalf is blocked at the target service boundary. Crucially, access logs retain the full lineage trace, mapping every database mutation or downstream RPC dispatch back to both the approving engineer and the autonomous execution loop that triggered it.

A subtler, potentially catastrophic operational hazard lies in distributed network timeouts and agent retry loops. Standard conversational AI clients maintain aggressive tool-call timeouts—typically between 30 and 60 seconds. In contrast, complex enterprise RPCs that trigger asynchronous Cadence workflows, multi-cluster batch reads, or downstream mesh fanouts routinely exceed that window.

When a network connection times out at the MCP proxy boundary, the client agent experiences an abrupt failure. Because LLMs are tuned to self-correct and recover from errors, frontier models will almost immediately regenerate the tool call and retry the identical invocation.

Agent Turn N:   tools/call(UpdateFeatureFlag)  ──>  [Timeout after 45s]
Agent Turn N+1: "Retrying previous call..."    ──>  tools/call(UpdateFeatureFlag)
                                                    ↳ DUPLICATE MUTATION!

If the underlying backend RPC lacks strict idempotency controls, this rapid retry behavior causes duplicate writes, repeated state mutations, or corrupt records.

The gateway mitigates this by enforcing deterministic, turn-level idempotency keys generated directly from the agent's session ID and execution turn counter:

$$\text{IdempotencyKey} = \text{HMAC-SHA256}(\text{SessionID} \parallel \text{TurnCounter} \parallel \text{ToolCallID})$$

The Proxy Gateway injects this key into standard mesh transport headers across every dispatched gRPC or TChannel request. When an agent experiences a transport timeout and retries an identical turn, downstream backends recognize the repeated key, deduplicate the mutation, and return the cached response. For operations known to exceed 10 seconds, the gateway converts the invocation into an asynchronous polling job, returning an immediate status handle to prevent client disconnects entirely.

Architectural Tradeoffs: Custom Mesh Gateways Versus Off-the-Shelf AI Proxies

When evaluating Uber’s architecture, my immediate instinct as an operator is to separate platform necessity from organizational inertia. Building a bespoke IDL-to-mesh data plane makes complete sense when you are sitting on ten thousand microservices communicating over heterogeneous binary protocols like TChannel and gRPC via an internal mesh like Muttley. If that is your operational footprint, off-the-shelf tools simply cannot speak your dialect. For almost everyone else, copying this architecture verbatim is textbook over-engineering.

The market has fractured into distinct architectural archetypes, each answering a different operational problem:

Architecture Archetype	Discovery Model	Protocol Support	Auth Delegation	Operational Overhead
Custom IDL Mesh Bridge (Uber)	Automated IDL mining (Protobuf/Thrift via Cadence) + LLM enrichment	Polyglot binary RPC (gRPC, TChannel, HTTP/REST)	Cryptographic dual assertion (Human OIDC + Agent session)	Very High (Bespoke platform team, internal mesh tie-in)
Traditional AI Gateways (Kong AI Gateway, Tyk)	Config-driven / OpenAPI route ingestion	HTTP/REST, SSE, WebSockets to upstream REST/GraphQL	Per-route API keys, OAuth 2.1 scopes, standard WAF tokens	Low to Medium (Plugs into existing reverse-proxy infrastructure)
SaaS Tool Hubs (Composio, Merge.dev)	Managed catalog of public third-party SaaS tools	Standardized HTTP wrappers to external vendor APIs	Managed user-level OAuth credential stores	Low (Fully managed vendor integration plane)
Open-Core Catalogs (Obot, TrueFoundry)	Containerized registry UI with manual server onboarding	Standard MCP JSON-RPC over stdio or HTTP/SSE	Team RBAC, ambient server-level environment variables	Medium (Managing Kubernetes-native MCP server deployments)

Traditional reverse proxies like Kong and Tyk treat MCP JSON-RPC as another upstream route. They bring battle-tested rate limiting, circuit breaking, and RFC 9728 token exchanges to agent calls out of the box, but they assume upstream services already expose standard HTTP contracts. They will not crawl an internal Protobuf repository or dynamically convert a JSON-RPC invocation into a Thrift struct.

SaaS tool brokers like Composio solve the opposite headache: authenticating third-party applications like GitHub, Slack, and Salesforce. If your agentic workflows focus on enterprise SaaS automation rather than internal database mutations, an internal IDL bridge is irrelevant. Meanwhile, open-core catalogs like Obot provide clean management interfaces for running containerized MCP servers, but they retain the $N \times M$ operational tax of maintaining independent server wrappers.

I would draw a hard boundary here: reach for a custom mesh bridge only when internal binary RPC protocols cannot be bridged by standard HTTP gateways, or when the cost of manual OpenAPI curation across thousands of internal endpoints demonstrably exceeds the cost of staffing a dedicated platform team. If your services already speak HTTP and conform to OpenAPI, an off-the-shelf reverse proxy paired with smart discovery layers delivers eighty percent of the value at a fraction of the operational debt.

Practical Implementations: Adapting Gateway Patterns to Smaller Engineering Stacks

You do not need a multi-cluster service mesh or proprietary workflow engines to capture the mechanical benefits of Uber’s gateway. Stripping away the enterprise substrate reveals three patterns that any mid-sized engineering team can implement using standard developer tooling.

The first adaptation addresses zero-touch onboarding through a CI-Driven OpenAPI-to-MCP Catalog Generator. Instead of standing up continuous crawlers, teams can implement an automated GitHub Actions pipeline that triggers on merge whenever internal services update their OpenAPI 3.1 specifications. A lightweight Python script extracts route definitions, pipes undocumented or abbreviated query parameters through an inexpensive model to synthesize agent-readable descriptions, and uploads versioned MCP tool manifests directly to a private S3 bucket or a lightweight FastAPI service. The critical failure mode I watch for here is silent schema drift: if an engineering team marks an optional query parameter as required in downstream code without bumping the OpenAPI spec version, the cached MCP schema causes the agent to assemble broken invocation payloads that fail validation at runtime.

OpenAPI 3.1 Spec  ──>  GitHub Actions  ──>  LLM Schema Enrichment  ──>  Private S3 / FastAPI Registry

To solve token bloat without building a distributed registry, teams can construct a Progressive Meta-Tool Dispatcher for Existing Reverse Proxies. Instead of mounting a bloated catalog directly into client harnesses, write a compact Go or Node.js MCP server that mounts in front of your existing Kong or Envoy gateway. This server exposes just four meta-tools: list_categories, search_tools, get_schema, and execute_tool. The server caches tool definitions and vector embeddings in Redis, letting the agent search semantic domains dynamically while routing execution calls through Envoy for mTLS and API key injection. The operational hazard to guard against is discovery thrashing: if multiple services share overlapping semantic domain labels—such as billing, invoice, and payment endpoints—the agent will spend multiple iterative round-trips searching for schemas rather than completing the task. Namespace your domain scopes strictly before vectorizing them.

Finally, you can implement context offloading locally using Disk-Buffered Shell Tooling for Large Payload Microservices. When internal endpoints return multi-megabyte enterprise responses, wrap your execution tools in a Python or Go layer that inspects payload size. If an upstream response exceeds an 8 KB ceiling, the wrapper writes the raw JSON object to a local tmpfs mount or isolated container volume, returning only a compact file handle along with a few pre-computed jq recipes. The agent can then use standard POSIX utilities inside its execution sandbox to query the specific fields it needs, preventing context blowout entirely. The production risk is unbounded storage consumption and cross-session state leakage: if your execution harness does not enforce strict per-session scratch namespaces with aggressive TTL teardown policies, orphaned JSON payloads will eventually exhaust container disk quotas.

Resources

• modelcontextprotocol.io