Building an agentic stack in Envoy: routing models, tokens, and MCP
Google Cloud expanded Envoy into an open-source agentic networking stack with token rate limits, KV-cache routing, and Model Context Protocol tool guardrails.
By Dillip Chowdary • Nov 18, 2026 • Source: Google Cloud Blog

At a glance
- Product
- Envoy agentic stack
- Contributor
- Google Cloud
- Date
- November 19, 2026
- Standard
- MCP spec 2026-07-28
- Runtime
- Agent Substrate on GKE
- Security
- ext_proc callouts
In this post
Google Cloud detailed how the open-source Envoy proxy has expanded into a dedicated agentic networking stack designed for autonomous artificial intelligence workloads, according to Google Cloud Blog's report. Authors Abhi Maras and Boteng Yao explained that Envoy, which already powers Google Cloud Application Load Balancers and the Agent Substrate dataplane on Google Kubernetes Engine, now provides protocol routing, token-aware rate limiting, and tool governance for multi-turn agent systems.
This technical breakdown examines how autonomous agents break standard Layer 7 proxy models and how Envoy addresses these issues across four architectural pillars. The analysis serves systems architects, platform engineers, and infrastructure leads building production agent deployments that combine Model Context Protocol tools, heterogeneous inference backends, and high-density sandboxes.
01Four bottlenecks across higher network layers
Autonomous agents introduce Layer 8 interaction protocols such as large language model APIs, Anthropic's Model Context Protocol, and Google's Agent2Agent running over JSON-RPC 2.0 or gRPC. They also introduce Layer 9 coordination frameworks such as UCP, AP2, and Google Cloud's Agent Development Kit, where execution graphs are non-linear. Fronting these higher-order layers with standard Layer 7 networking introduces four operational failures.
First, payload blindness occurs because varied model requests share identical paths like /v1/chat/completions while key attributes remain inside bodies. Second, standard request counters create a rate-limiting fallacy, as a single 500,000-token prompt consumes vastly more compute than dozens of smaller requests. Third, stateless load balancers disregard server-side cache state, forcing inference engines to rebuild context on every conversation turn.
Fourth, ungoverned tool execution leaves database queries, code sandboxes, and third-party APIs unmonitored at the wire level. Autonomous agents can trigger prompt injections, data leaks, or runaway execution loops without inline inspection. To resolve these four operational bottlenecks, Envoy implements four core pillars that convert the network edge into an active governance layer.
| Core Pillar | Operational Functionality |
|---|---|
| Native body parser | Payload blindness: Read request bodies, not just headers. Line-rate payload normalization across OpenAI, Anthropic, and Gemini schemas with SSE stream handling. |
| Tokenomics engine | Request-rate fallacy: Count tokens, not just requests. Pre-inference token estimation, tokens per minute (TPM)/tokens per day (TPD) rate limiting. |
| Model & MCP routing | Stateless load balancing: Keep agent sessions on the right servers. Endpoint Picker (EPP) model dispatch, KV-cache prefix affinity, MCP Stateless support. |
| AI security guardrails | Ungoverned tool execution: Policy tool calls. Inline policy, decision point (ext_proc) callouts for guardrails: Model Armor, ShieldGemma, and data loss prevention. |
02Line-rate body parsing and dynamic transcoding
Upstream client applications frequently face vendor lock-in when written against a single provider API schema. Envoy integrates native body parsers that normalize requests across OpenAI, Anthropic, and Gemini schemas directly on the wire. Envoy inspects incoming JSON payloads at line-rate, extracts the target model identifier, and transcodes the request into the native format expected by the downstream provider.
This wire-level abstraction eliminates code changes in upstream client applications. Engineering teams can swap external commercial application programming interfaces for self-hosted open-source models like Gemma or Llama running on vLLM through Envoy configuration updates alone. Client applications continue transmitting their established request payloads without localized translation software.
The parser also handles streaming server-sent events response streams in real time. Envoy extracts token metrics from streaming usage chunks as they pass through the proxy. This architecture preserves ultra-low time-to-first-token latency while collecting exact consumption data for accounting and downstream scheduling.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Token rate limits and runaway loop protection
Envoy replaces standard request counters with a dedicated tokenomics engine that treats tokens as the foundational unit of network capacity. The proxy enforces tokens per minute and tokens per day quotas organized by individual tenant, project, or model identifier. This design protects shared inference infrastructure from lopsided resource consumption caused by long-context queries.
Before routing a request to inference backends, Envoy estimates incoming token volume from payload byte lengths and checks usage against a central quota database. If an incoming prompt exceeds remaining capacity, Envoy rejects the call with an HTTP 429 status code before the payload reaches the GPU cluster. Response tokens are later recorded to maintain exact quota ledgers.
The engine also protects against runaway agent execution loops. Autonomous multi-turn workflows can repeatedly loop through reasoning cycles without completing. Envoy tracks cumulative token usage and message counts per session or agent identifier, throttling execution loops whenever an agent breaches its allocated budget threshold.
04Prefix cache affinity and MCP tool routing
Envoy optimizes inference distribution through KV-cache prefix affinity using the Kubernetes Gateway API Inference Extensions and Endpoint Picker dispatching. When incoming prompts share identical system instructions, Envoy directs them to the exact GPU worker holding that specific cache in memory. This routing choice reduces time-to-first-token latency and prevents duplicated prompt processing on accelerator hardware.
For backend resiliency, Envoy monitors model responses for failures and HTTP 429 status codes. When primary model backends become overloaded or fail, Envoy executes automated retries or reroutes requests to secondary model pools without surfacing errors to the client application.
Tool interactions adhere to the 2026-07-28 Model Context Protocol specification over JSON-RPC 2.0 bodies and HTTP headers. Envoy reads Mcp-Method and Mcp-Name headers to route traffic to tool backends without full body parsing overhead. Downstream backends block tampering by rejecting header-body mismatches with an HTTP 400 error carrying the -32020 HeaderMismatch code.
05Substrate scaling and external security callouts
Running hundreds of thousands of idle enterprise agents inside persistent containers or virtual machines generates high infrastructure costs. Envoy integrates with Google Cloud Agent Substrate on Google Kubernetes Engine, which uses Cloud Hypervisor, microVMs, and gVisor sandboxes capable of suspending and resuming within milliseconds. When traffic targets a suspended agent, Envoy buffers initial packets while triggering an instance resume.
Envoy also establishes perimeter defense as both an ingress and egress gateway for agent environments. The proxy restricts outbound agent connections strictly to verified domains, preventing untrusted code from contacting rogue endpoints. For inline safety, Envoy uses standardized external processing callouts to connect with Model Armor, ShieldGemma, or local classifiers without degrading network throughput.
Frequently asked questions
How does Envoy route prompts to optimize GPU cache usage?
Envoy uses Kubernetes Gateway API Inference Extensions and Endpoint Picker to identify matching prompt prefixes and route requests to the GPU worker holding the active KV-cache.
How does Envoy enforce the Model Context Protocol security rules?
Envoy routes tool traffic using Mcp-Method and Mcp-Name headers, while backends reject header-body mismatches with an HTTP 400 status and -32020 HeaderMismatch error code.
How does Envoy prevent runaway billing loops in autonomous agent workflows?
Envoy tracks cumulative token consumption and message counts per session or agent ID, throttling requests and returning HTTP 429 codes before inference clusters are reached.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement