[{"content":"The AI Protocol Manager (AIPM) is Envoy\u0026rsquo;s filter for LLM API traffic. It parses the request body, runs small AI filters over it, can turn one vendor\u0026rsquo;s API into another\u0026rsquo;s, and reads token usage off the response. It is alpha and moving fast, so this page pins down how it works on Envoy main today (5d3b0dcaf4, October 26, 2026) and collects configs for the setups I keep getting asked about.\nEvery config here runs in an integration test: a real Envoy, mock OpenAI, Anthropic, Gemini and Vertex AI backends that refuse what the real APIs refuse, and an ext_proc server for the routing example. The configs and tests live next to this blog. The snippets below keep only the part that matters; the linked files are complete.\nHow it handles traffic How a request moves through the AI Protocol Manager A client request reaches Envoy. On a route declared as an AI endpoint, the AI Protocol Manager holds the headers, parses the body, and runs AI filters in order: schema validation, request info, and two transcoders that convert the client's API into OpenAI Chat and then into the backend's API. The body is serialized and passed to the router, which sends it to the provider. On the way back, token usage is read from the provider's bytes, and the transcoders convert the response back to the client's API. Client any LLM SDK Envoy AI Protocol Manager only on routes that declare request: 1 hold headers, parse the body (bad JSON → 400) 2 schema_validation 3 request_info → envoy.ai.request_info 4 transcoder: client API → OpenAI Chat 5 transcoder: OpenAI Chat → backend API 6 serialize, release the headers use only the AI filters you need ext_proc, … router response: token usage read from the provider's bytes → envoy.ai.token_usage the transcoders convert the response (JSON or SSE) back to the client's API Provider backend API A declared AI endpoint has its body parsed once and handed to the AI filters. The router and everything after it see a normal request in the backend's API. A route opts in. Per-route config with request: makes a route an AI endpoint and names the API its clients speak. Every other route passes straight through: no buffering, no parsing.\nThe body is parsed before the headers move on. On an AI endpoint, AIPM holds the headers, offloads the body as it arrives, and parses it as JSON. Bad JSON gets a 400 right there. Then the AI filters run, in order, on the parsed document:\nschema_validation checks the body against its API\u0026rsquo;s schema. request_info publishes the model, the streaming flag, the token cap and the message and tool counts as typed metadata (envoy.ai.request_info), and the model as filter state (envoy.ai.model.request). With token_estimation it also estimates input tokens from the body size. transcoder converts between APIs. Everything goes through one common form, OpenAI Chat Completions, so you run two instances: the first turns the client\u0026rsquo;s API into the common form, the second turns that into the backend\u0026rsquo;s API. Filters between them only ever see OpenAI Chat. Then the document is written back into the body and the headers continue down the chain. With reserialize_body: DISABLE, the original bytes go out instead, which is what you want when the filters only read.\nThe response is handled twice. Token usage is read from the provider\u0026rsquo;s bytes, in the backend\u0026rsquo;s API, and published when the stream ends as typed metadata (envoy.ai.token_usage). The counts mean the same thing for every API: input includes cached tokens, output includes reasoning tokens. Separately, the transcoders convert the response, JSON or SSE, back into the client\u0026rsquo;s API.\nThe route decides the backend, and filter state can override it per request. The route\u0026rsquo;s response: names the backend\u0026rsquo;s API. A filter that runs earlier can override it, and more, for a single request:\nenvoy.ai.llm_protocol.request: the client\u0026rsquo;s API. envoy.ai.llm_protocol.response: the backend\u0026rsquo;s API. envoy.ai.binding.response: how the backend carries its API on HTTP. It is either a path pattern such as /v1beta/models/{model}:{generateContent|streamGenerateContent?alt=sse}, or a JSON binding that also edits the body. envoy.ai.binding.request: the same for the client\u0026rsquo;s side. envoy.ai.model.resolved: the model to send instead of the one the client asked for. Anything that writes filter state can set these: set_filter_state, Lua, or ext_proc through headers.\nIt runs on the listener or on the cluster. In http_filters it runs once per request. In a cluster\u0026rsquo;s http_filters, or the router\u0026rsquo;s upstream_http_filters, it runs once per attempt, after Envoy picks the cluster and host, and still sees the request\u0026rsquo;s filter state.\nReading the output. AIPM publishes typed metadata only. An access log reads it with %TYPED_CEL(metadata.typed_filter_metadata['envoy.ai.token_usage'])%, and ext_proc receives it with metadata_options.forwarding_namespaces.typed.\nThe examples Observe native APIs: validate, describe and count tokens, without changing a byte. Any client API, any backend: OpenAI SDK to Claude or Gemini, Claude Code to Gemini or OpenAI, Gemini SDK to Claude, and Vertex AI. Retries Several backends: split traffic, fail over to another API with a composite cluster or a cluster specifier, and route by model. Egress through a forward proxy: any provider, no per-provider config. Routing decisions outside Envoy: ext_proc picks the backend and keeps a token budget. Gaps on main: what didn\u0026rsquo;t work and why. Observe native APIs Clients keep their SDKs and their own keys. Envoy validates and describes each request, counts tokens on each response, and forwards everything as is. Full config: observe.yaml.\nObserving native LLM APIs The client calls its provider through Envoy with its own SDK and key. The AI Protocol Manager validates the body, describes the request, and forwards the client's exact bytes. On the way back it reads token usage from the provider's response without changing it. Both records are published as typed metadata for the access log or ext_proc. Client own SDK, own key AI Protocol Manager schema_validation: a bad body gets its 400 here request_info: model, stream, token cap, counts token_usage: read off the provider's response Provider native API request response same bytes untouched envoy.ai.request_info · envoy.ai.token_usage → access log, ext_proc Observing only reads. The provider gets the client's bytes and key, and Envoy keeps a description of each request and its token count. virtual_hosts: - name: providers domains: [\u0026#34;*\u0026#34;] request_headers_to_remove: [accept-encoding] routes: - match: {path: /v1/messages} route: {cluster: anthropic, auto_host_rewrite: true} typed_per_filter_config: envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {llm_protocol: ANTHROPIC_MESSAGES} # The same for /v1/chat/completions, /v1/responses and /v1beta/models/. http_filters: - name: envoy.filters.http.ai_protocol_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager request_handling: {reserialize_body: DISABLE} response_handling: {token_usage: {}} filters: - name: envoy.http.ai_filters.schema_validation typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.schema_validation.v3.SchemaValidation - name: envoy.http.ai_filters.request_info typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.request_info.v3.RequestInfo - name: envoy.filters.http.router typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router Why each part is there:\nrequest: {llm_protocol: ...} makes the route an AI endpoint, and picks the schema to validate against and the usage format to read. reserialize_body: DISABLE: these filters only read, so the provider gets the client\u0026rsquo;s exact bytes. token_usage: {} publishes one usage record per response. request_headers_to_remove: [accept-encoding]: usage can\u0026rsquo;t be read from a gzip body. Envoy then asks the provider for plain bytes. A request through it logs:\n{\u0026#34;model\u0026#34;:\u0026#34;gpt-4o-mini\u0026#34;,\u0026#34;status\u0026#34;:200, \u0026#34;request_info\u0026#34;:{\u0026#34;input_llm_protocol\u0026#34;:\u0026#34;OPENAI_CHAT_COMPLETIONS\u0026#34;,\u0026#34;message_count\u0026#34;:1,\u0026#34;model\u0026#34;:\u0026#34;gpt-4o-mini\u0026#34;}, \u0026#34;token_usage\u0026#34;:{\u0026#34;extraction_status\u0026#34;:\u0026#34;COMPLETE\u0026#34;,\u0026#34;llm_protocol\u0026#34;:\u0026#34;OPENAI_CHAT_COMPLETIONS\u0026#34;,\u0026#34;model\u0026#34;:\u0026#34;gpt-4o-mini\u0026#34;, \u0026#34;input_tokens\u0026#34;:\u0026#34;11\u0026#34;,\u0026#34;output_tokens\u0026#34;:\u0026#34;7\u0026#34;,\u0026#34;total_tokens\u0026#34;:\u0026#34;18\u0026#34;,\u0026#34;provider_total_tokens\u0026#34;:\u0026#34;18\u0026#34;}} Things the tests ran into:\nOpenAI streams carry usage only when the client sets stream_options.include_usage: true. Without it there is no record, and token_usage_missing counts it. The Responses API has usage and request info, but no schema yet, so validation skips it. Gemini names the model in the path, and request info reads it from there. Malformed JSON, a body that breaks its schema, and an Anthropic request without max_tokens all get a 400 without reaching the provider. Any client API, any backend One filter chain serves every route. The base URL picks the backend and the path the client\u0026rsquo;s API, so each pairing is one route. Full config: gateway.yaml.\nBase URL Client Backend Needs, besides the two APIs /claude/v1 OpenAI SDK Claude backend path /gemini/v1 OpenAI SDK Gemini nothing /gemini Claude Code Gemini model /openai Claude Code OpenAI-compatible backend path, model /claude Gemini SDK Claude client path, backend path, model /vertex/claude/v1 OpenAI SDK Claude on Vertex AI backend path with body edits /vertex/gemini/v1 OpenAI SDK Gemini on Vertex AI backend path /v1 Claude Code on Vertex AI Gemini on Vertex AI backend path, model \u0026ldquo;Model\u0026rdquo; is envoy.ai.model.resolved, for clients that ask for a model the backend doesn\u0026rsquo;t have. \u0026ldquo;Path\u0026rdquo; is a binding, explained below.\nhttp_filters: - name: envoy.filters.http.set_filter_state # per-route model overrides typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config - name: envoy.filters.http.ai_protocol_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager request_handling: {} response_handling: {token_usage: {}} filters: - name: envoy.http.ai_filters.request_info typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.request_info.v3.RequestInfo - name: envoy.http.ai_filters.transcoder # client\u0026#39;s API \u0026lt;-\u0026gt; OpenAI Chat typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder request_handling: TO_IR response_handling: FROM_IR - name: envoy.http.ai_filters.transcoder # OpenAI Chat \u0026lt;-\u0026gt; backend\u0026#39;s API typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder request_handling: FROM_IR response_handling: TO_IR - name: envoy.filters.http.router typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router Credentials belong to the gateway. The virtual host strips whatever the client sent (authorization, x-api-key, x-goog-api-key), and each route adds the backend\u0026rsquo;s key. most_specific_header_mutations_wins: true on the route config makes the route\u0026rsquo;s additions run after the virtual host\u0026rsquo;s removals. The other order would strip the key the route just added.\nOpenAI SDK to Claude base_url=\u0026quot;http://localhost:10000/claude/v1\u0026quot;:\nOne request from the OpenAI SDK to Claude The client sends an OpenAI Chat request to /claude/v1/chat/completions with its own key. Envoy drops the client's key and adds Claude's, the first transcoder keeps OpenAI Chat as is, the second writes Anthropic Messages, the binding sets the path to /v1/messages, and the host is rewritten to Claude's. Claude receives a Messages request with a top-level system field and max_tokens. Claude's answer, with its usage, is read for token usage and converted back into an OpenAI chat completion. request Client sends\u0026#160;OpenAI Chat POST /claude/v1/chat/completions authorization: Bearer ‹client› {\"model\": \"claude-sonnet-4-5\", \"messages\": [system, user]} Envoy virtual host: drop the client's key route: add Claude's key and version transcoder 1: OpenAI Chat as is transcoder 2: → Anthropic Messages binding: path → /v1/messages auto_host_rewrite: Claude's host Claude receives\u0026#160;Messages POST /v1/messages host: api.anthropic.com x-api-key: ‹gateway's› anthropic-version: 2023-06-01 {\"model\": \"claude-sonnet-4-5\", \"system\": \"Be brief.\", \"messages\": [user], \"max_tokens\": …} response Client gets\u0026#160;OpenAI Chat {\"object\": \"chat.completion\", \"choices\": [{\"message\": …}], \"usage\": {…}} token_usage: 12 in, 6 out transcoder 2: → OpenAI Chat transcoder 1: OpenAI Chat as is JSON and SSE alike Claude answers\u0026#160;Messages {\"content\": [text block], \"usage\": {\"input_tokens\": 12, \"output_tokens\": 6}} The client and Claude each see only their own API. The key, path, host and body shape change inside Envoy. - match: {path: /claude/v1/chat/completions} route: {cluster: anthropic, auto_host_rewrite: true} request_headers_to_add: - header: {key: x-api-key, value: \u0026#34;%ENVIRONMENT(ANTHROPIC_API_KEY)%\u0026#34;} - header: {key: anthropic-version, value: \u0026#34;2023-06-01\u0026#34;} typed_per_filter_config: envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {llm_protocol: OPENAI_CHAT_COMPLETIONS} response: {llm_protocol: ANTHROPIC_MESSAGES, binding: {uri_pattern: /v1/messages}} response.llm_protocol tells the second transcoder to write Anthropic Messages, and tells token usage to read Anthropic\u0026rsquo;s format. binding: {uri_pattern: /v1/messages}: the transcoder rewrites the body but leaves the path alone unless a binding names one. Without it, Claude gets a Messages body at /claude/v1/chat/completions and answers 404. Only Gemini has a built-in path, because its path carries the model. OpenAI SDK to Gemini response: {llm_protocol: GEMINI_GENERATE_CONTENT} This needs no binding: Gemini\u0026rsquo;s built-in one renders /v1beta/models/gemini-2.5-flash:generateContent, or :streamGenerateContent?alt=sse for stream: true, and removes model and stream from the body. Its key goes in x-goog-api-key.\nClaude Code to Gemini, or to any OpenAI-compatible backend ANTHROPIC_BASE_URL=http://localhost:10000/gemini:\n- match: {path: /gemini/v1/messages} route: {cluster: gemini, auto_host_rewrite: true} request_headers_to_add: - header: {key: x-goog-api-key, value: \u0026#34;%ENVIRONMENT(GEMINI_API_KEY)%\u0026#34;} typed_per_filter_config: envoy.filters.http.set_filter_state: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config on_request_headers: - object_key: envoy.ai.model.resolved format_string: {text_format_source: {inline_string: gemini-2.5-flash}} envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {llm_protocol: ANTHROPIC_MESSAGES} response: {llm_protocol: GEMINI_GENERATE_CONTENT} envoy.ai.model.resolved: Claude Code asks for Claude models. This replaces the model wherever the backend carries it (here the path), and the response still names the model the client asked for. For an OpenAI-compatible backend (OpenAI, vLLM, …), the route at /openai/v1/messages sends gpt-4o-mini with response: {llm_protocol: OPENAI_CHAT_COMPLETIONS, binding: {uri_pattern: /v1/chat/completions}}. Claude Code\u0026rsquo;s tools come out as OpenAI functions, and its SSE comes back as Anthropic events.\nGemini SDK to Claude http_options={\u0026quot;base_url\u0026quot;: \u0026quot;http://localhost:10000/claude\u0026quot;}, so the path is /claude/v1beta/models/gemini-2.5-flash:generateContent:\nrequest: llm_protocol: GEMINI_GENERATE_CONTENT binding: {uri_pattern: \u0026#34;/claude/v1beta/models/{model}:{generateContent|streamGenerateContent?alt=sse}\u0026#34;} response: {llm_protocol: ANTHROPIC_MESSAGES, binding: {uri_pattern: /v1/messages}} request.binding: Gemini puts the model and the streaming mode in the path. Envoy knows Gemini\u0026rsquo;s own path and Vertex AI\u0026rsquo;s, but not your prefix, so this pattern says where {model} sits. {generateContent|streamGenerateContent?alt=sse} reads stream. maxOutputTokens becomes max_tokens. envoy.ai.model.resolved: claude-sonnet-4-5, set the same way as for Claude Code below: the SDK asks for a Gemini model. Vertex AI OpenAI clients to Claude on Vertex AI:\n- match: {path: /vertex/claude/v1/chat/completions} route: {cluster: vertex, auto_host_rewrite: true} request_headers_to_add: - header: {key: authorization, value: \u0026#34;Bearer %ENVIRONMENT(VERTEX_ACCESS_TOKEN)%\u0026#34;} typed_per_filter_config: envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {llm_protocol: OPENAI_CHAT_COMPLETIONS} response: llm_protocol: ANTHROPIC_MESSAGES binding: uri_pattern: \u0026#34;/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/{model}:{rawPredict|streamRawPredict}\u0026#34; mutations: - {body_field: model} - {body_field: anthropic_version, value: vertex-2023-10-16, keep_existing: true} How a binding renders a Vertex AI request The second transcoder writes an Anthropic Messages body with model and stream. The binding's pattern takes the model from the body into the path, and stream picks rawPredict or streamRawPredict. The first mutation removes model from the body, and the second adds anthropic_version. Vertex AI receives a POST to the streamRawPredict path for the model, with anthropic_version, stream, messages and max_tokens in the body. 1 transcoder 2 writes Anthropic Messages {\"model\": \"claude-sonnet-4-5@20250929\",\u0026#160;\"stream\": true, \"messages\": […], \"max_tokens\": …} 2 the binding renders the path and edits the body uri_pattern: /v1/projects/…/publishers/anthropic/models/{model}:{rawPredict|streamRawPredict} mutations:\u0026#160;{body_field: model},\u0026#160;{body_field: anthropic_version, value: vertex-2023-10-16} {model} takes the body's model, and the first mutation removes it from the body. stream picks the method and stays. 3 Vertex AI receives POST /v1/projects/…/publishers/anthropic/models/claude-sonnet-4-5@20250929:streamRawPredict {\"anthropic_version\": \"vertex-2023-10-16\",\u0026#160;\"stream\": true, \"messages\": […], \"max_tokens\": …} A binding maps between the body and the path. On the client side it works the other way: it reads the model and the method out of the path, as in the Claude Code on Vertex AI route below. Claude on Vertex AI speaks Anthropic Messages, but it takes the model in the path and needs anthropic_version in the body. The two mutations remove model and add the version. The method follows stream. Gemini on Vertex AI needs only the pattern (.../publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}). Gemini\u0026rsquo;s own edits still apply. The token comes from gcloud auth print-access-token here. In production, let the gcp_authn filter fetch it. Claude Code on Vertex AI (CLAUDE_CODE_USE_VERTEX=1, ANTHROPIC_VERTEX_BASE_URL=http://localhost:10000/v1) to Gemini, in the project and region the client named:\n- match: {prefix: /v1/projects/} route: {cluster: vertex, auto_host_rewrite: true} typed_per_filter_config: envoy.filters.http.set_filter_state: # model.resolved: gemini-2.5-flash ... envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {llm_protocol: ANTHROPIC_MESSAGES} response: llm_protocol: GEMINI_GENERATE_CONTENT binding: {uri_pattern: \u0026#34;/{version}/projects/{project}/locations/{location}/publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}\u0026#34;} No request binding is needed: Envoy knows rawPredict and streamRawPredict. It reads the model, the streaming mode, {version}, {project} and {location} from the client\u0026rsquo;s path, and takes anthropic_version out of the body. The backend pattern renders {model} from the body (after model.resolved), and every other variable from what it read. Access logs can show those values with %FILTER_STATE(envoy.ai.binding.read:FIELD:project)%. I left schema validation out of this chain. It doesn\u0026rsquo;t read bindings yet, so it rejects a Vertex AI body whose model is in the path. The second transcoder validates against the backend\u0026rsquo;s schema before sending anyway.\nRetries route: cluster: anthropic retry_policy: {retry_on: 5xx, num_retries: 1} A retry with transcoding on the listener The AI Protocol Manager on the listener converts the OpenAI Chat request into Anthropic Messages once, before the router. The router buffers the body. Attempt 1 to the anthropic cluster gets a 503. Attempt 2 sends the same bytes, gets a 200, and is the attempt that reports token usage. Client OpenAI SDK AIPM on the listener OpenAI Chat → Messages once, before the router router buffers the body for a retry attempt 1 → anthropic POST /v1/messages ✗ 503 attempt 2 → anthropic the same bytes ✓ 200, token usage from here Transcoded once, sent twice. Only the attempt that answered reports token usage. With AIPM on the listener, transcoding happens once, before the router. The router buffers the transcoded body and sends the same bytes on the retry, and token usage comes from the attempt that answered. There is nothing AI-specific to configure. Retrying on a backend with a different API needs transcoding on each attempt, which is the next section.\nSeveral backends: split, failover, by model Full config: multi-backend.yaml.\nTranscoding split between the listener and each attempt The AI Protocol Manager on the listener converts the client's API to OpenAI Chat once. The router picks a cluster per attempt: a weighted split, a composite cluster, or a priority group. On each attempt, an AI Protocol Manager on the upstream side converts OpenAI Chat into the API of the backend it reached. In the example, attempt 1 goes to Claude on Vertex AI and fails with 503, and attempt 2 goes to Gemini on Vertex AI and succeeds. Client any API AIPM on the listener client API → OpenAI Chat once per request router split, composite, priority group, retry attempt 1 → vertex_claude AIPM: OpenAI Chat → Claude .../anthropic/models/…:rawPredict ✗ 503 attempt 2 → vertex_gemini AIPM: OpenAI Chat → Gemini .../google/models/…:generateContent ✓ 200, back to the client's API Transcode the client's side once, and the backend's side on every attempt. A retry that lands on another API is then converted for that API. Split the transcoding in two. The AIPM on the listener converts the client\u0026rsquo;s API into OpenAI Chat. A second AIPM runs on each attempt, after Envoy has picked the cluster and host, and converts OpenAI Chat into whatever that backend speaks. Each endpoint says what it speaks in its metadata:\n- endpoint: {address: {socket_address: {address: us-east5-aiplatform.googleapis.com, port_value: 443}}} metadata: filter_metadata: envoy.ai: api: GEMINI_GENERATE_CONTENT model: gemini-2.5-flash binding: \u0026#34;/v1/projects/my-project/locations/us-east5/publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}\u0026#34; The router\u0026rsquo;s upstream filters turn that metadata into the AIPM\u0026rsquo;s instructions on every attempt:\nupstream_http_filters: # every cluster without filters of its own - name: envoy.filters.http.header_mutation typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.header_mutation.v3.HeaderMutation mutations: request_mutations: - append: {header: {key: x-ai-api, value: \u0026#34;%UPSTREAM_METADATA(envoy.ai:api)%\u0026#34;}, append_action: OVERWRITE_IF_EXISTS_OR_ADD} - append: {header: {key: x-ai-binding, value: \u0026#34;%UPSTREAM_METADATA(envoy.ai:binding)%\u0026#34;}, append_action: OVERWRITE_IF_EXISTS_OR_ADD} - append: {header: {key: x-ai-model, value: \u0026#34;%UPSTREAM_METADATA(envoy.ai:model)%\u0026#34;}, append_action: OVERWRITE_IF_EXISTS_OR_ADD} - name: envoy.filters.http.lua typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua default_source_code: inline_string: | function envoy_on_request(handle) local state = handle:streamInfo():filterState() for header, key in pairs({[\u0026#34;x-ai-api\u0026#34;] = \u0026#34;envoy.ai.llm_protocol.response\u0026#34;, [\u0026#34;x-ai-binding\u0026#34;] = \u0026#34;envoy.ai.binding.response\u0026#34;, [\u0026#34;x-ai-model\u0026#34;] = \u0026#34;envoy.ai.model.resolved\u0026#34;}) do state:set(key, key, handle:headers():get(header)) handle:headers():remove(header) end end - name: envoy.filters.http.ai_protocol_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager request_handling: {} response_handling: {token_usage: {}} filters: - name: envoy.http.ai_filters.transcoder typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder request_handling: FROM_IR response_handling: TO_IR - name: envoy.filters.http.upstream_codec typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec Three values relayed from the picked host to the AI Protocol Manager On each attempt, the picked host's envoy.ai metadata holds the backend's API, its binding and its model. header_mutation copies them into the x-ai-api, x-ai-binding and x-ai-model headers using UPSTREAM_METADATA. Lua moves each header into filter state: envoy.ai.llm_protocol.response, envoy.ai.binding.response and envoy.ai.model.resolved. The AI Protocol Manager then writes that API, renders that path and sends that model. the backend's API how it carries it on HTTP the model to send picked host envoy.ai metadata api: GEMINI_GENERATE_CONTENT binding: \"/v1/projects/…\" model: gemini-2.5-flash header_mutation %UPSTREAM_METADATA% x-ai-api x-ai-binding x-ai-model Lua headers → filter state envoy.ai.llm_protocol.response envoy.ai.binding.response envoy.ai.model.resolved AI Protocol Manager on each attempt writes this API renders this path sends this model On every attempt, three values relay from the host Envoy picked to the AI Protocol Manager. header_mutation can read the host but not set filter state, and Lua the other way round, so they hand off through headers that Lua then removes. Endpoint metadata, not cluster config: a composite cluster runs its own upstream filters for every sub-cluster, so the only thing that identifies the backend on an attempt is the host that was picked. %UPSTREAM_METADATA% reads that host\u0026rsquo;s metadata. header_mutation, then Lua: set_filter_state can\u0026rsquo;t run on a cluster yet, and Lua can\u0026rsquo;t read the host\u0026rsquo;s metadata. header_mutation can format %UPSTREAM_METADATA%, and Lua can set filter state, so they pass the values along in three headers. Every endpoint sets all three values. The filter state survives across attempts, so a value one endpoint left out would be the previous attempt\u0026rsquo;s. Token usage on this AIPM reads the backend that answered. Only the winning attempt publishes. Then the routes. All of them declare request: {llm_protocol: OPENAI_CHAT_COMPLETIONS} (or the client\u0026rsquo;s API) for both AIPMs.\nSplit half the traffic between Claude and Gemini:\nroute: weighted_clusters: clusters: [{name: vertex_claude, weight: 50}, {name: vertex_gemini, weight: 50}] auto_host_rewrite: true Fail over to Gemini when Claude fails, using a composite cluster (the first attempt uses the first cluster, the retry uses the next):\nroute: {cluster: vertex_failover, auto_host_rewrite: true, retry_policy: {retry_on: 5xx, num_retries: 1}} # cluster - name: vertex_failover lb_policy: CLUSTER_PROVIDED cluster_type: name: envoy.clusters.composite typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.clusters.composite.v3.ClusterConfig clusters: [{name: vertex_claude}, {name: vertex_gemini}] In the test, attempt 1 sends rawPredict with anthropic_version and gets a 503. Attempt 2 sends generateContent with contents. The OpenAI client gets a normal chat completion, and Claude Code, on its own route, gets Anthropic SSE.\nFail over across providers with their own keys, Anthropic first, then Gemini, using the priority group cluster specifier:\nroute: inline_cluster_specifier_plugin: extension: name: envoy.router.cluster_specifier_plugin.priority_group typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.router.cluster_specifiers.priority_group.v3.PriorityGroupClusterSpecifier priority_groups: - {name: primary, clusters: [{cluster_name: anthropic, weight: 1}]} - {name: fallback, clusters: [{cluster_name: gemini, weight: 1}]} auto_host_rewrite: true retry_policy: {retry_on: 5xx, num_retries: 1, refresh_cluster_on_retry: true} refresh_cluster_on_retry makes the router ask the specifier again on the retry, and the attempt then runs the new cluster\u0026rsquo;s filters. Each cluster brings its own key. A cluster\u0026rsquo;s own http_filters replace the router\u0026rsquo;s, so it lists a header_mutation that adds its key, followed by the same three filters. It also removes the other provider\u0026rsquo;s key: upstream filters edit the request in place, so a retry starts with whatever the previous attempt added. The test checks that the Gemini attempt carries no x-api-key. Route by model: routes can match the model request_info reads from the body:\n- match: path: /v1/chat/completions filter_state: [{key: envoy.ai.model.request, string_match: {prefix: gemini-}}] route: {cluster: vertex_gemini, auto_host_rewrite: true} # ... claude- to vertex_claude, then: - match: {path: /v1/chat/completions} direct_response: {status: 404, body: {inline_string: \u0026#34;no backend serves this model\u0026#34;}} http_filters: - name: envoy.filters.http.ai_protocol_manager # request_info, client-side transcoder ... - name: envoy.filters.http.set_filter_state typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config clear_route_cache: true Routing by the model in the body When the headers arrive the model is unknown, so the catch-all route matches; it declares request, so the body is parsed. The AI Protocol Manager's request_info reads the model into envoy.ai.model.request. set_filter_state with clear_route_cache drops the picked route, and the routes are matched again: a gemini model goes to vertex_gemini, a claude model to vertex_claude, and anything else gets a 404 without calling a provider. 1 headers arrive no model yet, so the catch-all matches; it declares\u0026#160;request: 2 AI Protocol Manager request_info reads the model into envoy.ai.model.request 3 set_filter_state clear_route_cache: true drops the route Envoy picked 4 routes match again gemini-* → vertex_gemini Gemini on Vertex AI claude-* → vertex_claude Claude on Vertex AI anything else → 404 no provider is called Envoy picks a route before the body arrives. Clearing the route cache after AIPM has read the model makes it pick again, and this time the model can decide. The catch-all route must also declare request:: Envoy picks a route before the body is read, and only an AI endpoint gets its body parsed. clear_route_cache: true, after AIPM, makes Envoy pick the route again once the model is known. A model no backend serves gets the 404 without calling any provider. Egress through a forward proxy One route, no per-provider config. The request\u0026rsquo;s Host picks the destination: clients set HTTP_PROXY=http://localhost:10000 and use an http:// base URL, and Envoy originates TLS. Full config: egress.yaml.\nEgress through a dynamic forward proxy A client with HTTP_PROXY set sends plain HTTP to Envoy. The AI Protocol Manager on the listener detects the API from the path, a header or the body, then validates and describes the request. The dynamic forward proxy resolves the request's Host. The AI Protocol Manager on the cluster reads token usage on each attempt, detecting the API from the response. The cluster's TLS takes its SNI and SAN check from the Host, so one route reaches api.openai.com, api.anthropic.com or generativelanguage.googleapis.com. Client HTTP_PROXY set, http:// base URL Envoy AIPM detects the API from path, header or body, validates, describes DFP resolves the request's Host AIPM on the cluster token usage on each attempt, API from the response cluster TLS: auto_sni and auto_san_validation follow the Host the Host api.openai.com api.anthropic.com generativelanguage .googleapis.com One route for every provider. The Host decides where the request goes, the listener's AIPM works out which API it is, and the cluster's AIPM reads usage from whichever provider answered. routes: - match: {prefix: /} route: {cluster: egress} request_headers_to_remove: [accept-encoding] typed_per_filter_config: envoy.filters.http.ai_protocol_manager: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute request: {} # listener: AIPM (schema_validation, request_info, DISABLE) -\u0026gt; dynamic_forward_proxy -\u0026gt; router # cluster `egress` (dynamic forward proxy), upstream filters: http_filters: - name: envoy.filters.http.ai_protocol_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager response_handling: {token_usage: {}} - name: envoy.filters.http.upstream_codec typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec request: {} declares an AI endpoint without naming the API. Schema validation detects it from the path (/chat/completions, /v1/messages, :generateContent), then the anthropic-version header, then the body, and validates against it. Request info reads with the API it found. Every body on the route is parsed strictly, so keep this route for LLM traffic. Token usage on the cluster detects the response\u0026rsquo;s API from its shape. It runs on each attempt, and only the attempt that answered publishes. auto_sni and auto_san_validation on the cluster use the Host, so TLS names and checks whichever provider the request went to. Routing decisions outside Envoy A decision point (PDP) behind ext_proc picks the backend for each request and charges each tenant for the tokens it used. Full config: pdp.yaml, and the PDP is about 150 lines of Go.\nModel routing with an external decision point The request flows from the client through the AI Protocol Manager on the listener, which describes it, to ext_proc, where the PDP decides the backend. set_filter_state turns the PDP's headers into filter state, and the router with the dynamic forward proxy sends the request to the cluster, where a second AI Protocol Manager transcodes it and reads token usage from the provider's response. The usage record goes back to ext_proc on trailers, and the PDP charges the tenant. Client model: smart AIPM describe ext_proc PDP decides headers → filter state router + DFP AIPM on the cluster Claude or Gemini sees the model and a token estimate, not the prompt token usage on the response trailers → the PDP charges the tenant over budget: the next \"smart\" request goes to Gemini The PDP decides from a description of the request and settles from the usage Envoy reports. It never handles the prompt or the response body. http_filters: # schema_validation, request_info with token_estimation, reserialize_body: DISABLE - name: envoy.filters.http.ai_protocol_manager ... - name: envoy.filters.http.ext_proc typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor grpc_service: {envoy_grpc: {cluster_name: pdp}} processing_mode: request_header_mode: SEND response_header_mode: SKIP response_trailer_mode: SEND metadata_options: forwarding_namespaces: typed: [envoy.ai.request_info, envoy.ai.token_usage] - name: envoy.filters.http.set_filter_state typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config on_request_headers: - object_key: envoy.ai.llm_protocol.response format_string: {text_format_source: {inline_string: \u0026#34;%REQ(x-ai-api)%\u0026#34;}} - object_key: envoy.ai.binding.response format_string: {text_format_source: {inline_string: \u0026#34;%REQ(x-ai-binding)%\u0026#34;}} - object_key: envoy.ai.model.resolved format_string: {text_format_source: {inline_string: \u0026#34;%REQ(x-ai-model)%\u0026#34;}} # the route sets host_rewrite_header: x-ai-host - name: envoy.filters.http.dynamic_forward_proxy ... - name: envoy.filters.http.router ... # cluster `llm` (dynamic forward proxy), upstream filters: - name: envoy.filters.http.ai_protocol_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager request_handling: {} response_handling: token_usage: {usage_signal: SYNTHESIZE_TRAILERS} filters: - name: envoy.http.ai_filters.transcoder typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder request_handling: FROM_IR response_handling: TO_IR - name: envoy.filters.http.upstream_codec ... ext_proc comes after the first AIPM. request_info is published before the headers move on, so the PDP gets the model and the token estimate with the request headers. The body is never sent to it. The PDP answers with headers, and set_filter_state turns them into instructions, since ext_proc can\u0026rsquo;t write filter state. The PDP overwrites all four x-ai-* headers on every request, and the route strips them before the request goes upstream. Otherwise a client could send x-ai-host and pick the host itself; the test sends one and checks it\u0026rsquo;s ignored. One transcoder instance on the cluster is enough: these clients already speak OpenAI Chat, the common form. usage_signal: SYNTHESIZE_TRAILERS with response_trailer_mode: SEND delivers the usage record at the end of the response, without streaming the body to the PDP. In the test, the PDP has a 10-token budget. The first smart request goes to Claude and costs 18 tokens. The next smart request goes to Gemini, and an unknown model gets a 400 from the PDP before any provider is called.\nGaps on main These are the things that didn\u0026rsquo;t work as I first wrote them. All of them are on main as of this writing.\nSchema validation doesn\u0026rsquo;t read bindings. A Vertex AI body names its model in the path, and validation rejects it with missing required field: model. Leave validation out of chains that take Vertex AI clients. set_filter_state doesn\u0026rsquo;t run on a cluster. It isn\u0026rsquo;t registered as an upstream filter, so per-attempt instructions go through header_mutation and Lua. auto_sni doesn\u0026rsquo;t follow auto_host_rewrite. The SNI is taken from the Host before the rewrite applies, so the TLS check fails. Set sni and the SAN check on each cluster, as the configs above do. Upstream filters edit the request in place, and a retry starts from the previous attempt\u0026rsquo;s edits. That\u0026rsquo;s why clusters remove other providers\u0026rsquo; keys. It also breaks the client-side transcoder when it runs on the cluster: on a retry it reads the path the first attempt rewrote, and a Gemini SDK client gets a 400 (missing required field: model) instead of the retry. Keep the client side on the listener. The Responses API gets usage and request info, but no schema or transcoding yet. The AI Protocol Manager is alpha, and all of this will move. If one of these configs stops working on a newer Envoy, the tests will tell you which part changed. Issues and ideas are welcome on GitHub.\n","permalink":"https://botengyao.github.io/posts/envoy-ai-protocol-manager-examples/","summary":"How Envoy\u0026rsquo;s AI Protocol Manager handles LLM traffic today, and a set of tested configs: observing native APIs, translating between OpenAI, Claude and Gemini (Vertex AI included), retries and failover across APIs, a forward-proxy egress, and model routing with ext_proc.","title":"Envoy AI Protocol Manager by Example"},{"content":"AI agents use the Model Context Protocol (MCP) to discover and call tools. The revision published on 2026-07-28 changes MCP in a way that matters to anyone running it at scale: MCP is now stateless. There\u0026rsquo;s no connection handshake and no session to keep track of. Each request carries everything a server needs to handle it.\nThat makes MCP traffic look a lot like ordinary HTTP traffic, which is exactly what Envoy is built for. Envoy 1.40 supports the new revision and lets you:\nroute MCP requests by tool name, using standard Envoy route configuration; stop malformed or spoofed requests at the proxy, before they reach your servers; control who can call which tool, with Envoy\u0026rsquo;s RBAC and external authorization filters; turn an existing REST API into an MCP server, with config alone and no new code. What changed in MCP Before 2026-07-28, a client opened a session with an initialize handshake. The server returned a session ID, and every later request had to reach the server that held that session. Behind a load balancer, that meant sticky sessions.\nThe new revision removes that step. Every request now includes the protocol version and the client\u0026rsquo;s capabilities. Routing information is also copied into HTTP headers, so a proxy can read it without parsing the JSON body. A tool call now looks like this:\nPOST /mcp HTTP/1.1 Content-Type: application/json Accept: application/json, text/event-stream MCP-Protocol-Version: 2026-07-28 Mcp-Method: tools/call Mcp-Name: get_weather {\u0026#34;jsonrpc\u0026#34;:\u0026#34;2.0\u0026#34;,\u0026#34;id\u0026#34;:3,\u0026#34;method\u0026#34;:\u0026#34;tools/call\u0026#34;, \u0026#34;params\u0026#34;:{\u0026#34;name\u0026#34;:\u0026#34;get_weather\u0026#34;,\u0026#34;arguments\u0026#34;:{\u0026#34;city\u0026#34;:\u0026#34;Paris\u0026#34;}, \u0026#34;_meta\u0026#34;:{\u0026#34;io.modelcontextprotocol/protocolVersion\u0026#34;:\u0026#34;2026-07-28\u0026#34;, \u0026#34;io.modelcontextprotocol/clientCapabilities\u0026#34;:{}}}} Three headers carry the important routing facts:\nMCP-Protocol-Version: the protocol version this request uses; Mcp-Method: what the client wants to do (tools/call); Mcp-Name: which tool it wants (get_weather). Why this is a good fit for Envoy Stateless requests can go to any replica, so you can scale MCP servers the way you scale any other HTTP service. Envoy can spread the load, retry, and fail over without worrying about sessions.\nEnvoy 1.40 also understands MCP itself, through two extensions. Each one works on its own, so you pick the one that fits your backend:\nThe MCP filter (envoy.filters.http.mcp) goes in front of MCP servers. It reads each MCP request, checks that it is well formed, and exposes the method and tool name to the rest of Envoy for routing, access control and logging. The MCP REST bridge (envoy.filters.http.mcp_json_rest_bridge) goes in front of REST APIs and presents them as an MCP server. You can run each one in its own filter chain, for example one listener for your MCP servers and another for your REST APIs, or combine them in one chain. Here are the two paths:\nTwo ways to put Envoy 1.40 in front of MCP traffic An MCP client sends requests to Envoy. In front of an MCP server, the MCP filter checks each request and routes it by tool name. In front of a REST API, the REST bridge answers discovery calls itself and turns tool calls into REST calls. Each works on its own, and each rejects bad requests with an MCP error before they reach the backend. MCP client POST /mcp Envoy 1.40 MCP filter in front of an MCP server ✓version is supported ✓MCP headers are present ✓headers match the body ✓capabilities are declared → routes by tool name ✗ rejects bad requests with an MCP error or REST bridge in front of a REST API answers server/discover answers tools/list turns tools/call into GET /v1/weather/Paris ✓headers match the body ✗ rejects bad requests with an MCP error MCP server speaks MCP REST API no MCP code Put the MCP filter in front of MCP servers, the REST bridge in front of REST APIs, or use both. Each works on its own, and each stops bad requests before they reach your backend. Route requests by tool name Each tool can have its own backend. For example, get_weather can go to a weather service and every other tool to a default MCP server. The MCP filter writes the method and tool name into Envoy\u0026rsquo;s request metadata, and a normal route can match on that metadata:\nhttp_filters: - name: envoy.filters.http.mcp typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.mcp.v3.Mcp attribute_source: HEADERS clear_route_cache: true - name: envoy.filters.http.router typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router route_config: virtual_hosts: - name: tools domains: [\u0026#34;*\u0026#34;] routes: - match: prefix: /mcp dynamic_metadata: - filter: envoy.filters.http.mcp path: [{key: params}, {key: name}] value: {string_match: {exact: get_weather}} route: {cluster: weather} - match: {prefix: /mcp} route: {cluster: default_mcp} Two settings do the work:\nattribute_source: HEADERS tells the filter to read the tool name from the Mcp-Name header. Envoy can then route without waiting for the request body, so large requests are routed just as fast as small ones. clear_route_cache: true makes Envoy pick the route again after the filter has added the tool name. The filter also handles the spec\u0026rsquo;s encoding rules. A tool name that isn\u0026rsquo;t plain ASCII is sent in Base64 form (=?base64?…?=), and the filter decodes it before routing. Both of these requests reach the weather backend:\nmcp-name: get_weather -\u0026gt; routed to weather mcp-name: =?base64?Z2V0X3dlYXRoZXI=?= -\u0026gt; routed to weather HEADERS is the fastest mode, and it relies on your MCP servers to confirm that the headers match the body, which every 2026-07-28 server must do. If the route is also a security decision, for example when different tools have different access rules, set attribute_source to BODY (the default) or VERIFY. Envoy then reads the tool name from the body itself.\nTip: Match on the filter\u0026rsquo;s metadata, as in the config above, rather than on the raw mcp-name header. The metadata holds the decoded name, and in BODY or VERIFY mode it always reflects what is in the request body.\nCatch bad requests at the proxy Because routing information now appears twice, once in the headers and once in the body, a client could make the two disagree. For example, it could name an allowed tool in the header and a different tool in the body. The MCP spec requires such requests to be rejected, and Envoy does it at the proxy, so they never reach your servers.\nSet traffic_mode: REJECT_NO_MCP on the MCP filter, and Envoy checks every 2026-07-28 request:\n$ curl -si localhost:10000/mcp ... -H \u0026#39;mcp-protocol-version: 2026-07-28\u0026#39; \\ -H \u0026#39;mcp-method: tools/call\u0026#39; -H \u0026#39;mcp-name: get_weather\u0026#39; \\ -d \u0026#39;{\u0026#34;jsonrpc\u0026#34;:\u0026#34;2.0\u0026#34;,\u0026#34;id\u0026#34;:4,\u0026#34;method\u0026#34;:\u0026#34;tools/call\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;name\u0026#34;:\u0026#34;delete_everything\u0026#34;,...}}\u0026#39; HTTP/1.1 400 Bad Request {\u0026#34;jsonrpc\u0026#34;:\u0026#34;2.0\u0026#34;,\u0026#34;error\u0026#34;:{\u0026#34;message\u0026#34;:\u0026#34;MCP header attributes do not match request body\u0026#34;,\u0026#34;code\u0026#34;:-32020},\u0026#34;id\u0026#34;:4} The client gets a standard MCP error (-32020, HeaderMismatch), so MCP SDKs can recognize it. Envoy checks that:\nthe required headers are present; the headers match the body; the protocol version in the header matches the one in the body; the request declares the client\u0026rsquo;s capabilities. Not ready to block traffic yet? Leave traffic_mode at its default, PASS_THROUGH, with attribute_source set to BODY or VERIFY. Envoy then lets every request through, routes on what the body says, and counts header mismatches in the mcp.header_mismatch stat, so you can see what enforcement would do before you turn it on.\nControl who can call which tool Not all tools carry the same risk. Reading the weather is harmless; deleting a record is not. The MCP filter puts the tool name into Envoy\u0026rsquo;s request metadata, so Envoy\u0026rsquo;s existing authorization filters can make per-tool decisions. You don\u0026rsquo;t need a new policy engine. Add the authorization filter after the MCP filter in the same chain.\nOn the MCP filter, set traffic_mode: REJECT_NO_MCP and attribute_source: VERIFY (or BODY). The tool name then comes from the request body, which is the call the server will actually run. Requests whose headers disagree with the body are rejected before any policy is checked.\nRBAC: rules in config The RBAC filter can match on the MCP filter\u0026rsquo;s metadata. This rule blocks delete_record for any caller without the admin role:\n- name: envoy.filters.http.rbac typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC rules: action: DENY policies: delete-requires-admin: permissions: - metadata: filter: envoy.filters.http.mcp path: [{key: params}, {key: name}] value: {string_match: {exact: delete_record}} principals: - not_id: header: {name: x-user-role, string_match: {exact: admin}} Calling each tool with and without the role gives:\ntools/call get_weather -\u0026gt; 200, reaches the MCP server tools/call delete_record -\u0026gt; 403 RBAC: access denied tools/call delete_record (x-user-role: admin) -\u0026gt; 200, reaches the MCP server Spoofing the header doesn\u0026rsquo;t get around the rule. A request that names get_weather in Mcp-Name but delete_record in the body is rejected with -32020 before RBAC runs. A request that claims an older protocol version is still denied, because the tool name comes from the body.\nIn production, set the role from a verified identity, for example a claim in a JWT validated by Envoy\u0026rsquo;s JWT authentication filter. Never use a value the client sends directly.\next_authz: decisions in your own service When the decision depends on more than static rules, such as user entitlements, tenant limits, or an approval step, use the external authorization filter with a gRPC authorization service. metadata_context_namespaces tells Envoy to include the MCP filter\u0026rsquo;s metadata in each check:\n- name: envoy.filters.http.ext_authz typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz transport_api_version: V3 grpc_service: envoy_grpc: {cluster_name: authz} metadata_context_namespaces: [envoy.filters.http.mcp] For a delete_record call, the authorization service receives this metadata alongside the usual request attributes:\n{ \u0026#34;method\u0026#34;: \u0026#34;tools/call\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;delete_record\u0026#34;, \u0026#34;_meta\u0026#34;: { \u0026#34;io.modelcontextprotocol/protocolVersion\u0026#34;: \u0026#34;2026-07-28\u0026#34;, \u0026#34;io.modelcontextprotocol/clientCapabilities\u0026#34;: {} } }, \u0026#34;id\u0026#34;: 1, \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;is_mcp_request\u0026#34;: true, \u0026#34;status\u0026#34;: \u0026#34;mcp_ok\u0026#34; } The service reads params.name, applies its own logic, and answers allow or deny. A small test service that requires the admin role for delete_record produced:\ntools/call get_weather -\u0026gt; 200, reaches the MCP server tools/call delete_record -\u0026gt; 403 tool delete_record requires the admin role tools/call delete_record (x-user-role: admin) -\u0026gt; 200, reaches the MCP server Turn a REST API into an MCP server Many tools you want to give an agent already exist as REST APIs. The MCP REST bridge lets Envoy present them as an MCP server, with no changes to the API:\n- name: envoy.filters.http.mcp_json_rest_bridge typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.mcp_json_rest_bridge.v3.McpJsonRestBridge server_info: max_supported_protocol_version: \u0026#34;2026-07-28\u0026#34; server_discovery_cache_config: ttl: 3600s tool_config: tool_list_local: {} tools: - name: get_weather http_rule: get: /v1/weather/{city} tool_list_config: description: Current weather for a city. input_schema: \u0026#39;{\u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: {\u0026#34;city\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;}}, \u0026#34;required\u0026#34;: [\u0026#34;city\u0026#34;]}\u0026#39; This config defines one tool, get_weather. When an agent calls it with {\u0026quot;city\u0026quot;: \u0026quot;Paris\u0026quot;}, Envoy sends GET /v1/weather/Paris to your API and wraps the JSON response in an MCP result:\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 3, \u0026#34;result\u0026#34;: { \u0026#34;content\u0026#34;: [{\u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;{\\\u0026#34;city\\\u0026#34;:\\\u0026#34;Paris\\\u0026#34;,\\\u0026#34;temp_c\\\u0026#34;:18}\u0026#34;}], \u0026#34;isError\u0026#34;: false, \u0026#34;resultType\u0026#34;: \u0026#34;complete\u0026#34; } } Envoy also takes care of the rest of the new protocol for you:\nit answers tools/list from the config, so agents can discover the tool; it answers server/discover, the new call clients use to learn what a server supports; it checks the MCP headers against the body itself, so it doesn\u0026rsquo;t need the MCP filter in front of it. A server/discover call returns the supported protocol versions, the server\u0026rsquo;s capabilities, and a hint that clients may cache the answer for an hour:\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 1, \u0026#34;result\u0026#34;: { \u0026#34;supportedVersions\u0026#34;: [\u0026#34;2024-11-05\u0026#34;, \u0026#34;2025-03-26\u0026#34;, \u0026#34;2025-06-18\u0026#34;, \u0026#34;2025-11-25\u0026#34;, \u0026#34;2026-07-28\u0026#34;], \u0026#34;capabilities\u0026#34;: {\u0026#34;tools\u0026#34;: {\u0026#34;listChanged\u0026#34;: false}}, \u0026#34;_meta\u0026#34;: {\u0026#34;io.modelcontextprotocol/serverInfo\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;mcp.example.com\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;1.0.0\u0026#34;}}, \u0026#34;ttlMs\u0026#34;: 3600000, \u0026#34;cacheScope\u0026#34;: \u0026#34;public\u0026#34;, \u0026#34;resultType\u0026#34;: \u0026#34;complete\u0026#34; } } Serve old and new clients together Not every client will move to the new revision at once, and you don\u0026rsquo;t have to choose between them. One Envoy listener serves both:\nClients on 2026-07-28 get the stateless behavior described above. Clients on older versions keep their initialize handshake and sessions, and work exactly as before. You decide when to adopt the new revision with one setting, max_supported_protocol_version. On the REST bridge, setting it to \u0026quot;2026-07-28\u0026quot; turns the new behavior on. On the MCP filter, it caps the versions you accept. A client that asks for a version you don\u0026rsquo;t support gets a clear error listing the versions you do:\n{\u0026#34;jsonrpc\u0026#34;:\u0026#34;2.0\u0026#34;,\u0026#34;id\u0026#34;:null,\u0026#34;error\u0026#34;:{\u0026#34;code\u0026#34;:-32022,\u0026#34;message\u0026#34;:\u0026#34;Unsupported MCP protocol version: 2027-01-01\u0026#34;, \u0026#34;data\u0026#34;:{\u0026#34;requested\u0026#34;:\u0026#34;2027-01-01\u0026#34;,\u0026#34;supported\u0026#34;:[\u0026#34;2024-11-05\u0026#34;,\u0026#34;2025-03-26\u0026#34;,\u0026#34;2025-06-18\u0026#34;,\u0026#34;2025-11-25\u0026#34;,\u0026#34;2026-07-28\u0026#34;]}}} Try it Save this as envoy.yaml. It runs both extensions in one filter chain to show them together, though each also works on its own. A fixed response stands in for the weather API:\nstatic_resources: listeners: - name: mcp address: socket_address: {address: 0.0.0.0, port_value: 10000} filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: mcp http_filters: - name: envoy.filters.http.mcp typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.mcp.v3.Mcp traffic_mode: REJECT_NO_MCP attribute_source: VERIFY max_supported_protocol_version: \u0026#34;2026-07-28\u0026#34; - name: envoy.filters.http.mcp_json_rest_bridge typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.mcp_json_rest_bridge.v3.McpJsonRestBridge server_info: max_supported_protocol_version: \u0026#34;2026-07-28\u0026#34; server_discovery_cache_config: ttl: 3600s tool_config: tool_list_local: {} tools: - name: get_weather http_rule: get: /v1/weather/{city} tool_list_config: description: Current weather for a city. input_schema: \u0026#39;{\u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: {\u0026#34;city\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;}}, \u0026#34;required\u0026#34;: [\u0026#34;city\u0026#34;]}\u0026#39; - name: envoy.filters.http.router typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.http.router.v3.Router route_config: virtual_hosts: - name: weather domains: [\u0026#34;*\u0026#34;] routes: - match: {prefix: /v1/weather/} direct_response: status: 200 body: {inline_string: \u0026#39;{\u0026#34;city\u0026#34;:\u0026#34;Paris\u0026#34;,\u0026#34;temp_c\u0026#34;:18}\u0026#39;} response_headers_to_add: - header: {key: content-type, value: application/json} - match: {prefix: /} direct_response: {status: 404} Start Envoy:\ndocker run --rm -p 10000:10000 envoyproxy/envoy:v1.40.0 --config-yaml \u0026#34;$(cat envoy.yaml)\u0026#34; Call the tool:\ncurl -s localhost:10000/mcp \\ -H \u0026#39;content-type: application/json\u0026#39; -H \u0026#39;accept: application/json, text/event-stream\u0026#39; \\ -H \u0026#39;mcp-protocol-version: 2026-07-28\u0026#39; -H \u0026#39;mcp-method: tools/call\u0026#39; -H \u0026#39;mcp-name: get_weather\u0026#39; \\ -d \u0026#39;{\u0026#34;jsonrpc\u0026#34;:\u0026#34;2.0\u0026#34;,\u0026#34;id\u0026#34;:3,\u0026#34;method\u0026#34;:\u0026#34;tools/call\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;name\u0026#34;:\u0026#34;get_weather\u0026#34;,\u0026#34;arguments\u0026#34;:{\u0026#34;city\u0026#34;:\u0026#34;Paris\u0026#34;},\u0026#34;_meta\u0026#34;:{\u0026#34;io.modelcontextprotocol/protocolVersion\u0026#34;:\u0026#34;2026-07-28\u0026#34;,\u0026#34;io.modelcontextprotocol/clientCapabilities\u0026#34;:{}}}}\u0026#39; Then change mcp-name to another tool, or remove the mcp-name header entirely, and watch Envoy reject the request.\nThe MCP extensions are marked alpha in 1.40. Try them out, and share feedback on GitHub.\nLearn more MCP 2026-07-28 changelog and Streamable HTTP transport SEP-2575: Make MCP stateless Envoy 1.40.0 release notes and MCP filter docs Envoy authorization: RBAC filter and external authorization filter ","permalink":"https://botengyao.github.io/posts/stateless-mcp-envoy/","summary":"MCP\u0026rsquo;s 2026-07-28 revision makes the protocol stateless, so MCP traffic now looks a lot like ordinary HTTP. Envoy 1.40 builds on that: route MCP requests by tool name, stop malformed or spoofed requests at the proxy, control who can call which tool, and turn any REST API into an MCP server using only config.","title":"Stateless MCP in Envoy 1.40"},{"content":"Method note, up front, because it changes how you should read this. I did not read atenet line by line. I pointed an AI agent at Agent Substrate\u0026rsquo;s networking code and had it read the whole thing in seven passes, each with a different question, then had a second round of agents try to refute everything the first round produced. Eight findings did not survive that; four of them are quoted at the bottom. Every citation below resolves to a file and line at a named commit, which is the only reason a list produced this way is worth publishing: you can check any of it yourself, and I would rather you did than take it on trust.\nAgent Substrate multiplexes many agent \u0026ldquo;actors\u0026rdquo; onto a smaller pool of already-warm worker pods, which is the whole trick: it keeps the Kubernetes control plane out of the request path so a request can reach a suspended agent in milliseconds rather than seconds. atenet is the part that gets traffic there. It is an Envoy xDS control plane, an ext_proc server that resolves which worker currently holds a given actor and resumes it if nobody does, and a second gateway on the way back out that authenticates the actor behind every outbound CONNECT.\nIt is a genuinely good piece of engineering, and the comments in it are unusually honest — several of the items below are things the code already says about itself in a TODO. What follows is the engineering half of what the scan turned up: 52 items about correctness, capacity, operations, observability, and test coverage.\nThere is a security half too. It is not in this post. The repo ships a SECURITY.md asking for private reports through GitHub advisories and asking reporters to hold details until a fix lands, and it seemed poor form to read that file and then ignore it. Those findings are going through that channel. If they get advisories, I\u0026rsquo;ll write them up then.\nThe shape of it Read at agent-substrate/substrate @ 69828945 Surface cmd/atenet, both gateway manifests, the agentgateway overlays, internal/atunnel, the EgressPolicy chain Method 7 reading passes, then adversarial verification of every finding Findings 129 raised, 121 survived verification, deduplicated into 76 items Items here 52 — 6 P0, 29 P1, 17 P2 Already TODO\u0026rsquo;d in-tree 6 Held for private disclosure 24 Tests go test ./cmd/atenet/... ./internal/atunnel/... passes clean Three of the findings are worth walking through, less because they are the most severe than because each one is a small mechanical thing standing in for a larger habit — and because they are the three that were re-checked directly against the code, rather than taken on the agent\u0026rsquo;s word.\nThe line that was never backported Substrate ships two egress manifests. atenet-egress.yaml is the default; atenet-egress-with-sdsmint.yaml is the variant that terminates and re-originates TLS so it can see inside the tunnel. They are siblings, and they drifted.\nThe sdsmint one sets this on its CONNECT route:\n- match: { connect_matcher: {} } route: # ...five lines of comment about the MITM listener... cluster: mitm_internal # Set request timeout to disabled (0) as CONNECT is a long-running stream. # There will still be a stream_idle_timeout (5 minutes by default). timeout: 0s upgrade_configs: - upgrade_type: CONNECT connect_config: {} The default one does not:\n- match: { connect_matcher: {} } route: cluster: egress_forward_proxy upgrade_configs: - upgrade_type: CONNECT connect_config: {} Envoy applies the route timeout to a CONNECT tunnel\u0026rsquo;s entire lifetime, not just to its headers. Left at the 15-second default, that is not a timeout on a slow request — it is a hard cap on how long any actor\u0026rsquo;s outbound connection may exist. Server-sent events, gRPC streaming, an SSH session, a slow model call, any download over fifteen seconds: all cut, mid-stream, on the default install path.\nThe router\u0026rsquo;s own xDS code gets this right for the ingress CONNECT listener, and says why:\n// Envoy applies the route timeout to a CONNECT tunnel\u0026#39;s whole // lifetime, not just its headers; left at Envoy\u0026#39;s 15s default // this would silently kill every tunnel after 15s. 0 disables // it -- the tunnel\u0026#39;s lifetime is bounded by idle timeout, // drain, and the peers themselves. Timeout: durationpb.New(0), So the knowledge is in the tree three times — twice in prose, once in Go — and missing from the one file hack/install-ate.sh actually applies. This is the cheapest fix on the list and the one I would do first. The durable version of it is an end-to-end test that holds a tunnel open past twenty seconds and asserts bytes still flow, so the two manifests cannot drift apart again quietly.\nThis is also the kind of finding the method is genuinely good at. Nobody reading either file alone would notice; the bug exists only in the diff between two files nobody diffs.\nA reconcile that lists, then throws it away The router keeps Envoy configured through a loop that ticks every five seconds:\nfunc (c *Controller) reconcile(ctx context.Context) error { _, err := c.atStore.readyTemplates(ctx) if err != nil { /* ... */ } if err := c.xdsSrv.UpdateSnapshot(); err != nil { /* ... */ } return nil } Two things are happening here, and neither is doing what it looks like.\nreadyTemplates is an unpaginated, cluster-wide List of every ActorTemplate, and the result is assigned to _. It is also not served from an informer cache — the client is built with client.New, which is the direct, uncached one — so this is a real API-server round trip. Nothing in the generated xDS resources derives from ActorTemplate at all; the only genuine consumer is the /statusz page, which does its own fetch.\nUpdateSnapshot then increments the version counter unconditionally and calls SetSnapshot. go-control-plane pushes on a version change, so Envoy re-ingests the full listener, cluster, route and secret set every five seconds, forever, whether or not a single byte differs. On an idle cluster that is roughly seventeen thousand pushes and seventeen thousand \u0026ldquo;Deploying updated xDS configuration snapshot\u0026rdquo; log lines per replica per day.\nThe obvious reason someone might have written it this way is that the periodic push refreshes the TLS material. It doesn\u0026rsquo;t: the secret is delivered as a DataSource_Filename with a WatchedDirectory, so Envoy re-reads the file on its own when kubelet rotates it. The push is not load-bearing. That check is the kind of thing the verification round is for — the first pass reported the churn, the second pass went looking for the reason it might be deliberate, and not finding one is what makes the finding worth acting on.\nThe fix is to hash the built resource set and skip the version bump when nothing changed, and to delete the discarded List. What makes it worth writing down is the test that goes with it, which is one line of intent: two back-to-back reconciles should produce exactly one SetSnapshot.\nThe benchmark measures a different gateway The shipped ingress Deployment starts Envoy like this:\n- \u0026#34;--component-log-level\u0026#34; - \u0026#34;upstream:debug,router:debug,ext_proc:debug\u0026#34; No comment, no time-box, no overlay that removes it. Those are three of Envoy\u0026rsquo;s hottest components, each emitting several lines per request on the data path, and ext_proc debug output puts request headers — actor hostnames included — into cluster logs by default. The egress Envoy, running the same binary, carries no such flag.\nThe part worth stopping on is in the load-test harness: before it measures anything, nighthawk_ingress.py strategic-patches the Envoy container\u0026rsquo;s entire command to remove those flags — and documents why, in the docstring of the function that does it. So the harness is honest about measuring something other than the default. What nobody says is the corollary: the configuration that actually ships, debug logging and all, has never been benchmarked, and the manifest carries no note to that effect.\nIt sits next to two other manifest gaps that read the same way — as things nobody decided, rather than things somebody chose. The ingress Envoy container has no readiness, liveness, or startup probe of any kind, while its egress twin has two; and neither gateway sets resource requests or limits, which puts both in BestEffort QoS, first to be evicted under node pressure. In each case the sibling manifest already demonstrates the intended shape. That is what makes them cheap: there is nothing to design, only something to copy.\nThe list Priorities and effort estimates below are the scan\u0026rsquo;s, not a maintainer\u0026rsquo;s. P0 means it would not ship a cluster without it; P1 means it will hurt within a quarter; P2 means it is worth doing when the file is open anyway. Items marked tracked already have a TODO or roadmap entry in-tree — kept in the list because a TODO is a note to self, not a plan, and several of these have been notes to self for a while.\nALL 52P0P1P2 1Envoy / xDS hardening9 1.1Stop pushing an identical xDS snapshot every 5s, and drop the discarded ActorTemplate LISTP1Mcontroller.go:49,64-67 · atstore.go:41-44 · router.go:106-108 · xds.go:418,481\nreconcile does _, err := c.atStore.readyTemplates(ctx) — an unpaginated, cluster-wide LIST on an uncached client, result discarded — then bumps versionCount++ unconditionally. About 17k pushes and log lines per replica per day on a fully idle cluster. Nothing in the generated resources derives from ActorTemplates, and cert rotation does not depend on the push.\nDoHash the built resource set and skip SetSnapshot when unchanged; delete the readyTemplates call or move it behind an informer. Test: two back-to-back reconciles produce exactly one SetSnapshot.\n1.2Surface xDS NACKs — add callbacks, a logger, and freshness metricsP1Mxds.go:207-208 — NewSnapshotCache(true, IDHash{}, nil) and NewServer(ctx, cache, nil)\nNil logger, nil callbacks. A rejected Listener (unreadable cert path, an SDS secret not yet projected, an unknown extension) leaves Envoy on stale or empty config while the router logs the push as a success. Nothing in /statusz distinguishes accepted from rejected, and no atenet.router.xds.* instrument exists.\nDoPass a slog logger and serverv3.CallbackFuncs; log and count req.GetErrorDetail() in OnStreamRequest. Export an xDS NACK counter, snapshot-version gauge, and last-accepted-age gauge. Add a --xds-node-id flag — the magic string substrate-envoy-node must match the manifest with no compile-time link.\n1.3Add slowloris and connection-lifetime bounds to the ingress listenersP1Mxds.go:1060-1087, :908-949, :981\nNeither HCM sets RequestTimeout, an explicit StreamIdleTimeout, CommonHttpProtocolOptions, or MaxConcurrentStreams. Nothing bounds accept→complete-headers, so a slow-drip client pins a connection and a stream indefinitely and the route timeout never arms. Note: per_connection_buffer_limit_bytes (1 MiB) and stream_idle_timeout (5m) already have safe defaults — the real gaps are the header-phase timer, the stream cap, and connection lifetime.\nDoSet RequestTimeout (a few seconds), an explicit StreamIdleTimeout, CommonHttpProtocolOptions{IdleTimeout, MaxConnectionDuration, MaxRequestsPerConnection}, and Http2ProtocolOptions{MaxConcurrentStreams ~100}. Expose as flags beside --route-timeout.\n1.4Replace the default Envoy access-log format with a structured oneP1Strackedxds.go:1039 (bare StdoutAccessLog), reused at :916-923, under the TODO at :914-915\nThe default format carries no %RESPONSE_CODE_DETAILS%, %CONNECTION_TERMINATION_DETAILS%, or %UPSTREAM_TRANSPORT_FAILURE_REASON% — so a parking shed 503, a circuit-breaker UO 503, an ext_proc message timeout, an atunnel mTLS handshake failure, and a route-timeout 504 are all one indistinguishable line. The egress manifest already proves the pattern is acceptable here.\nDoAdd a shared SubstitutionFormatString json_format applied in both HCM builders. Closes the existing TODO.\n1.5Give the actor ORIGINAL_DST cluster explicit circuit breakers and a cleanup_intervalP1Sxds.go:745-786 (buildOriginalDstCluster) — only ConnectTimeout, LbConfig, TransportSocket, protocol options\nThe cluster carrying all actor traffic runs on Envoy's implicit 1024 max_connections / max_requests, cluster-wide rather than per actor — the 1025th request gets an opaque UO 503 from a limit no flag names, while the deliberately-sized ext_proc breaker above it is 2048. Separately, cleanup_interval defaults to 5s for ORIGINAL_DST, so every actor idle \u0026gt;5s pays a fresh TCP connect plus a full mTLS handshake — the common case for a many-intermittent-actors substrate.\nDoSet CleanupInterval: 60s and explicit CircuitBreakers, both flag-driven, validated against --extproc-max-requests the way --parked-request-max already is.\n1.6Decouple the ext_proc message timeout from the parking budgetP1Sdataplane.go:73-77 · xds.go:1005,1013 · ingress/resumer.go:38\nSetExtProcMessageTimeout(budget + 5s) runs only when parking is enabled, and lands in both GrpcService.Timeout and MessageTimeout. So with parking on, the millisecond-scale fast path inherits a 10s+ timeout; with --parked-request-max=0 the timeout stays 5s while failFastResumeBudget is 15s — any cold restore over 5s is abandoned by Envoy and, being fail-closed, the client gets a generic 500 instead of the router's mapped 503.\nDoKeep MessageTimeout at the default; set MaxMessageTimeout to max(budget, failFastResumeBudget) + margin and emit override_message_timeout from the ingress handler at the moment it parks. Set FailureModeAllow: false explicitly.\n1.7Add a retry policy and edge normalization to the actor routeP2Sxds.go:802-814 — no RetryPolicy anywhere in cmd/atenet/internal/router/\nA closed idle pooled mTLS connection surfaces as an immediate 503 UC/UF even though the request never reached the actor — common while cleanup_interval stays at 5s, so fix 1.5 first. Separately no NormalizePath, MergeSlashes, or PathWithEscapedSlashesAction: the gateway doesn't route on path, so this isn't a bypass — it just means every actor inherits unnormalized paths for its own authorization decisions.\nDoFlag-gated RetryPolicy (connect-failure,refused-stream,reset-before-request-headers-sent), off by default for non-idempotent workloads. Set NormalizePath, MergeSlashes, HeadersWithUnderscoresAction: REJECT_REQUEST.\n1.8Validate generated resources, and stop discarding the marshal errorP2Mxds.go:465-481 (only snapshot.Consistent()) · newAny at :1344-1347\nConsistent() validates name references, not resource validity; the vendored PGV Validate() is unused and no CI step runs envoy --mode validate. newAny swallows the anypb.New error at 17 call sites, so a nil typed_config would ship, pass Consistent(), log as success, and be NACKed invisibly.\nDoAdd a CI check rendering the snapshot into a bootstrap and running envoy --mode validate against the pinned image — that is the valuable half. Call ValidateAll() before SetSnapshot; make newAny return an error or panic at snapshot-build time.\n1.9Add HTTP/2 keepalive to the ext_proc and OTLP clustersP2Sxds.go:560-568, :608-616 — Http2ProtocolOptions{} with no ConnectionKeepalive\nWeakest for ext_proc (a single loopback endpoint); genuinely useful for buildOtlpCollectorCluster, which is STRICT_DNS to an off-pod collector where a silently dropped connection through a NAT or LB blackholes spans until TCP notices.\nDoSet ConnectionKeepalive{Interval, Timeout} on both.\n2Ingress path7 2.1Don't treat the :authority port as the actor target port on plain ingressP1Mingress/ingress.go:110-117 · xds.go:876-905, wired at :1146 and :1209\ntargetPort is derived from the authority unconditionally, and the same dev.ate.authority filter state feeds every listener — nothing distinguishes CONNECT-terminated traffic. The sharp case: grpc-go sets :authority to the dial target including the port, so a client dialing …:443 at the HTTPS ingress gets targetPort=443 and atunnel proxies to 443 on the actor instead of 80 — connection refused, nothing in the logs naming the cause. Separately SplitHostPort(\"host:abc\") succeeds, ParsePort rejects, and the code silently falls back to 80.\nDoHonour the authority port only for CONNECT-terminated traffic (a marker in filter state, or key off :method); return a 400 when a port component is present but unparseable. Add test cases for :0, :99999, :abc, and a plain-ingress :8080 — today there is not one authority-with-port case in the table.\n2.2Add a per-actor cap to the parking lotP1Mparking.go:163-176 (enter takes no actor identity) · ingress.go:124 · parking.go:37\nThe only state is l.active vs l.cfg.Max (default 1024). One saturated actor — or one client hammering one name — can hold every slot for the full budget, after which every other actor is shed with \"router at capacity\". Singleflight does not help: it coalesces the ateapi calls but every joined request still holds its own slot. The ext_proc breaker above it is a strictly larger global bound, not a per-actor one.\nDoChange to enter(ctx, actorRef) with a map[ActorRef]int under the same mutex; reject on either the global Max or a new --parked-request-max-per-actor. Label the rejected counter with which limit fired. Test: saturating actor A still admits actor B.\n2.3Bound concurrent detached ResumeActor flightsP1Mresumer.go:183,192,202 · ingress.go:130-131 · ateinterceptors.go:78-83\nThe flight context is built from context.Background(), cancellation is stripped, and the RPC runs with no deadline; the parking slot is released the moment the caller stops waiting. The only ceiling is ateapi's server-side 10-minute max deadline. So --parked-request-max bounds concurrent callers, not concurrent flights — under churn across distinct cold actors, in-flight RPCs accumulate at admission rate × 10 min.\nDoAdd a semaphore held for the whole flight lifetime capping concurrent detached ResumeActor RPCs. Do not add a per-attempt deadline alone — that reintroduces the stranded-worker bug (#675) the code deliberately avoids.\n2.4Propagate trace context into ResumeActorP1Sresumer.go:167-169 starts the span from the caller ctx; :183 roots the flight at context.Background()\nThe otelgrpc-instrumented client emits every outbound ResumeActor span as a disconnected trace root, and baggage is dropped. The single most valuable trace — \"why did this request take 4 seconds?\" — is split in two, and it looks correct in a trace UI because the router-local span is parented fine.\nDobgCtx := context.WithTimeout(context.WithoutCancel(ctx), r.budget), matching the stated intent in the comment above it, keeping the context.WithoutCancel at :192 — it strips the deadline as well as cancellation, which is what stops the budget from cancelling an in-flight ResumeActor (#675). Add a span link or joiner-count attribute since the leader's span is shared.\n2.5Demote the four per-request INFO log linesP2Singress/ingress.go:87, 129, 145, 159 · egress.go:127\nFour slog.InfoContext records per successful request (three via reflective slog.Any), JSON-encoded through one shared stdout writer at the default level, duplicating what the route-duration metric, the trace, the /statusz recorder, and Envoy's own access log already carry. Line :87 logs raw client-controlled md.Host at INFO for scanner traffic.\nDoDemote :87/:129/:145 to Debug, keep at most one INFO, replace slog.Any with explicit slog.String. Update TestHandleRequestHeadersDoesNotLogSensitiveData alongside — it pins what these lines may contain.\n2.6Give invalid-host rejections their own metric outcomeP2Singress/errors.go:36-39 · ingress.go:100-107\nBoth the missing-attribute case and the unparseable-DNS-name case return NotFound, which classifyOutcome maps to not_found — the same label a genuine ateapi NotFound produces. The valuable half: dev.ate.authority is set on every ingress listener, so an empty attribute can only mean a broken dataplane config that would silently 404 all traffic — today indistinguishable from scanner noise.\nDoTyped sentinels so classifyOutcome emits invalid_host and missing_authority_attribute separately; alert on the latter \u0026gt; 0.\n2.7Fix the retryable-code table in docs/request-parking.mdP2Sdocs/request-parking.md:101, :109, :116 vs ingress/resumer.go:143-161\nThe doc names FailedPrecondition as the worker-pool-saturation code; the real capacity code is ResourceExhausted. An on-call engineer building a dashboard filter from the table sees nothing. Lines :23-24 and :33-34 are correct — only the table and :116 are wrong.\nDoAdd a ResourceExhausted (no free workers) row, retitle the FailedPrecondition row, fix :116.\n3Egress path \u0026amp; actor identity7 3.1Add timeout: 0s to the default egress CONNECT routeP0Smanifests/ate-install/atenet-egress.yaml:107-112 — no route timeout anywhere in the HCM (78-155)\nEnvoy's 15s route timeout kills every actor egress tunnel 15 seconds after establishment regardless of activity: long-poll, SSE, gRPC streaming, slow LLM calls, SSH, any download over 15s. This is drift, not a design difference — commit 6fd47c13 added timeout: 0s with the rationale \"the route timeout runs until the response is completely processed, which for a stream is never\" to the sdsmint manifest only, never backported. hack/install-ate.sh:243-249 confirms atenet-egress.yaml is the default.\nDoOne line. Add an e2e holding a tunnel past 20s and asserting bytes still flow, so the two manifests cannot drift again.\n3.2Make the actor-identity CA trust bundle reloadableP1Mtrackedrouter.go:238-249 (one-shot os.ReadFile) · atenet-egress.yaml:70-76 (static trusted_ca) · internal/localca/localca.go:56-64\nNothing re-reads the bundle, and the Envoy side uses a static validation_context while the serving cert ten lines above deliberately goes through filesystem SDS with a watched_directory. localca is built around seamless publish-new-root / age-in / switch rotation — the egress path defeats it: the moment a new CA becomes ActiveForSigning, every fresh actor certificate fails both the handshake and leaf.Verify, and all actor egress in the cluster stops until someone re-runs the installer and restarts both containers.\nDoPublish the roots as a ClusterTrustBundle (the TODO at hack/install-ate.sh:382; the pattern exists in egressmitmtrust_controller.go), consume via a projected volume, deliver the validation context over filesystem SDS with a watched_directory, and replace the one-shot read with an atomic.Pointer[x509.CertPool].\n3.3Put deadlines on atunnel's certificate mint and gateway dialP1Sinternal/atunnel/egress.go:119, :145, :265 · credential.go:145-149 · client.go:136-160\nThe activation ctx has no deadline. A broker that accepts but never answers parks the renewal goroutine forever, the retry/backoff never runs, the cert expires, and every new tunnel is refused with no recovery short of Deactivate/Activate. On the dial side, ReadResponse ignores ctx entirely and sets no read deadline, and the context.AfterFunc that closes both conns is registered only after DialContext returns — so a gateway that completes TLS and never answers CONNECT pins that goroutine and both sockets permanently.\nDoBound the mint (e.g. 10s) inside BrokerCertificateSource.Mint, and add a configurable handshake+CONNECT budget inside Client.DialContext, including an explicit read deadline around ReadResponse.\n3.4Bound and cache the per-CONNECT GetActorP1Mtrackedegress/egress.go:167-170 — TODO cites issue #592, directly above an uncached, deadline-free GetActor\nThere is no admission control on this path at all — the parking lot is built only inside ingress.New. So an actor opening many short-lived connections multiplies ateapi QPS one-for-one, undamped, and because the filter is fail-closed, ateapi slowness becomes egress unavailability with no shedding in between. The \"leaked goroutine per CONNECT\" claim is wrong — Envoy's message_timeout reset cancels the stream context; the real cost is the serial Process loop holding the stream for the full 5s.\nDoWrap GetActor in a sub-5s timeout (2s matches the configured grpc_service timeout); add singleflight coalescing as the safe first step, then a short-TTL positive cache keyed by (atespace, name, uid) — noting a positive cache defers revocation. Add a bounded-concurrency gate mirroring the ingress lot.\n3.5Clamp MITM leaf NotAfter to the signing root, and alert on root expiryP1Strackedinternal/localca/localca.go:179-185 (TODO, signs unmodified) · sdsmint/minter.go:105 · deltastream.go:192-198\nThe consumer already assumes the clamp exists — deltastream.go says \"The floor is for the clamp, where notAfter is the CA's and can be nearer than that.\" The root is a fixed year with nothing monitoring it, so at the one-year mark every actor's HTTPS egress breaks at once while sdsmint keeps minting happily.\nDoThe durable half first: have sdsmint refuse to start (or log Error and export a notAfter gauge) when the active CA has less remaining life than --leaf-cert-ttl, so root expiry is visible weeks ahead. Then implement the clamp.\n3.6Serve the last-good CA pool on refresh failure; stop reporting mint failures as unknown SNIsP1Sinternal/localca/localca.go:107-126 · sdsmint/deltastream.go:161-169 · minter.go:100\nOn a read error refreshIfNecessary returns without assigning p.pool or advancing p.nextLoad — so a transient FS error fails the call despite a good in-memory pool, and every subsequent call re-reads the file. Downstream, every minter error collapses into \"tell Envoy the name does not exist\" even though the minter already distinguishes errHostNotAllowed. A CA-pool read failure is reported identically to a syntax rejection, with no error on the SDS stream and no metric (there is no telemetry file in sdsmint at all).\nDoKeep serving p.pool on a failed reload (log, back off nextLoad); hard-fail only when there is no pool. Branch handleSubscribe on errors.Is(err, errHostNotAllowed) — only that becomes RemovedResources; any other error returns so Envoy retries, with a distinct counter.\n3.7Fix the --leaf-cert-ttl help textP2Ssdsmint/cmd.go:44 vs deltastream.go:198\nThe flag says the xDS resource TTL is \"half its length, so an actively used name is re-minted about twice per lifetime\". The actual derivation is max(time.Until(cert.NotAfter)-time.Minute, time.Second) — lifetime minus one minute, as the manifest correctly documents. An operator sizing sdsmint from --help gets the re-mint rate wrong by ~2×: --help advertises about two re-mints per leaf lifetime where the real TTL (notAfter − 1m) gives about one.\nDoRewrite the flag description; add a unit assertion pinning pack()'s TTL to notAfter − 1m.\n4ext_proc server4 4.1Set concurrency and message-size limits, and add panic recoveryP2Sextproc/extproc.go:80-85 — the whole gRPC server construction\ngrpc-go defaults are maxConcurrentStreams: MaxUint32 and a 4 MiB receive limit; each stream is a goroutine plus a map built from every header. grpc-go also does not recover handler panics, and this process hosts the xDS control plane — so one panic takes config service down for the whole ingress fleet. No reachable panic path was found today, and the keepalive claims were refuted — grpc-go already enforces them.\nDogrpc.MaxConcurrentStreams sized from extProcMaxRequests(), grpc.MaxRecvMsgSize ~256 KiB, grpc.MaxHeaderListSize; add a stream interceptor that recovers, logs the stack, counts a panic, and returns codes.Internal so only that stream dies.\n4.2Reject unexpected ProcessingRequest types instead of answering themP2Sextproc/extproc.go:104-115\nThe default: branch logs at Error once per message and answers with an empty RequestHeaders response regardless of what arrived. Unreachable from a correctly configured Envoy (both dataplanes set body NONE / trailers SKIP), so the real cost is an unrate-limited error-log amplifier from a non-Envoy client.\nDoReturn status.Errorf(codes.InvalidArgument, …) to close the stream; log at Warn with a first-per-stream guard.\n4.3Truncate and fix the /statusz query ring bufferP2Sextproc/record.go:100-117 · metadata.go:64-66\nHost, redacted path, method, and :authority are stored verbatim with no length bound, retained until overwritten 100 requests later. Separately the \"Client\" column is a bug: Client is set from m.Headers[\":authority\"], which is already what Host holds — so the one operator-facing \"who sent this\" field is always a duplicate, which looks like real data during an incident and is not.\nDoTruncate stored strings to ~256 bytes; request the downstream remote address as a CEL attribute alongside the authority attribute and populate Client from it — or drop the column.\n4.4Parse the /statusz dashboard template onceP2Sstatus.go:191 — template.New(...).Parse(dashboardHTML) inside handleStatusz\nRe-runs lex, parse, and escape analysis against an 18 KB embedded constant on every request. It also means a template syntax error ships as a per-request 500 instead of failing at init.\nDoHoist to var dashboardTmpl = template.Must(...).\n5Deployment, manifests \u0026amp; ops12 5.1Give the ingress Envoy container a readiness probeP0Satenet-router.yaml:269-311 — no readiness, liveness, or startup probe · router.go:188-193 · components/agentgateway/kustomization.yaml:41-75\nThe pod's only probes are on the Go container, backed by a zero-value Readiness that reports ready until the drain flips it. Verified ordering: the readyz server starts before startDataplane() brings up the xDS listener, so /readyz is 200 while Envoy provably has no LDS/CDS — yet the Service targets Envoy's ports. A failed initial UpdateSnapshot is logged and Serve continues, and a NACK is invisible (1.2), so a config error becomes an outage instead of a stuck rollout. The egress Envoy gets both probes; ingress is the odd one out, and agentgateway ingress gets zero probes of any kind.\nDoAdd readiness and startup probes on /ready port admin to the ingress envoy container (and /healthz/ready for agentgateway); wire routerHealth.Report().Dataplane.Healthy plus a last-snapshot-accepted signal into the Readiness, keeping /healthz unconditional so liveness and readiness still diverge during drain.\n5.2Set resource requests and limits on both gatewaysP0MNo resources: on any container in atenet-router.yaml or atenet-egress.yaml · benchmarking/automation/testtypes/nighthawk_ingress.py:126-149\nBoth gateways land in BestEffort QoS: no CPU guarantee, first to be evicted under node memory pressure, scheduled as if free — and an Envoy memory blowup takes neighbours down rather than itself. The only counter-example is the sdsmint container. Direct evidence the shipped shape is not the intended one: the benchmark harness has to patch requests=limits in before it can measure (cpu only — memory is absent even there).\nDoAdd explicit cpu+memory requests/limits to the atenet-router, envoy, and ext-proc containers for Guaranteed QoS, matching Envoy's --concurrency, sizing memory above the fixed_heap threshold from 5.6.\n5.3Run more than one replica of each gateway, with PDBs and spreadingP0Matenet-router.yaml:150 · atenet-egress.yaml:211 · atenet-egress-with-sdsmint.yaml:738 — all replicas: 1\nNeither Deployment sets strategy, topologySpreadConstraints, or affinity, and the only PodDisruptionBudget in manifests/ belongs to the API server. So a node drain, cluster upgrade, spot preemption, or single OOM is 100% ingress or 100% egress offline, with nothing telling the eviction API the pod is critical — and the careful drain sequence has nowhere to hand traffic off to. The control plane already got this treatment; the data path did not.\nDo≥2 replicas with rollingUpdate: {maxUnavailable: 0, maxSurge: 1}, a maxUnavailable: 1 PDB each, and topology spread on hostname. Confirm first that per-replica in-memory parking state behaves acceptably split across replicas — --parked-request-max becomes per-replica capacity.\n5.4Drop the debug component log levels from the shipped router manifestP0Satenet-router.yaml:274-276 — --component-log-level upstream:debug,router:debug,ext_proc:debug\nNo comment, no time-box, no overlay that removes it; the egress Envoy carries no such flag. Three of Envoy's hottest components emit multiple lines per request on the data path, and ext_proc debug output puts client-supplied request headers — actor hostnames included — into cluster logs by default. The benchmark harness strategic-patches the entire envoy command to remove these flags before measuring, so any capacity number produced by that harness describes a configuration that does not ship. The harness explains why it patches; the manifest carries no note that its shipped default is unmeasured.\nDoRemove the args; move them to a debug-only kustomize overlay or drive them from the router's existing --log-level.\n5.5Scrape and trace the egress gateway — it currently emits nothingP0Satenet-egress.yaml:214-218 (labels only) · :305-357 (no envFrom, no OTEL_RESOURCE_ATTRIBUTES) · one PodMonitoring, selecting app: atenet-router\nNo prometheus.io/scrape annotation, and the kind Prometheus job keys entirely on it; OTEL_EXPORTER_OTLP_ENDPOINT is unset so the exporter runs into the void. Envoy's egress_connect stats, TLS handshake failures, and every egress authorization decision are unobservable in both environments — the only signal is the stdout access log. The metrics server is already running in egress mode, so this needs no code change.\nDoAdd the scrape annotations plus envFrom: [configMapRef: {name: ate-otel-config}] and OTEL_RESOURCE_ATTRIBUTES to the egress ext-proc container, and a PodMonitoring for app: atenet-egress scraping Envoy's /stats/prometheus.\n5.6Add an overload_manager to every Envoy bootstrapP1MNo overload_manager or global_downstream_max_connections anywhere in manifests/ or cmd/ · atenet-egress-with-sdsmint.yaml:817-830, :536-553\nWith no global downstream connection limit and no memory-pressure shedding — and, per 5.2, no memory limit for fixed_heap to work against — an accepting Envoy keeps taking connections and buffering until the container OOMs, which on the ingress gateway is a full outage rather than graceful degradation. The sdsmint variant is the sharpest case: it states a sizing contract against \"Envoy's fixed_heap\" and its circuit breakers explicitly defer to it — reasoning from two guards that are not configured.\nDoAdd overload_manager with envoy.resource_monitors.fixed_heap driving shrink_heap and stop_accepting_requests, plus global_downstream_max_connections, to all three bootstraps at the thresholds the sdsmint comment assumes. If those numbers are aspirational, delete the comment rather than leave a contract nothing enforces.\n5.7Fix the monitoring selectors that break in agentgateway modeP1Satenet-router-monitoring.yaml:29-35 vs components/agentgateway/kustomization.yaml:52-64, :74-75\nThe PodMonitoring scrapes named port: admin, but the overlay replaces the container's ports with http/https/connect/connect-tls/readiness/stats — no admin port exists after the patch, so on a cluster switched to agentgateway the leftover PodMonitoring resolves no target — a fresh agentgateway install never gets one, since base/kustomization.yaml does not list it. The patch also never touches the pod-template annotations, so prometheus.io/port: \"9090\" survives while the container that served 9090 is replaced by one that does not. Both environments silently lose gateway metrics; agentgateway's own stats on 15020 are collected by nothing.\nDoPatch the annotation to 15020 with the right path, and either rename the stats port to admin or add an agentgateway-specific PodMonitoring.\n5.8Give the ingress router pod the securityContext the egress pod already hasP1Satenet-router.yaml:162-163 — no pod-level securityContext, neither container has one · atenet-egress.yaml:221-236\nThe pod terminating all untrusted client traffic runs stock envoyproxy/envoy as the image's default user with the full default capability set, no privilege-escalation block, a writable root filesystem, and an unconfined seccomp profile — and unlike egress it needs no sysctl escape, since every listener is above 1024. grep seccompProfile manifests/ hits only the kind Prometheus manifest.\nDoCopy the egress securityContext onto both router containers (runAsNonRoot, runAsUser: 65532, allowPrivilegeEscalation: false, drop: [ALL]) and add readOnlyRootFilesystem with an emptyDir for Envoy scratch and add seccompProfile: {type: RuntimeDefault} at pod level on all three gateway manifests.\n5.9Set automountServiceAccountToken: false on the egress and agentgateway-ingress podsP1Sgrep automountServiceAccountToken manifests/ — zero hits repo-wide · atenet-egress.yaml:25-29 · components/agentgateway/kustomization.yaml\nFor egress the token is provably unused — the manifest comment asserts the invariant (\"No RBAC is bound to this ServiceAccount on purpose … It reaches the control plane over the ate API only\") and the container's args and mounts confirm it. Worse in agentgateway ingress mode: the overlay removes the only Kubernetes client but never touches serviceAccountName, so the pod keeps an automounted token carrying cluster-wide actortemplates get/watch/list that nothing in it uses — a standing least-privilege violation on the pod that terminates untrusted traffic.\nDoAdd automountServiceAccountToken: false to both egress manifests, and patch it (plus ideally a dedicated no-RBAC ServiceAccount) into the agentgateway ingress overlay.\n5.10Stop applying the whole ate-install directory on the GKE install pathP1Shack/install-ate.sh:224 (run_ko resolve -f manifests/ate-install) vs :283-289 · manifests/ate-install/base/kustomization.yaml:21-29\nApplying the directory applies both mutually exclusive egress manifests in one pass — they define the same ConfigMap, Deployment, and Service. Only afterwards does apply_atenet_egress pick one. The ordering is deterministic (ko walks the directory in sorted order and kubectl applies one stream), but which egress Deployment briefly exists is decided by filename collation rather than --experimental-use-sdsmint, every install pushes two conflicting Deployment specs at the API server, and a non-sdsmint install churns through a pod mounting a secret that does not exist yet.\nDokubectl kustomize manifests/ate-install/base | run_ko resolve -f -, with a GKE-only kustomization adding the monitoring manifest.\n5.11Digest-pin the remaining imagesP1Scomponents/agentgateway/kustomization.yaml:45, :118 · atenet-dns.yaml:95, :114\nExactly four unpinned refs under manifests/: cr.agentgateway.dev/agentgateway:v1.5.0 twice, busybox:1.36, and coredns/coredns:1.11.1. Every Envoy ref under manifests/ is digest-pinned. A re-pushed tag or a differing node cache silently runs different binaries across replicas with no way to tell from the manifest.\nDoPin all four to digests, and add a hack/verify/ check that every non-ko:// image reference carries @sha256: — it would catch four regressions today.\n5.12Make the egress readiness probe observe the router's drain flipP2Satenet-egress.yaml:339-341 (tcpSocket) vs atenet-router.yaml:242-247 (httpGet /readyz)\nPort 9090 is neither declared nor probed in the egress Deployment, and the TCP listener stays open for the whole --drain-delay, so the readiness flip is observed by nobody. Overstated in one review: on a normal rollout Kubernetes drops the pod from ready endpoints on deletionTimestamp regardless of probes — what is genuinely lost is the SIGTERM-without-deletion case, the operational drain signal, and consistency between two Deployments running the same binary.\nDoAdd the metrics port and switch to httpGet /readyz in both egress manifests; keep the tcpSocket check as a startupProbe.\n6Observability \u0026amp; metrics6 6.1Add a direction attribute and an egress-denial counterP1Sextproc.go:141 computes dir and :158 never passes it · metrics.go:63-73, :96-110 · dispatch.go:44-51\nrecordRouteDuration records only template namespace/name, outcome, and resume — the first two are always empty for egress and resume is meaningless there. classifyOutcome has no Forbidden case, so most egress denials are labelled outcome=\"resume_error\" on a metric named route.duration — an egress cert-rotation outage renders on dashboards as router resume failures. Worst of all, the code documents the exact silent failure this needs to catch: if Envoy cannot parse the CEL attribute it sends an empty map, directionOf returns ingress, and an --mode=egress instance 404s every CONNECT — a total outage that today looks like a spike of not_found with empty labels.\nDoThread dir into recordRouteDuration; add a denied bucket for Forbidden/Unauthorized/MethodNotAllowed in classifyOutcome; add atenet.router.egress.denied with a bounded reason label matching the branches (no_ca, no_client_cert, cert_invalid, not_signed, bad_identity, uid_mismatch, not_running, control_plane_unavailable); add a counter for direction-not-served refusals. Register all in the metric registry.\n6.2Emit the shutdown-outcome metric the drain code already asks forP1Stracked ×3drain.go:131-133, :151-154 (the TODOs) · docs/metrics/substrate.yaml:235-237 (recorded blind spot)\nBoth branches emit only a warn log. A force-stop cancels every in-flight ext_proc stream — including parked requests, whose clients then see a fail-closed 500 — the single most user-visible thing a rollout can do wrong, observable only by grepping logs of already-terminated pods. The egress case is worse: the manifest itself states that long-lived CONNECT tunnels keep downstream_cx_active above zero, so every rollout with active egress hits the incomplete-drain branch — a permanently-firing condition nobody can graph.\nDoAdd atenet.router.drain.completed{outcome=clean|dataplane_incomplete|extproc_force_stopped} and a drain-duration histogram (verified: Run blocks on drainDone before the deferred MeterProvider shutdown, so the datapoints flush); clear the blind-spot entry. Then give egress its own shorter drain window so the signal means something.\n6.3Add prometheus to the generated Corefile, and stop logging every DNS queryP1Strackedcmd/atenet/internal/dns/corefile.go:35-39 · docs/metrics/substrate.yaml:224-228\nThe Corefile appends exactly log, errors, health, ready, reload — no prometheus, and log is unconditional with no tie to the dns command's --log-level. Actor DNS is on the request path for every client and emits no telemetry at all: no QPS, no NXDOMAIN rate, no latency, and no signal when a Corefile reload silently fails.\nDoAppend prometheus :9153 in buildTemplate and add a PodMonitoring; remove log from the defaults or gate it on debug.\n6.4Build data-path dashboards and alertsP1Mtools/setup-gcp/dashboards/ — three JSONs, no parking reference · grep AlertPolicy tools/setup-gcp/ returns zero hits\nThe two signals that say the ingress gateway is saturating — parking.active and parking.rejected — are exported and never displayed. There is no alerting at all: no 5xx rate, no ext_proc error rate, no xDS staleness, no certificate expiry, despite the whole system depending on rotating bundles.\nDoA gateway dashboard (parking active/rejected, route duration by outcome and direction, downstream rq by code) plus alert policies for 5xx rate, parking.rejected \u0026gt; 0, and ext_proc failure rate. Depends on 1.2 and 3.5 to make the last two alertable at all.\n6.5Resolve the OTel Tracer once, and gate span attributes on IsRecording()P2Singress/ingress.go:94 · resumer.go:167 · vendor otel internal/global/trace.go:78 · sdk/trace/provider.go:155-163\nEach calls otel.Tracer(...) per request, which goes through the global delegating provider's exclusive lock and then the SDK provider's exclusive lock plus a map lookup on a freshly built scope — four acquisitions of two process-global mutexes per ingress request, on a server built for concurrent streams, for an object that never changes after startup. ateattr.ActorRefAttributes also allocates a fresh slice per call while the default root sampling ratio discards ~99%.\nDoResolve the tracer once after InitTracing and store it on the Server / Handler / Resumer; gate attribute construction on span.IsRecording() or move to SetAttributes after the sampling decision.\n6.6Raise the health-check interval and demote its per-tick INFOP2Scmd.go:59 (--health-interval defaults to 1s) · health.go:102, :212, :232\nEach tick runs a real API-server /version call and a real ListActors store query, and logs at Info. Verified nothing gates traffic on the result — /readyz and /healthz come from serverboot's Readiness, and the health report is consumed only by handleStatusz. So it is 1 QPS per replica to two dependencies plus 86,400 INFO lines per pod per day, purely for a dashboard.\nDoDefault to 10–15s; demote the per-tick line to Debug and keep failure logs at Error.\n7Test coverage7 7.1Cover Server.Process with a real gRPC stream testP1Mextproc_test.go:43 exercises processRequestHeaders directly · extproc.go:88-121 untested · coverage 69.1%\nThe stream loop owns EOF handling, the off-protocol branch, stream.Send failure, client-disconnect propagation, and behaviour under GracefulStop with a parked request in flight — and the drain sequence depends on that last one, so a regression surfaces only as reset client requests during a live rollout.\nDoA bufconn test over the real service asserting: clean EOF returns nil; a RequestBody message is rejected rather than answered; client cancellation propagates into the handler; a panicking handler kills only that stream; GracefulStop lets an in-flight parked stream finish.\n7.2Wire the drain verification into CI, and add a grace-period arithmetic testP1Mhack/verify-all.sh:24-28 iterates only hack/verify/*.sh · hack/verify-atenet-drain.sh appears in no workflow\nCorrection to one reviewer: unit coverage of the drain sequence is not absent — drain_test.go covers ordering, force-stop, dataplane failure, four envoyDrainer cases, and marker lifecycle. What is genuinely missing is CI execution of the live-cluster script and any test comparing the total budget to the grace period: 13s delay + 15s dataplane window + 20s derived ext_proc timeout = 48s against terminationGracePeriodSeconds: 60, so raising the park budget (or setting an explicit --drain-timeout) can silently push shutdown past it — --route-timeout is deliberately excluded from the derivation (config.go:198-208).\nDoRun hack/verify-atenet-drain.sh in a serialized kind job (its header explains why it must run alone), add an egress equivalent, and add a unit test asserting drain-delay + dataplane window + drainTimeout \u0026lt; a declared grace-period constant.\n7.3Add Go microbenchmarks for the ext_proc hot pathP1Mgrep -rn '^func Benchmark' cmd internal pkg returns nothing · benchmarking/automation/tests.yaml:405-434\nThe repo has zero Go benchmarks. The only performance signal is four 30-minute nighthawk runs requiring a live GKE cluster and a 50-worker pool, whose end-to-end variance is dominated by the ateapi RPC — so a change adding a per-request allocation or a lock merges green. Real per-request work at hand: a map per request in metadata.go:48, reflective structpb.NewStruct at ingress.go:163, the one process-global per-request mutex at record.go:55, and a full x509 chain Verify per CONNECT at egress.go:229. The fakes already exist.\nDoBenchmarkNewRequestMetadata, BenchmarkIngressHandleRequestHeaders, BenchmarkEgressVerifyActorCertificate, BenchmarkQueryRecorderAdd; run with -benchmem in CI and gate on allocs/op.\n7.4Cover the ingress port-parsing and parking-fairness edge casesP1Singress_test.go:136-297 (no authority-with-port case at all) · :343 · actorref_test.go (TestParseActorRef)\nTestParseActorRef covers :8080, a trailing dot, and :invalid:port but no out-of-range port; the parking-full test uses one actor against a 1-slot lot. These are exactly the behaviours the port-parsing and parking-fairness items above change, with nothing to lock the fixes in.\nDoCases asserting 400 for :0, :99999, :abc; a fairness test (actor B admitted while A holds its per-actor cap). A client-supplied x-ate-target-port case would pass today — that one is a regression guard, not a gap.\n7.5Benchmark the egress CONNECT pathP2Lbenchmarking/automation/tests.yaml — 22 entries, all locust or nighthawk-ingress; none drives egress\nEgress does strictly more per-request work than ingress — an x509 chain verify plus a synchronous GetActor — and the only acknowledgement of that cost is a code comment. Nobody knows how many CONNECTs/s an egress gateway sustains, or where GetActor starts hurting ateapi.\nDoAdd a type: nighthawk-egress runner and tests.yaml entries mirroring the ingress sweep (envoyCpu 2/4/8/16), driving actor-originated CONNECTs with real actor-identity client certs.\n7.6Test the QueryRecorder ring bufferP2Sno record_test.go in cmd/atenet/internal/router/extproc/ · record.go:64-88, :94-98\nUntested: the two distinct index paths in Get (while filling vs after wrap), the exactly-full boundary, redactPath's query-string strip, and the one per-request global mutex under contention. A silent ordering regression would have an operator read the last 100 requests in the wrong order during an incident.\nDoAdd the file covering partial / exactly-full / post-wrap ordering, redaction, and a concurrent Add/Get loop — make test already runs -race.\n7.7Add fuzz targets for the hand-rolled XFCC and authority parsersP2Sno func Fuzz anywhere · egress.go:287-410 (parseXFCCChain, splitXFCCUnquoted's quote/escape state machine, unquoteXFCC)\nA hand-rolled grammar feeding url.PathUnescape and pem.Decode directly in front of the egress authentication decision, whose table tests cover three malformed-header cases (egress_test.go:452-475) but never the quote/escape state machine itself. Honest scoping: no reachable panic was found — splitXFCCUnquoted does no indexing and unquoteXFCC guards its slice — and any surviving leaf still faces the full chain Verify plus IsCA/EKU/purpose checks. This is robustness hardening on pure, total functions, not a live bypass.\nDoFuzzParseXFCCChain (no panic; a successful parse round-trips the leaf DER), FuzzSplitXFCCUnquoted, FuzzUnquoteXFCC, FuzzParseActorRef, seeded from existing corpora, with a short -fuzztime CI run.\nHow this was actually produced Worth being specific, since the method is doing a lot of the work here.\nSeven agents read the same surface in parallel, each given a different question: the generated xDS, the ingress request path, the egress path and actor identity, the ext_proc server, the manifests and operational posture, the EgressPolicy API, and performance plus test coverage. Each returned structured findings with file-and-line citations.\nThen a second round, and this is the part that matters. Every finding went to a fresh agent whose only instruction was to knock it down — open the cited file, read around the line, and decide whether the code already handles it, whether the failure can actually occur in the deployed topology, or whether the Envoy or Kubernetes default it complains about is quietly fine. That round killed eight of 129 claims and corrected the severity or the reasoning on several more.\nThe refutations are the most useful output of the whole exercise, so here are four of them — the ones that don\u0026rsquo;t touch the findings I\u0026rsquo;m holding back:\nUnbounded ingress buffers and stream idle. One reviewer claimed per_connection_buffer_limit_bytes and stream_idle_timeout are unbounded on the ingress listeners. Envoy already defaults to 1 MiB and 5 minutes, and max_concurrent_streams defaults to 1024 (not 231). Kept only the genuine gaps: the header-phase request_timeout, the stream cap, and connection lifetime. A goroutine leaks per CONNECT while ateapi is wedged. Refuted — Envoy\u0026rsquo;s message_timeout reset cancels the server-side stream context. The deadline in 3.4 is still worth adding, for the serial Process loop, not for a leak. codes.Canceled is mismapped to a 500. Dropped entirely — the claimed trigger (ateapi Stop() during drain) is wrong; grpc-go surfaces that as Unavailable, and nothing in the repo returns Canceled. The egress ext-proc container should be a native sidecar. Refuted — the ext-proc container has its own readinessProbe, so the pod is not added to Service endpoints until it is listening. The \u0026ldquo;burst of failures on every pod start\u0026rdquo; does not happen. The pattern in all four is the same, and it is the characteristic failure mode of a fast reader, machine or otherwise: a real fact, cited correctly, attached to a consequence that doesn\u0026rsquo;t follow. Defaults are the richest source of it. \u0026ldquo;No max_concurrent_streams is set\u0026rdquo; is true and sounds alarming; the default is 1024, which is fine. \u0026ldquo;The Envoy container has no readiness probe\u0026rdquo; is true, and on a normal rollout the pod still leaves the Service on deletionTimestamp, so the thing you were worried about doesn\u0026rsquo;t happen and a different, smaller thing does.\nWhat I would and would not trust here An agent reading a codebase is very good at the class of bug that lives in the gap between two files. The CONNECT timeout above is the clearest example: two sibling manifests, one line different, and no human reviewer diffs manifests against each other. Same for the discarded List, the debug flags the benchmark strips, the probe that exists on one gateway and not its twin. These are all drift findings, and drift is exactly what tireless comparison is for.\nIt is much weaker at intent. It cannot tell you that a limit is unset because someone measured it and decided the default was right, and it will confidently produce a plausible failure mode for a default that has never failed. That is what the refutation round is patching, imperfectly — eight caught out of 129 is a floor on the error rate, not a ceiling.\nSo: treat the citations as reliable, the reasoning as a strong hypothesis, and the priorities as one outsider\u0026rsquo;s opinion with no production context behind it. The three findings I led with are the ones I am asserting rather than merely reporting: each was re-checked directly against the cited files, not against an agent\u0026rsquo;s summary of them. For the remaining forty-nine, the honest description is that a machine read the code carefully, a second machine argued with it, and I am publishing the transcript because the citations make it checkable.\nI would rather publish that with the corrections attached than a cleaner list that quietly dropped them.\n","permalink":"https://botengyao.github.io/posts/substrate-gateway-notes/","summary":"I pointed an AI agent at Agent Substrate\u0026rsquo;s networking layer and had it read the whole thing. This is what came back, after a second round of agents tried to knock it all down: 52 items about the ingress and egress gateways, and a note on which parts of the method I would and would not trust.","title":"Fifty-Two Notes on Agent Substrate's Gateway"},{"content":"Envoy\u0026rsquo;s TLS write path copies 16 KB for every 16 KB it sends — not occasionally, but for the life of a connection. This is what causes it and how the fix works.\nWhere SslSocket::doWrite Cost one 16 KB malloc + 16 KB memcpy + free, per 16 KB of TLS egress Trigger any HTTP response: small headers in front of a full-size body Fix write the contiguous front slice as one short TLS record Result −10.3% CPU on the affected path, copies per iteration 10 → 1 The line int rc = SSL_write(rawSsl(), write_buffer.linearize(bytes_to_write), bytes_to_write); SSL_write() needs contiguous memory. Envoy\u0026rsquo;s write buffer is a chain of slices. linearize() bridges them: if the front slice already holds enough it returns a pointer for free, otherwise it allocates a slice, copies into it, and returns that.\nIt reads like a fallback for an awkward case. On a proxied response it is the steady state.\nWhy it never recovers Two numbers matter, and they are the same number: the most Envoy hands to one SSL_write() is 16384 bytes, and Slice::default_slice_size_ — the size of each slice the read path reserves — is also 16384.\nThat is a reservation, not a limit. reservation.commit() only commits what was actually read, and other paths size a slice to whatever was written into it, so slices smaller than 16 KB are ordinary. But a body streamed from a fast upstream fills them right up, and that is the shape this section is about.\nThe short slice at the front is the response headers. ConnectionImpl::write() moves the codec\u0026rsquo;s output into the connection buffer wholesale, so a few hundred bytes of headers land as their own slice ahead of the body\u0026rsquo;s full-size ones. Call that size H.\nHere is what linearize(16384) does to that buffer:\nThe leftover is always exactly H, for any H. The copy takes H bytes off the front and leaves H bytes behind on the next slice, so the offset is conserved rather than worn down. The short slice in the \u0026ldquo;after\u0026rdquo; state is a different H bytes — the tail of slice 2, not the original headers.\nCall a buffer in this shape — a short slice of H bytes sitting ahead of full-size ones — a misaligned chain. The point is that linearize() does not clear it. It reproduces it, so every write from then on allocates and copies 16 KB.\nWhy the write buffer never re-aligns Step one: the buffer is a short header slice of H bytes ahead of full 16 KB slices. Step two: linearize copies H plus 16384 minus H into a fresh 16 KB slice and drains that much, which leaves the second slice holding exactly H bytes. Step three: writing and draining the fresh slice returns the buffer to the shape it had in step one, so the next write copies again. 1 — the write buffer: headers ahead of full-size body slices H 16384 16384 … H = the response headers, a few hundred bytes 2 — linearize(16384) copies H + (16384 − H) into a fresh slice, and drains that much 16384 — fresh copy H 16384 … slice 2 gave up (16384 − H) and keeps 16384 − (16384 − H) = H bytes 3 — the 16384 record is written and drained, so the fresh slice is gone H 16384 16384 … identical to step 1 — but this H is the tail of slice 2 repeats One 16 KB malloc + one 16 KB memcpy + one free, per 16 KB, for the life of the connection. linearize() reproduces the exact shape it was called to remove, so the next write faces the same buffer as the last one. The fix When a write would have to copy again right after the last one did, write just the contiguous front slice instead. That is one short TLS record, and from then on the buffer is aligned and the writes that follow cost no copy at all.\nHow one short record escapes the cycle Step one: copied last write and about to copy again. Step two: write only the 200-byte front slice as a short record and drain it. Step three: the buffer now starts on a slice boundary, so every following write is contiguous and free. 1 — copied last write, about to copy again 200 B 16 KB 16 KB … 2 — write just the front slice: one short record, no copy 200 B written and drained — this consumes the offset 3 — aligned 16 KB 16 KB 16 KB … Every write from here on is contiguous: no allocation, no memcpy. The cost isn't avoided — it's paid once instead of on every write. One short record buys permanent alignment. That is the trade the change makes. The rule, in three cases Each write asks for up to 16 KB. What happens depends on the buffer:\nthe buffer what it does front slice already holds enough write it directly — no copy needed front slice is short, and the last write had to copy, and the slice behind it can cover the next write write just the front slice: one short record, and the chain is aligned anything else copy into a new slice, as before The first case is not new — linearize() already returns that pointer untouched. The third is the old behaviour. Only the middle one is added:\nif (front_slice.len_ \u0026gt;= bytes_to_write) { // Already contiguous — linearize() would return this pointer without copying. return {front_slice.mem_, bytes_to_write, false}; } if (avoid_repeated_linearize \u0026amp;\u0026amp; linearized_last_write \u0026amp;\u0026amp; nextSliceCoversWrite(write_buffer, bytes_to_write)) { // Copied last time, about to copy again, and one short record fixes it. return {front_slice.mem_, front_slice.len_, false}; } return {write_buffer.linearize(bytes_to_write), bytes_to_write, true}; All three conditions in the middle case are load-bearing. Dropping either of the last two breaks it in a different way.\nWhy \u0026ldquo;the last write had to copy\u0026rdquo;, and not a size threshold A natural guard would be \u0026ldquo;only write the front slice directly if it is large enough to be worth a syscall\u0026rdquo;. It never fires. A threshold asks whether the front slice is big; the state it needs to act on is the one where the front slice is permanently small — held at H by the conservation shown above. The answer is no on exactly the writes where the fast path would help.\nWhy a size threshold can never fire A scale of front-slice size from 0 to 16 KB. The fast path opens only above a 4 KB threshold, but on a misaligned chain the front slice stays at 200 bytes, far below it, so the guard is never true. Front slice size — where each path applies copy path fast path — write the front slice directly 4 KB threshold 0 16 KB front slice on a misaligned chain: 200 B, permanently The fast path opens at 4 KB. A misaligned chain holds the front slice at H. The guard is never true. A threshold tests for a large front slice. A misaligned chain never has one, so no threshold value helps. Why the slice behind it has to be checked Writing the front slice only re-aligns the buffer if the slice behind it can satisfy the next write on its own:\nbool nextSliceCoversWrite(Buffer::Instance\u0026amp; write_buffer, uint64_t bytes_to_write) { const Buffer::RawSliceVector slices = write_buffer.getRawSlices(/*max_slices=*/2); return slices.size() == 2 \u0026amp;\u0026amp; slices[1].len_ \u0026gt;= bytes_to_write; } Without that check, a chain of equal sub-16 KB slices goes wrong — the slice behind the front one is short too, so nothing gets aligned.\nHow badly depends on what 16384 leaves over. A linearize consumes whole slices until it needs part of one more, and that slice keeps slice_size − (16384 mod slice_size) bytes. Pick a size just above an exact divisor of 16 KB and that remainder is almost nothing: 4097 is one byte over 16384/4, so it strands 4 bytes; 5462 is just over 16384/3 and strands 2. Those few bytes are all the short record has to write.\nThese are slice sizes fed to a unit test, not constants in Envoy — a slice holds whatever its producer wrote, so any size occurs. Taking 4097 as the worst case:\nWhy a short record achieves nothing on a chain of equal 4097-byte slices A linearize of 16 KB consumes three whole 4097-byte slices plus 4093 bytes of the fourth, leaving that slice holding 4 bytes. The short record then writes only those 4 bytes, but the slice behind it is still 4097 bytes and cannot satisfy the next 16 KB write, so the next write copies again. 1 — linearize(16 KB) takes 3 whole slices plus 4093 B of the 4th 4097 4097 4097 4093 of it 4097 … 4 B left over (not to scale) 2 — the short record writes those 4 bytes, and aligns nothing 4097 4097 4097 … a 4-byte TLS record — one syscall, ~24 B on the wire the slice behind it is 4097 — still short of 16 KB 3 — so the next write copies again, and the cycle repeats from step 1 4097 4097 4097 … Bytes copied: unchanged. Records and syscalls: doubled. The short record only helps if the slice behind the front one can cover the next write. Here it never can, so each copy is followed by a near-empty record that aligns nothing. Measured in that test, 64 slices of each size:\nslice size TLS records, without the check 4096 B 16 → 25 (1.56×) 4097 B 17 → 32 (1.88×) 5462 B 22 → 42 (1.90×) Nearly double the records and writev syscalls, for identical bytes copied. A 2-byte TLS record still costs about 24 bytes on the wire and one full syscall.\n4096 is in the table to show the problem is not about awkward remainders: it divides 16 KB exactly, so nothing is stranded, and the short record writes a whole 4096-byte slice instead. The slice behind it is still 4096, so the next write copies anyway. Any chain of equal sub-16 KB slices fails this way; the odd sizes just make it look worse. Real buffers take that shape whenever framing is interleaved with data — HTTP/1 chunked encoding, small HTTP/2 frames, addBufferFragment chains.\nThe check also means a write that drains the whole buffer is never split in two, since it can only pass when data remains behind the front slice.\nWhen a write doesn\u0026rsquo;t go through If the socket is full, SSL_write() writes nothing and returns SSL_ERROR_WANT_WRITE. Envoy sends the same write again when the socket is ready.\nA retry must not re-run the decision above. By then the buffer has already been linearized, so the code would see a contiguous front slice, conclude that no copy was needed, and clear linearized_last_write. When the retry lands and drains, the short remainder is back at the front with no history behind it — and it copies again. On a congested connection that repeats on every write, and the change never takes effect at all.\nSo the pending write is stored and replayed unchanged: same pointer, same length, and the same record of whether it came from a copy. BoringSSL requires the identical pointer in any case, since Envoy does not set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER.\nOne ordering detail: that pending write is cleared before write_buffer.drain(), not after. drain() can run callbacks that re-enter doWrite(), and a nested call must not find a write still pending that has already been sent.\nResults Eight alternating paired runs — baseline, fixed, baseline, fixed — so machine interference lands on both arms:\nPer-pair CPU change, misaligned buffer versus aligned control On the misaligned buffer all eight paired runs are faster, ranging from 8.2 to 12.6 percent, with a median of 10.3 percent. On the aligned control the eight pairs scatter above and below zero with a median of 0.2 percent, showing no consistent effect. Misaligned buffer Aligned buffer (control) 0% −5 −10 faster ↓ −12.6 median −10.3% · 8 of 8 pairs faster one direction, every time median +0.2% · 3 faster, 4 slower, 1 flat no consistent direction — as intended dashed line = median Each bar is one baseline-versus-fixed pair. The control panel is the point: where the buffer is not misaligned, the change does nothing. buffer median spread misaligned −10.3% 8 of 8 pairs faster, −8.2% to −12.6% aligned (control) +0.2% signs mixed — no regression The timing is macOS, where a memcpy is cheap relative to a syscall; the balance differs in production. The deterministic number is the copy counter: 10 copies per iteration → 1.\nRollout Guarded by envoy.reloadable_features.tls_avoid_repeated_linearize, default on. The flag is read when a TLS connection is created, so disabling it applies to new connections rather than to connections already open.\n","permalink":"https://botengyao.github.io/posts/envoy-tls-linearize/","summary":"Envoy\u0026rsquo;s TLS write path copies 16 KB for every 16 KB it sends, for the life of a connection. What causes it, and the change that fixes it.","title":"A Copy That Never Stops Copying"},{"content":"These are learning notes rather than settled advice. I am working through how trust boundaries should sit in a multi-tenant agent platform, and writing it down is how I find the holes. Some of it will turn out to be wrong, as the first half of this post demonstrates.\nI spent a while convinced of a sentence that turned out to be wrong. Working out why settled an architecture question I had been circling separately, about where the L7 proxy should run. The two turned out to be the same question, which is why they are in the same post.\nThe sentence The setup is a platform where agent sessions migrate between machines many times a day. A gateway sits in front of them and terminates the agent\u0026rsquo;s TLS connection so it can read the request, work out where that session currently lives, and forward it there. By any honest definition the gateway is a man in the middle. But it never holds a session\u0026rsquo;s private key, and on that basis I wrote something close to:\nThe gateway is trusted with availability, and trusted with nothing about identity.\nIt does not survive contact with what a terminating proxy actually does.\nA proxy that terminates TLS and re-originates is not passing your connection along. It is an endpoint on both sides: facing the agent it behaves exactly like the destination server, and facing the destination it behaves exactly like the agent. In between there is a moment where the request exists as plaintext in that process\u0026rsquo;s memory. From there the gateway can read everything, rewrite requests, drop or reorder them, fabricate responses and attribute them to the agent, and — unless client identity is separately signed and bound to each request — impersonate clients to the agents themselves.\nPassthrough versus authorized interception In passthrough, one TLS session runs from the agent to the origin and the proxy forwards bytes it cannot read. In authorized interception the proxy terminates one TLS session and originates a second, so the request exists as plaintext inside the proxy. Passthrough one TLS session, encrypted end to end agent proxy forwards opaque bytes origin Authorized interception agent proxy plaintext lives here endpoint of both sides origin TLS #1 TLS #2 two separate sessions — the proxy terminates one and originates the other The whole argument turns on the bottom row. Interception is not one connection being observed; it is two connections joined by a process that holds your request in the clear. So a per-session certificate prevents exactly one thing. The gateway cannot go somewhere else and claim to be tenant 42\u0026rsquo;s agent. That is worth having, but it is a much smaller claim than the one I made. I had conflated \u0026ldquo;cannot steal the agent\u0026rsquo;s credential\u0026rdquo;, which is true, with \u0026ldquo;cannot impersonate the agent at the application boundary\u0026rdquo;, which is not.\nThe version I would defend now:\nThe gateway is trusted for the confidentiality and integrity of every flow it terminates. Per-session identity keeps the agent\u0026rsquo;s long-lived private key out of the gateway, but it does not eliminate portable authority: any bearer token, API key, or replayable credential exposed in plaintext can still be stolen and used elsewhere. Compromise controls the traffic that traverses the gateway and any exportable credentials that traffic carries; it does not automatically confer session identities or signing authority that never reached it.\nThat second clause matters more than it looks, and I had to walk back a stronger version of this paragraph to get it right. My first instinct was that per-session credentials stop in-path authority from becoming portable authority. They do not. They prevent one specific form of in-path authority — possession of the agent\u0026rsquo;s private key — from becoming portable. Eliminating other portable authority requires sender-constrained credentials, request signing, or credential injection beyond the terminating proxy.\nThe point lands hardest on the egress path discussed later. A model provider API key is portable authority in its purest form: long-lived, bearer-shaped, valid from anywhere, and useful to an attacker long after the connection that leaked it is gone. If your gateway injects provider credentials — and mine does — then it handles precisely the class of secret that its per-session certificate scoping does nothing to protect. What remains true, and is still worth building for, is narrower: compromise does not hand over identities and signing keys belonging to sessions that never passed through that gateway.\nIf you want the stronger property, transport security cannot give it to you, because the gateway is an endpoint of that transport. You have to sign above it: the client signs method, target identity, body hash, nonce and expiry; the agent verifies that itself rather than taking the gateway\u0026rsquo;s word for it; the agent signs its response bound to the request ID. One detail is easy to miss, which is that replay protection has to survive checkpoint and restore, since restoring an older snapshot silently reopens a replay window you thought was closed. This costs latency and SDK surface, and deciding not to pay for it is reasonable. Describing your system as though you had paid for it is not.\nThree principals Part of why the bad sentence was easy to write is vagueness about whose identity is being checked. On the single hop from gateway to worker there are three principals, not one, and they are checked in different directions.\nPrincipal Its credential proves Checked by Gateway \u0026ldquo;I am an authorized platform hop\u0026rdquo; The worker, and the client downstream Agent \u0026ldquo;This instance represents session X\u0026rdquo; The gateway, and peer agents Client \u0026ldquo;This request came from principal Y\u0026rdquo; The agent, or an authorization service The worker validates the gateway\u0026rsquo;s identity; the gateway validates that the server it reached presents the expected session identity; the agent separately validates forwarded client authorization. Collapsing these produces sentences like \u0026ldquo;the worker validates the session certificate\u0026rdquo;, which has the direction backwards — the session lives on the worker, so it is the thing being authenticated, not the thing authenticating.\nThree principals and the direction each is checked in Client, gateway and worker sit in a line. The gateway checks that the worker presents the expected session identity. The worker checks that the gateway is an authorized hop. The agent on the worker checks the forwarded client authorization all the way back to the client. 2 are you really session X? client end user gateway terminates TLS worker agent, session X 1 are you an authorized hop? 3 did this request really come from user Y? one hop, three checks, three different directions Arrows point from the party doing the checking toward the party being checked. Checks 1 and 2 run in opposite directions across the same hop, which is what makes them easy to conflate. Check 3 is dashed because transport cannot provide it — it needs evidence signed by the client and verified by the agent. Most identity-confusion bugs I have seen are one of these rows quietly doing double duty, usually a host identity standing in for a workload identity, or a tenant ID read from an HTTP header standing in for one bound to the tunnel. While I am correcting myself: I had also described tampering with a checkpointed snapshot as a man-in-the-middle attack on memory. It is a good image but a bad threat model, because it hides the fact that signatures do not fix all of it. An attacker who swaps generation 50 of a snapshot for generation 42 has not forged anything. Every digest verifies, every signature checks out, and the agent wakes into an authentic version of its own past, one where a revoked credential still worked. Integrity was never violated; freshness was. Snapshot manifests therefore have to bind a monotonic generation, checked against an authority that is not the storage itself.\nRollback of a validly signed snapshot Snapshot generations 42, 46 and 50 all carry valid signatures. An attacker with write access serves generation 42 instead of the current generation 50, and the restore succeeds because every signature verifies. snapshot history in object storage gen 42 signed ✓ gen 46 signed ✓ gen 50 signed ✓ · current restore verify, then wake expected attacker with write access serves gen 42 instead the restore succeeds — nothing was forged, so every signature still verifies Signatures answer \"what is this?\" and cannot answer \"is this the newest?\". Closing the gap needs a monotonic generation checked against a committed-checkpoint authority that the storage layer does not control. Where this lands architecturally Everything above is about how much a TLS-terminating component is trusted. That answers a question which looks unrelated: where it should run. The connection is one sentence. If a gateway is fully trusted for the traffic it terminates, then the population whose traffic it terminates is the population that shares its fate.\nThe pattern worth borrowing is the one sidecar-less meshes converged on, though not for the reason it usually gets cited. People summarize it as one proxy per node instead of one per pod, which misses the part that matters. The important move is the split: a node-local L4 layer handling identity, capture and secure transport, and a separate L7 layer handling TLS termination, protocol policy and routing.\nThat split is about function, not about which binary you run. Envoy is entirely capable of the L4 role — TCP proxy and network filters, no HTTP connection manager, no termination of application TLS — and using it in both places keeps one data plane, one configuration language and one operational story across the platform, which is worth a lot in practice. Some meshes ship a smaller purpose-built proxy for the node instead, and the argument for that is a reduced attack surface on the component running on every host. It is a reasonable hardening choice, but a second-order one. What actually constrains the blast radius is what the node-local component is permitted to do: give Envoy an L4-only configuration, no interception keys, and no application TLS termination, and it can be configured with the same intended data-plane authority, although a purpose-built proxy may still have a smaller binary and extension attack surface. Configuration scope is the primary control. The binary is a secondary one, not a non-existent one.\nThe second detail is that the tunnel carries the application stream opaquely. Wrap the workload\u0026rsquo;s TCP stream in an HTTP/2 CONNECT tunnel protected by mTLS, which is the usual construction, and if the application inside is speaking HTTPS then that inner session stays encrypted straight through the node layer.\nWhich gives the correction I find myself repeating most often: mTLS is not MITM. Mesh mTLS protects the link and tells you which workload is talking. Enforcing anything about HTTP paths, tool names, model IDs or token budgets requires plaintext, which means something was explicitly authorized to terminate the application\u0026rsquo;s own TLS. Every \u0026ldquo;we have mTLS, so we have visibility into agent traffic\u0026rdquo; claim dies there.\nNow the placement argument. A bare-metal node already owns a large failure domain — compromise it and you get every sandbox on it, its local identities, its cached state. That is unavoidable; it is the physics of sharing a host. Authorized TLS termination owns a different failure domain: the plaintext of every tenant whose traffic it decrypts, plus the leaf certificates and private keys it holds to do so. Put a full L7 interception proxy on every node and those two domains merge. Every host compromise becomes a plaintext and key compromise for whichever tenants happened to be scheduled there — and placement is a scheduling decision, not a security decision, which means you have handed your trust boundaries to the scheduler, and it will redraw them on every rebalance.\nLayered agent platform: node-local L4 identity, sharded L7 termination Bare-metal hosts run microVM sandboxes and a small node-local L4 agent that binds tenant identity and builds an authenticated tunnel. Tunnels route to tenant-affine Envoy L7 shards, which are the only place application TLS is terminated and interception keys are held. bare-metal host microVM sandbox tenant 42 · agent · execution microVM sandbox node L4 agent identity · capture · tunnel no TLS termination · no keys authenticated tunnel L7 shard (tenant-affine) Envoy terminates app TLS policy / quota filters upstream TLS origination plaintext + keys live here 50–100 tenants, bounded certificate svc (SDS) policy service credential broker model providers tool servers Two blast radii, kept apart host compromise → that node's sandboxes and its L4 identity scope shard compromise → that shard's tenants' plaintext and keys a per-node L7 proxy would make each imply the other When the inner application connection remains TLS-encrypted, node-local components hold neither application plaintext nor interception keys. Termination is concentrated in tenant-affine shards whose blast radius is a placement decision rather than a scheduling accident. Ingress and egress are not the same problem I have been writing \u0026ldquo;the L7 tier\u0026rdquo; as though it were one thing. It is two, and almost everything that matters differs between them.\nEgress is an agent calling out — a model provider, a tool server, some URL it discovered at runtime. The destination is not yours, frequently is not known until the request arrives, and the client is code you do not trust. This is where interception actually happens: you terminate the agent\u0026rsquo;s TLS to a third-party hostname by presenting a certificate minted under a private CA that only your own sandbox images trust.\nIngress is the outside calling in — a user\u0026rsquo;s browser or application reaching an agent session that may currently be asleep. The destination is yours. You terminate TLS for your own domain using an ordinary public certificate, which is not interception at all, because you own the name.\nIngress and egress paths and their separate certificate authorities On ingress, an external user reaches an edge gateway holding a public certificate for your own domain, which wakes and reaches the agent session. On egress, the agent leaves through the node L4 layer to an egress shard that receives leaf certificates over SDS and terminates TLS for approved third-party hostnames. The signing key stays in the certificate service. Ingress — the outside calling in end user browser, app edge gateway public cert for your domain holds request, wakes session agent session yours, possibly asleep you own the name — this is ordinary termination, not interception Egress — the agent calling out agent untrusted code node L4 binds identity egress shard leaf certificates via SDS terminates TLS for approved hostnames third party not yours you own neither the name nor the destination — this is the interception path two paths, two certificate authorities, which must never be the same one The certificate on the ingress path proves you are who you say you are. The certificate on the egress path asserts you are somebody else, and is trusted only because your own sandboxes were told to trust it. Three things follow. The two paths need separate CA hierarchies, and merging them is a serious mistake. Your ingress certificate is a public one for a name you own and is revocable through ordinary means. Your egress interception root can mint a certificate for any hostname that your sandboxes will accept, which is a far more dangerous capability, and it must never be trusted by anything outside those sandboxes. Keep private interception roots well away from public trust roots.\nOnly egress is really MITM. Ingress is standard edge termination. That said, the correction at the top of this post applies to both — an ingress gateway that terminates TLS is still fully trusted for that traffic, and still cannot be described as trusted with nothing. What is egress-specific is the interception machinery: a private CA, certificates minted per destination hostname, and the requirement that the client trust a root you control.\nThe per-node question is really an egress question. Nobody puts ingress on every node; it lives behind a load balancer at the edge by construction. Egress is the one where placement is genuinely open, because that traffic originates on the node and could be terminated there. So everything below is about the egress tier.\nThe other asymmetries fall out of the same split. Wake-on-request belongs to ingress, along with the routing invariant that a lookup returns an agent identity and a placement epoch and the gateway forwards nothing until the peer proves both. Confused-deputy risk belongs to egress: an agent that asks the proxy to fetch a link-local or metadata address is exploiting the fact that the proxy has network reach the sandbox does not, which is why destination policy has to be applied to resolved addresses rather than hostnames, and re-applied after every resolution.\nOn the node, or at a remote endpoint This is about the egress tier specifically. The L4 layer stays on the node either way, and it may well be Envoy. The question is only where the terminating proxy lives — the one that decrypts application TLS and holds interception keys.\nBoth candidates for that role are multi-tenant. Both run one Envoy process serving many tenants, so shared-versus-dedicated is not the axis people think it is. What differs is who picks the roommates and what the roommates can lose together.\nOn the node At a remote endpoint Who shares the process Whoever the scheduler placed here Whoever you assigned Isolation tiers Coarse and scheduler-coupled Dedicated shards where needed Private keys On every host Only in the L7 tier Host compromise costs Plaintext and keys for local tenants Sandboxes only Added latency Effectively zero One network hop Failure domain Aligned with the node Cuts across nodes; needs replicas Scaling Proxy count tied to node count Traffic and compute scale separately Certificate working set Same cert cached on every node Cached once per shard Rollout Bad version reaches every host Canary and drain one shard Where interception keys live under each topology With L7 on every node, every host holds leaf certificates and their private keys, so the number of leaf-key holders equals the host count. With a sharded L7 tier, hosts hold no keys and only a small number of shards do. The CA signing key stays in the certificate service either way. L7 on every node host keys host keys host keys host keys … every host in the fleet leaf-key holders = host count tenant mix chosen by the scheduler L7 in shards host no keys host no keys host no keys host no keys L7 shard keys leaf-key holders = shard count tenant mix chosen by you Failure containment improves as you split into more, smaller units. Compromise containment gets worse — every extra unit is another leaf-key and plaintext holder. The two properties pull in opposite directions, which is why \"a failure only affects one node\" and \"a compromise only affects one node\" are not the same claim. The row that matters most is the first. With a per-node L7 proxy, the set of tenants sharing a decrypting process is decided by bin-packing and re-decided on every rebalance. You can still express isolation, to be fair — node pools, affinity rules, taints, dedicated hosts — but only at node granularity, and only by coupling a security boundary to the scheduler. Promising a regulated customer that their plaintext never shares a process with an untrusted tenant then means pinning them to their own node pool, which surrenders the density and locality that made per-node attractive in the first place. Remote sharding makes tenant mix an explicit and stable placement decision at a finer granularity than a host, which is what turns a dedicated isolation tier into something you can sell rather than something you configure the scheduler to approximate.\nCertificate economics push the same way and are less obvious. Interception needs a leaf per destination hostname, and per node, every host with a tenant calling a given host fetches and caches its own copy — the same certificate held N times across N hosts, with N times the issuance load and a poor hit rate, since each node\u0026rsquo;s cache is small and starts cold. Concentrate that traffic into a shard serving fifty to a hundred tenants and the certificate is fetched once and reused constantly. Upstream connection pools behave identically: agent egress converges on a handful of model providers and tool servers, so pooling wants concentration, and per-node placement fragments precisely what benefits from being pooled.\nI do not want to strawman the other side, because per-node has one real advantage. When a per-node proxy dies it takes egress down for the sandboxes on that node, but those sandboxes already shared that node\u0026rsquo;s fate, so the failure domain is aligned with one that already existed. A remote shard introduces a domain that cuts across nodes, and you pay for that with replicas, health checks and drain procedures. The reason I still come down on the other side is that the per-node arrangement buys its clean failure property by making every host a leaf-key holder and a plaintext holder, which is a cheap availability win traded for an expensive security loss.\nLatency deserves a number rather than a shrug. An extra same-zone hop costs a few hundred microseconds. Against a model provider call taking hundreds of milliseconds that is under one percent and genuinely does not matter; against a 2 ms internal RPC it is a quarter of the budget and matters a great deal. Agent egress, which is the traffic you actually want to inspect and meter, is firmly the first kind, so latency should not be allowed to overrule the security argument here. For latency-critical east-west traffic, keep it on the node and do not intercept it.\nPer-node L7 is the right answer when the premise of this post does not hold — when every workload shares one trust domain, when the traffic is latency-critical east-west rather than egress, or at the edge where there is no remote tier to route to. The test I would apply is whether the tenants sharing that process would be acceptable roommates no matter how the scheduler shuffles them. If that sentence makes you uncomfortable, you need the remote tier.\nFor scale, the shape I would start from is one lightweight L4 component per bare-metal host, ten to twenty L7 shards per failure domain, fifty to a hundred tenants per shard, at least two replicas each, and dedicated shards for outsized or regulated tenants. Those are a load-test hypothesis rather than a capacity claim, and I would size by handshake rate, connection count and stream duration rather than request rate, because agent traffic is dominated by long-lived streams that request-rate dashboards hide. The number is not the point anyway. The point is that the affected population is known before an incident rather than discovered during one.\nThe principle One rule generates most of the decisions above. Share mechanisms broadly, share authority narrowly.\nNode-local capture, transport machinery, stateless policy binaries and public trust roots should be everywhere; they are mechanism, and duplicating them costs you nothing. Interception CA keys, provider credentials, plaintext access and control-plane mutation rights should be held by as few things as possible; they are authority, and every additional holder is another way to lose them. The same rule explains why a single global interception root is the real blast radius in these systems regardless of how carefully you shard the proxies underneath, and why on-demand certificate selection reduces the working set of private keys in a process without ever removing them.\nIt also explains why the correction at the top of this post and the placement argument at the bottom are the same argument. Once you accept that a TLS-terminating proxy is fully trusted for the traffic it terminates, where it runs stops being a question about latency and becomes a question about how many tenants you are willing to put inside one trust boundary. For a platform with a thousand tenants the question was never whether one proxy could technically hold a thousand certificates and routes. It probably can. The question is whether one process should own that much authority and that much shared fate.\nThat is where my thinking currently sits. The part I am least sure of is the revocation story for a workload that migrates faster than any network policy can follow, which I suspect is the next thing I have wrong.\n","permalink":"https://botengyao.github.io/posts/trust-boundaries-agent-substrate/","summary":"Learning notes: a sentence I believed about TLS interception was wrong, and working out why settled a separate architecture question — whether the L7 proxy belongs on every node.","title":"Share Mechanisms Broadly, Share Authority Narrowly"},{"content":"A first-principles tour of Google\u0026rsquo;s open-source Agent Substrate project — the problem it solves, the architecture it builds on top of Kubernetes, and the design tensions it still has to resolve.\n(Agent Substrate is an early-stage open-source project and not an officially supported Google product. APIs are explicitly unstable. Everything below describes the code as of mid-2026.)\nThe problem: agents are asleep 99% of the time Think about what an AI coding agent actually does all day. A user asks it to build a todo app. It works hard for forty seconds — writes code, runs tests, starts a dev server. Then the user reads the result, thinks, has lunch, comes back tomorrow. The session is stateful — the agent\u0026rsquo;s filesystem, terminal history, even in-memory variables all matter — and it may live for weeks. But it computes for perhaps 1% of that lifetime.\nNow run a million of these on Kubernetes the obvious way: one pod per session. Three things break, in order:\nIdle pods still bill. RAM is reserved whether the agent is thinking or sleeping. At 99% idle, you\u0026rsquo;re paying a 100× tax. The control plane can\u0026rsquo;t hold the objects. etcd and the kube-apiserver are built for tens of thousands of long-lived resources, not millions of churning ones. Cold starts are too slow. Scheduling a pod takes seconds — scheduler convergence, image pull, container start. Fine for a service that runs for days; unacceptable for a workload that wakes up for two seconds. Neither of the two standard platforms fits this workload shape. Serverless is priced for burstiness but is stateless by design — it forgets your filesystem and your process memory between invocations. Kubernetes remembers everything but bills you for every idle second. Agents are the awkward middle: stateful like a pet, bursty like a function.\nAgent Substrate\u0026rsquo;s answer is one idea applied ruthlessly: turn the process into data.\nThe core idea: virtual memory, one level up Substrate decomposes an agent session (it says \u0026ldquo;actor\u0026rdquo; — nothing here requires literal AI) into three independent things:\nState — a snapshot of RAM plus filesystem, stored as files in object storage (GCS/S3); Identity — credentials that follow the session, not the machine; Compute — any one of a pool of pre-warmed, interchangeable \u0026ldquo;worker\u0026rdquo; pods. An idle actor is just a snapshot file: zero CPU, zero RAM, zero pods. When a request arrives, the actor is restored into any free worker in ~100ms (the project\u0026rsquo;s p95 target) using gVisor\u0026rsquo;s checkpoint/restore, or a Kata + Cloud Hypervisor micro-VM with demand-paged memory. When it goes idle again, it\u0026rsquo;s checkpointed back to storage and the worker is wiped and returned to the pool for the next tenant.\nIf this sounds familiar, it should — it\u0026rsquo;s an operating system pattern promoted one level up the stack:\nOS concept Substrate equivalent Physical page frame Warm worker pod Virtual page Actor Swap space Snapshot bucket in GCS/S3 Page fault Request to a suspended actor Kubernetes already virtualizes nodes into pods. Substrate virtualizes pods into actors, and the overcommit math is the same as virtual memory\u0026rsquo;s: the demo multiplexes ~250 stateful sessions across 8 physical pods, and the stated ambition is a billion actors per cluster with a thousand wakeups per second.\nNode, pod, actor: who holds whom It\u0026rsquo;s worth being precise about the containment hierarchy, because one detail here shapes the whole system.\nA node hosts many worker pods — a WorkerPool custom resource declares the warm capacity (\u0026ldquo;50 gVisor workers\u0026rdquo;), and a controller reconciles it into an ordinary Deployment. Kubernetes does what it\u0026rsquo;s good at: the scheduler places these empty shells, the kubelet starts and probes them. Each worker pod contains exactly one sandbox (a gVisor kernel or a micro-VM), and that sandbox hosts at most one actor at any moment. The assignment field in the data model is singular, not a list — this is a design invariant, not a temporary limitation.\nSo where does \u0026ldquo;250 actors on 8 pods\u0026rdquo; come from? The multiplexing is temporal, not spatial:\nActors time-share worker pods A Kubernetes node contains worker pods; each hosts at most one actor at a time. Suspended actors live as snapshots in object storage and are resumed into any idle worker. Kubernetes node worker = pre-warmed slot Worker pod 1 actor A (running) Worker pod 2 actor B (running) Worker pod 3 idle slot Snapshot storage GCS / S3 suspended actors C…Z millions, zero compute suspend resume One worker holds one actor at a time — the multiplexing happens across time, not within a pod Actors rotate through a small pool of capacity-one workers; everyone else sleeps in object storage. At any instant, at most 8 actors are actually running; the other 240+ are files. And the capacity-one invariant pays for itself twice:\nIsolation stays clean. An actor never has a same-pod neighbor — the only \u0026ldquo;neighbor\u0026rdquo; is the previous tenant of the slot, in time. The security review reduces to \u0026ldquo;was the worker wiped properly between tenants,\u0026rdquo; not \u0026ldquo;can co-residents attack each other.\u0026rdquo; Scheduling collapses. When every slot is identical and holds one tenant, there is nothing to pack and nothing to score. More on this below. Note what this inverts: in Kubernetes, \u0026ldquo;how many things run in a pod\u0026rdquo; is a packing question. In Substrate, the pod is a fixed page frame and the actor is the thing that flows — asking \u0026ldquo;how many actors per pod\u0026rdquo; is like asking how many virtual pages fit in a physical frame. Always one; the interesting number is how many rotate through it per day.\nThe architecture: four planes on top of Kubernetes Substrate is best understood as four planes, each with a deliberate relationship to the Kubernetes machinery underneath.\nThe control plane is a small gRPC server (ate-api-server) backed by Valkey/Redis rather than etcd. It owns the two high-frequency records — Actors and Workers — and orchestrates the resume/suspend workflows. Actors are not Kubernetes objects, and that\u0026rsquo;s the single most important scale decision in the project: a billion actors live as cheap rows in a purpose-built store, while etcd holds only the slow-moving configuration.\nThe node plane runs the sandboxes. A per-node supervisor (atelet, a DaemonSet) moves snapshots between object storage and the node; inside every worker pod, a small herder process (ateom) drives the sandbox runtime — runsc checkpoint/restore for gVisor, or a Cloud Hypervisor snapshot for micro-VMs. Notably, the kubelet is not involved: it starts the empty worker shell and probes it, but never learns what program is running inside. Substrate pulls actor images and restores memory itself.\nThe network plane (atenet) answers the question Kubernetes has no primitive for: route a request to a workload that might not be running, and wake it if so. A CoreDNS instance resolves every actor name — \u0026lt;actor\u0026gt;.\u0026lt;atespace\u0026gt;.actors.resources.substrate.ate.dev — to one router address, and an Envoy-based router does the real addressing at layer 7 per request. More on this below.\nThe identity plane issues credentials that survive migration. A session-identity broker mints short-lived JWTs and SPIFFE certificates bound to app/user/session — not to any pod — because an actor may run on worker 3 today and worker 7 tomorrow, and its permissions must not care.\nThe relationship with Kubernetes follows one consistent rule: each Substrate component replaces exactly one Kubernetes component that fails at actor granularity, and everything else is reused as-is.\nKubernetes component Why it fails at this granularity Substrate\u0026rsquo;s replacement etcd + apiserver Can\u0026rsquo;t store 10⁹ objects or absorb thousands of writes/sec ate-api-server backed by Valkey/Redis kube-scheduler Async convergence in seconds vs a 100ms budget In-memory filter + compare-and-swap claim Services / cluster DNS Can\u0026rsquo;t materialize per-actor endpoints; no concept of \u0026ldquo;wake on traffic\u0026rdquo; atenet: DNS mesh + Envoy router that resumes actors on demand kubelet / containerd No checkpoint/restore; no \u0026ldquo;swap the program inside a running pod\u0026rdquo; atelet (per-node) + ateom (in-pod sandbox herder) Workload identity Identity is pinned to the pod; actors migrate across pods daily Session-identity broker minting per-session JWTs and certs The slow-moving configuration — worker pools, actor templates, pinned sandbox binaries — stays in ordinary CRDs, which buys RBAC, auditing, and kubectl for free. In effect, Substrate treats Kubernetes as an IaaS: nodes, warm pod shells, and CRD storage in; scheduling, addressing, identity, and image distribution out. A cluster running Substrate remains a perfectly normal cluster for everything else.\nThe life of a request Here\u0026rsquo;s what happens when a request hits a sleeping actor:\nclient ──① DNS──▶ substrate CoreDNS (answers: the router\u0026#39;s IP, always) │ ├──② HTTP───▶ Envoy (atenet router) holds the request… │ │ │ ├──③ ext_proc──▶ Go process: parse Host, call ResumeActor │ │ │ │ │ ├──④ pick a free worker (CAS in Valkey) │ │ └──⑤ atelet pulls snapshot → ateom │ │ runs `runsc restore` (~100ms) │ ◀── worker IP ──────┘ │ │ └──◀─⑥ response──┴──▶ forwards the held request to workerIP:80 Two details carry the whole trick:\nDNS deliberately knows nothing. Every actor name resolves to the same router address. Names must be stable — they end up in user configs and caches — while locations change many times a day. So the name→location binding is deferred to request time, at layer 7, where the Host header identifies the actor. Materializing per-actor DNS records or Services would just smuggle the scale problem back into the control plane.\nThe router can hold a request. Envoy\u0026rsquo;s external-processing filter pauses the request while a companion Go process looks up the actor and, if it\u0026rsquo;s suspended, triggers the resume. To the client, a sleeping actor is indistinguishable from a slow first byte. This is why the proxy must be L7: only an HTTP-aware hop can park a request for a few hundred milliseconds and then forward it. The forward then goes directly to the worker\u0026rsquo;s pod IP — no Service, no Endpoints, nothing created in etcd per actor.\nScheduling in microseconds I expected to find a scheduler. There isn\u0026rsquo;t one — no queue, no scoring loop, no controller. \u0026ldquo;Scheduling\u0026rdquo; is four synchronous steps inside the resume call:\nFilter eligible worker pools (sandbox class + label selectors, all from local caches); List free workers from an in-memory cache; Shuffle randomly and take the first — no bin-packing, because every worker is an identical, capacity-one slot; Claim it with an optimistic-concurrency write to Valkey. Two concurrent resumes racing for the same worker? One compare-and-swap wins; the loser backs off ~10ms and picks another. The insight is that the warm pool dissolved the scheduling problem before it started: when slots are homogeneous and hold one tenant each, random choice is optimal and placement takes microseconds. The entire 100ms activation budget goes where it should — moving snapshot bytes.\nFailure handling follows the same minimalism. Multi-step workflows (resume, suspend) are sequences of idempotent steps with a \u0026ldquo;client-driven forward recovery\u0026rdquo; contract: if a step fails, the error goes back to the caller, and the retry fast-forwards through whatever already completed. There is no internal job queue to babysit — the workflow\u0026rsquo;s only durable state is the actor record itself.\nThis is the project\u0026rsquo;s design signature, visible at every layer: slow events go through Kubernetes; fast paths close over local memory and Redis. Key rotations, pool resizing, template changes — apiserver. Per-request routing, per-wakeup placement — never.\nThe honest tension list Substrate is very young, and the most interesting parts are the unresolved ones:\nStreaming versus transparent suspend. Agent traffic is SSE and WebSockets — long-lived connections. But an actor holding an open connection can\u0026rsquo;t be suspended transparently; connection state lives in the client and the proxy too. I suspect \u0026ldquo;connection draining as an explicit phase of suspend\u0026rdquo; ends up in the design, and \u0026ldquo;transparent\u0026rdquo; gets an asterisk.\nThe router still calls home per request. Every request — even to a running actor — currently costs a control-plane lookup for the actor\u0026rsquo;s location. The steady-state fix is an event-driven route cache in the router; until then, the control plane is back in the data path through the side door.\nIdentity granularity. Every service mesh assumes workload = pod. Substrate\u0026rsquo;s workload is finer than a pod and migrates across pods — which is why it grows its own session-identity broker instead of adopting a mesh. Expect this mismatch to be a recurring source of design work, especially for per-actor network policy, where CNI reprogramming can never keep up with sub-second multiplexing. Policy will have to follow identity, not IPs.\nThe density ceiling. One actor = one sandbox = one pod means per-node concurrency is capped by kubelet\u0026rsquo;s pod limit — a few hundred, not the thousands per host that Firecracker-style platforms achieve. Temporal multiplexing makes this mostly irrelevant for low-duty-cycle agents, but if high-activity workloads ever matter, decoupling sandboxes from pods would be the biggest surgery on the roadmap.\nWhy this project is worth watching Strip away the components and Substrate is a bet about the shape of agent workloads: stateful, long-lived, and almost always idle — a shape that neither serverless (stateless by design) nor vanilla Kubernetes (billed while idle) serves well. The response is an old idea executed at a new layer: paging, applied to entire sandboxed processes, on top of a platform it respects enough not to fork and distrusts enough to bypass on every hot path.\nWhether this codebase wins or not, I\u0026rsquo;d wager the pattern does. The gap between \u0026ldquo;pod\u0026rdquo; and \u0026ldquo;agent session\u0026rdquo; is real, and someone will fill it.\nThe project lives at github.com/agent-substrate/substrate (Apache 2.0), with a demo video, a counter demo you can run on kind, and a weekly community call. It is emphatically not production-ready — which is exactly the stage at which reading a codebase teaches the most.\n","permalink":"https://botengyao.github.io/posts/agent-substrate/","summary":"A first-principles tour of the open-source Agent Substrate project — the problem it solves, the four-plane architecture it builds on top of Kubernetes, how nodes, pods, and actors relate, and the design tensions it still has to resolve.","title":"Agent Substrate: Running a Million Sleeping Agents on Kubernetes"},{"content":"Welcome to my blog! This is the first post.\nThis site is built with Hugo using the PaperMod theme, and deployed on GitHub Pages.\nWhat to expect I\u0026rsquo;ll be writing about things I\u0026rsquo;m learning and building. Stay tuned.\nprint(\u0026#34;Hello, world!\u0026#34;) ","permalink":"https://botengyao.github.io/posts/hello-world/","summary":"First post — this blog is now live.","title":"Hello, World"},{"content":"Hi, I\u0026rsquo;m Boteng Yao.\nI\u0026rsquo;m an engineer interested in the systems underneath the software everyone sees — distributed systems, cloud infrastructure, and lately the machinery it takes to run AI agents at scale. This blog is where I think out loud: first-principles tours of open-source projects, notes on design tensions that don\u0026rsquo;t have clean answers yet, and write-ups of things I\u0026rsquo;m learning and building.\nIf I\u0026rsquo;ve done my job, each post should teach you one idea worth stealing.\nYou can find me on GitHub, or reach me at botengyao@gmail.com.\n","permalink":"https://botengyao.github.io/about/","summary":"About me","title":"About"}]