The AI Protocol Manager (AIPM) is Envoy’s filter for LLM API traffic. It parses the request body, runs small AI filters over it, can turn one vendor’s API into another’s, and reads token usage off the response. It is alpha and moving fast, so this page pins down how it works on Envoy main today (5d3b0dcaf4, October 26, 2026) and collects configs for the setups I keep getting asked about.

Every config here runs in an integration test: a real Envoy, mock OpenAI, Anthropic, Gemini and Vertex AI backends that refuse what the real APIs refuse, and an ext_proc server for the routing example. The configs and tests live next to this blog. The snippets below keep only the part that matters; the linked files are complete.

How it handles traffic

How a request moves through the AI Protocol ManagerA client request reaches Envoy. On a route declared as an AI endpoint, the AI Protocol Manager holds the headers, parses the body, and runs AI filters in order: schema validation, request info, and two transcoders that convert the client's API into OpenAI Chat and then into the backend's API. The body is serialized and passed to the router, which sends it to the provider. On the way back, token usage is read from the provider's bytes, and the transcoders convert the response back to the client's API.Clientany LLM SDKEnvoyAI Protocol Manageronly on routes that declare request:1 hold headers, parse the body (bad JSON → 400)2 schema_validation3 request_info → envoy.ai.request_info4 transcoder: client API → OpenAI Chat5 transcoder: OpenAI Chat → backend API6 serialize, release the headersuse only the AI filters you needext_proc, …routerresponse: token usage read from the provider's bytes → envoy.ai.token_usagethe transcoders convert the response (JSON or SSE) back to the client's APIProviderbackend API
A declared AI endpoint has its body parsed once and handed to the AI filters. The router and everything after it see a normal request in the backend's API.

A route opts in. Per-route config with request: makes a route an AI endpoint and names the API its clients speak. Every other route passes straight through: no buffering, no parsing.

The body is parsed before the headers move on. On an AI endpoint, AIPM holds the headers, offloads the body as it arrives, and parses it as JSON. Bad JSON gets a 400 right there. Then the AI filters run, in order, on the parsed document:

  • schema_validation checks the body against its API’s schema.
  • request_info publishes the model, the streaming flag, the token cap and the message and tool counts as typed metadata (envoy.ai.request_info), and the model as filter state (envoy.ai.model.request). With token_estimation it also estimates input tokens from the body size.
  • transcoder converts between APIs. Everything goes through one common form, OpenAI Chat Completions, so you run two instances: the first turns the client’s API into the common form, the second turns that into the backend’s API. Filters between them only ever see OpenAI Chat.

Then the document is written back into the body and the headers continue down the chain. With reserialize_body: DISABLE, the original bytes go out instead, which is what you want when the filters only read.

The response is handled twice. Token usage is read from the provider’s bytes, in the backend’s API, and published when the stream ends as typed metadata (envoy.ai.token_usage). The counts mean the same thing for every API: input includes cached tokens, output includes reasoning tokens. Separately, the transcoders convert the response, JSON or SSE, back into the client’s API.

The route decides the backend, and filter state can override it per request. The route’s response: names the backend’s API. A filter that runs earlier can override it, and more, for a single request:

  • envoy.ai.llm_protocol.request: the client’s API.
  • envoy.ai.llm_protocol.response: the backend’s API.
  • envoy.ai.binding.response: how the backend carries its API on HTTP. It is either a path pattern such as /v1beta/models/{model}:{generateContent|streamGenerateContent?alt=sse}, or a JSON binding that also edits the body.
  • envoy.ai.binding.request: the same for the client’s side.
  • envoy.ai.model.resolved: the model to send instead of the one the client asked for.

Anything that writes filter state can set these: set_filter_state, Lua, or ext_proc through headers.

It runs on the listener or on the cluster. In http_filters it runs once per request. In a cluster’s http_filters, or the router’s upstream_http_filters, it runs once per attempt, after Envoy picks the cluster and host, and still sees the request’s filter state.

Reading the output. AIPM publishes typed metadata only. An access log reads it with %TYPED_CEL(metadata.typed_filter_metadata['envoy.ai.token_usage'])%, and ext_proc receives it with metadata_options.forwarding_namespaces.typed.

The examples

Observe native APIs

Clients keep their SDKs and their own keys. Envoy validates and describes each request, counts tokens on each response, and forwards everything as is. Full config: observe.yaml.

Observing native LLM APIsThe client calls its provider through Envoy with its own SDK and key. The AI Protocol Manager validates the body, describes the request, and forwards the client's exact bytes. On the way back it reads token usage from the provider's response without changing it. Both records are published as typed metadata for the access log or ext_proc.Clientown SDK, own keyAI Protocol Managerschema_validation: a bad body gets its 400 hererequest_info: model, stream, token cap, countstoken_usage: read off the provider's responseProvidernative APIrequestresponsesame bytesuntouchedenvoy.ai.request_info · envoy.ai.token_usage→ access log, ext_proc
Observing only reads. The provider gets the client's bytes and key, and Envoy keeps a description of each request and its token count.
virtual_hosts:
- name: providers
  domains: ["*"]
  request_headers_to_remove: [accept-encoding]
  routes:
  - match: {path: /v1/messages}
    route: {cluster: anthropic, auto_host_rewrite: true}
    typed_per_filter_config:
      envoy.filters.http.ai_protocol_manager:
        "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
        request: {llm_protocol: ANTHROPIC_MESSAGES}
  # The same for /v1/chat/completions, /v1/responses and /v1beta/models/.
http_filters:
- name: envoy.filters.http.ai_protocol_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager
    request_handling: {reserialize_body: DISABLE}
    response_handling: {token_usage: {}}
    filters:
    - name: envoy.http.ai_filters.schema_validation
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.schema_validation.v3.SchemaValidation
    - name: envoy.http.ai_filters.request_info
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.request_info.v3.RequestInfo
- name: envoy.filters.http.router
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Why each part is there:

  • request: {llm_protocol: ...} makes the route an AI endpoint, and picks the schema to validate against and the usage format to read.
  • reserialize_body: DISABLE: these filters only read, so the provider gets the client’s exact bytes.
  • token_usage: {} publishes one usage record per response.
  • request_headers_to_remove: [accept-encoding]: usage can’t be read from a gzip body. Envoy then asks the provider for plain bytes.

A request through it logs:

{"model":"gpt-4o-mini","status":200,
 "request_info":{"input_llm_protocol":"OPENAI_CHAT_COMPLETIONS","message_count":1,"model":"gpt-4o-mini"},
 "token_usage":{"extraction_status":"COMPLETE","llm_protocol":"OPENAI_CHAT_COMPLETIONS","model":"gpt-4o-mini",
                "input_tokens":"11","output_tokens":"7","total_tokens":"18","provider_total_tokens":"18"}}

Things the tests ran into:

  • OpenAI streams carry usage only when the client sets stream_options.include_usage: true. Without it there is no record, and token_usage_missing counts it.
  • The Responses API has usage and request info, but no schema yet, so validation skips it.
  • Gemini names the model in the path, and request info reads it from there.
  • Malformed JSON, a body that breaks its schema, and an Anthropic request without max_tokens all get a 400 without reaching the provider.

Any client API, any backend

One filter chain serves every route. The base URL picks the backend and the path the client’s API, so each pairing is one route. Full config: gateway.yaml.

Base URLClientBackendNeeds, besides the two APIs
/claude/v1OpenAI SDKClaudebackend path
/gemini/v1OpenAI SDKGemininothing
/geminiClaude CodeGeminimodel
/openaiClaude CodeOpenAI-compatiblebackend path, model
/claudeGemini SDKClaudeclient path, backend path, model
/vertex/claude/v1OpenAI SDKClaude on Vertex AIbackend path with body edits
/vertex/gemini/v1OpenAI SDKGemini on Vertex AIbackend path
/v1Claude Code on Vertex AIGemini on Vertex AIbackend path, model

“Model” is envoy.ai.model.resolved, for clients that ask for a model the backend doesn’t have. “Path” is a binding, explained below.

http_filters:
- name: envoy.filters.http.set_filter_state     # per-route model overrides
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config
- name: envoy.filters.http.ai_protocol_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager
    request_handling: {}
    response_handling: {token_usage: {}}
    filters:
    - name: envoy.http.ai_filters.request_info
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.request_info.v3.RequestInfo
    - name: envoy.http.ai_filters.transcoder      # client's API <-> OpenAI Chat
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder
        request_handling: TO_IR
        response_handling: FROM_IR
    - name: envoy.http.ai_filters.transcoder      # OpenAI Chat <-> backend's API
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder
        request_handling: FROM_IR
        response_handling: TO_IR
- name: envoy.filters.http.router
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Credentials belong to the gateway. The virtual host strips whatever the client sent (authorization, x-api-key, x-goog-api-key), and each route adds the backend’s key. most_specific_header_mutations_wins: true on the route config makes the route’s additions run after the virtual host’s removals. The other order would strip the key the route just added.

OpenAI SDK to Claude

base_url="http://localhost:10000/claude/v1":

One request from the OpenAI SDK to ClaudeThe client sends an OpenAI Chat request to /claude/v1/chat/completions with its own key. Envoy drops the client's key and adds Claude's, the first transcoder keeps OpenAI Chat as is, the second writes Anthropic Messages, the binding sets the path to /v1/messages, and the host is rewritten to Claude's. Claude receives a Messages request with a top-level system field and max_tokens. Claude's answer, with its usage, is read for token usage and converted back into an OpenAI chat completion.requestClient sends OpenAI ChatPOST /claude/v1/chat/completionsauthorization: Bearer ‹client›{"model": "claude-sonnet-4-5","messages": [system, user]}Envoyvirtual host: drop the client's keyroute: add Claude's key and versiontranscoder 1: OpenAI Chat as istranscoder 2: → Anthropic Messagesbinding: path → /v1/messagesauto_host_rewrite: Claude's hostClaude receives MessagesPOST /v1/messageshost: api.anthropic.comx-api-key: ‹gateway's›anthropic-version: 2023-06-01{"model": "claude-sonnet-4-5","system": "Be brief.","messages": [user],"max_tokens": …}responseClient gets OpenAI Chat{"object": "chat.completion","choices": [{"message": …}],"usage": {…}}token_usage: 12 in, 6 outtranscoder 2: → OpenAI Chattranscoder 1: OpenAI Chat as isJSON and SSE alikeClaude answers Messages{"content": [text block],"usage": {"input_tokens": 12,"output_tokens": 6}}
The client and Claude each see only their own API. The key, path, host and body shape change inside Envoy.
- match: {path: /claude/v1/chat/completions}
  route: {cluster: anthropic, auto_host_rewrite: true}
  request_headers_to_add:
  - header: {key: x-api-key, value: "%ENVIRONMENT(ANTHROPIC_API_KEY)%"}
  - header: {key: anthropic-version, value: "2023-06-01"}
  typed_per_filter_config:
    envoy.filters.http.ai_protocol_manager:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
      request: {llm_protocol: OPENAI_CHAT_COMPLETIONS}
      response: {llm_protocol: ANTHROPIC_MESSAGES, binding: {uri_pattern: /v1/messages}}
  • response.llm_protocol tells the second transcoder to write Anthropic Messages, and tells token usage to read Anthropic’s format.
  • binding: {uri_pattern: /v1/messages}: the transcoder rewrites the body but leaves the path alone unless a binding names one. Without it, Claude gets a Messages body at /claude/v1/chat/completions and answers 404. Only Gemini has a built-in path, because its path carries the model.

OpenAI SDK to Gemini

response: {llm_protocol: GEMINI_GENERATE_CONTENT}

This needs no binding: Gemini’s built-in one renders /v1beta/models/gemini-2.5-flash:generateContent, or :streamGenerateContent?alt=sse for stream: true, and removes model and stream from the body. Its key goes in x-goog-api-key.

Claude Code to Gemini, or to any OpenAI-compatible backend

ANTHROPIC_BASE_URL=http://localhost:10000/gemini:

- match: {path: /gemini/v1/messages}
  route: {cluster: gemini, auto_host_rewrite: true}
  request_headers_to_add:
  - header: {key: x-goog-api-key, value: "%ENVIRONMENT(GEMINI_API_KEY)%"}
  typed_per_filter_config:
    envoy.filters.http.set_filter_state:
      "@type": type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config
      on_request_headers:
      - object_key: envoy.ai.model.resolved
        format_string: {text_format_source: {inline_string: gemini-2.5-flash}}
    envoy.filters.http.ai_protocol_manager:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
      request: {llm_protocol: ANTHROPIC_MESSAGES}
      response: {llm_protocol: GEMINI_GENERATE_CONTENT}
  • envoy.ai.model.resolved: Claude Code asks for Claude models. This replaces the model wherever the backend carries it (here the path), and the response still names the model the client asked for.

For an OpenAI-compatible backend (OpenAI, vLLM, …), the route at /openai/v1/messages sends gpt-4o-mini with response: {llm_protocol: OPENAI_CHAT_COMPLETIONS, binding: {uri_pattern: /v1/chat/completions}}. Claude Code’s tools come out as OpenAI functions, and its SSE comes back as Anthropic events.

Gemini SDK to Claude

http_options={"base_url": "http://localhost:10000/claude"}, so the path is /claude/v1beta/models/gemini-2.5-flash:generateContent:

request:
  llm_protocol: GEMINI_GENERATE_CONTENT
  binding: {uri_pattern: "/claude/v1beta/models/{model}:{generateContent|streamGenerateContent?alt=sse}"}
response: {llm_protocol: ANTHROPIC_MESSAGES, binding: {uri_pattern: /v1/messages}}
  • request.binding: Gemini puts the model and the streaming mode in the path. Envoy knows Gemini’s own path and Vertex AI’s, but not your prefix, so this pattern says where {model} sits. {generateContent|streamGenerateContent?alt=sse} reads stream. maxOutputTokens becomes max_tokens.
  • envoy.ai.model.resolved: claude-sonnet-4-5, set the same way as for Claude Code below: the SDK asks for a Gemini model.

Vertex AI

OpenAI clients to Claude on Vertex AI:

- match: {path: /vertex/claude/v1/chat/completions}
  route: {cluster: vertex, auto_host_rewrite: true}
  request_headers_to_add:
  - header: {key: authorization, value: "Bearer %ENVIRONMENT(VERTEX_ACCESS_TOKEN)%"}
  typed_per_filter_config:
    envoy.filters.http.ai_protocol_manager:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
      request: {llm_protocol: OPENAI_CHAT_COMPLETIONS}
      response:
        llm_protocol: ANTHROPIC_MESSAGES
        binding:
          uri_pattern: "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/{model}:{rawPredict|streamRawPredict}"
          mutations:
          - {body_field: model}
          - {body_field: anthropic_version, value: vertex-2023-10-16, keep_existing: true}
How a binding renders a Vertex AI requestThe second transcoder writes an Anthropic Messages body with model and stream. The binding's pattern takes the model from the body into the path, and stream picks rawPredict or streamRawPredict. The first mutation removes model from the body, and the second adds anthropic_version. Vertex AI receives a POST to the streamRawPredict path for the model, with anthropic_version, stream, messages and max_tokens in the body.1 transcoder 2 writes Anthropic Messages{"model": "claude-sonnet-4-5@20250929", "stream": true, "messages": […], "max_tokens": …}2 the binding renders the path and edits the bodyuri_pattern: /v1/projects/…/publishers/anthropic/models/{model}:{rawPredict|streamRawPredict}mutations: {body_field: model}, {body_field: anthropic_version, value: vertex-2023-10-16}{model} takes the body's model, and the first mutation removes it from the body. stream picks the method and stays.3 Vertex AI receivesPOST /v1/projects/…/publishers/anthropic/models/claude-sonnet-4-5@20250929:streamRawPredict{"anthropic_version": "vertex-2023-10-16", "stream": true, "messages": […], "max_tokens": …}
A binding maps between the body and the path. On the client side it works the other way: it reads the model and the method out of the path, as in the Claude Code on Vertex AI route below.
  • Claude on Vertex AI speaks Anthropic Messages, but it takes the model in the path and needs anthropic_version in the body. The two mutations remove model and add the version. The method follows stream.
  • Gemini on Vertex AI needs only the pattern (.../publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}). Gemini’s own edits still apply.
  • The token comes from gcloud auth print-access-token here. In production, let the gcp_authn filter fetch it.

Claude Code on Vertex AI (CLAUDE_CODE_USE_VERTEX=1, ANTHROPIC_VERTEX_BASE_URL=http://localhost:10000/v1) to Gemini, in the project and region the client named:

- match: {prefix: /v1/projects/}
  route: {cluster: vertex, auto_host_rewrite: true}
  typed_per_filter_config:
    envoy.filters.http.set_filter_state:     # model.resolved: gemini-2.5-flash
      ...
    envoy.filters.http.ai_protocol_manager:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
      request: {llm_protocol: ANTHROPIC_MESSAGES}
      response:
        llm_protocol: GEMINI_GENERATE_CONTENT
        binding: {uri_pattern: "/{version}/projects/{project}/locations/{location}/publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}"}
  • No request binding is needed: Envoy knows rawPredict and streamRawPredict. It reads the model, the streaming mode, {version}, {project} and {location} from the client’s path, and takes anthropic_version out of the body.
  • The backend pattern renders {model} from the body (after model.resolved), and every other variable from what it read. Access logs can show those values with %FILTER_STATE(envoy.ai.binding.read:FIELD:project)%.

I left schema validation out of this chain. It doesn’t read bindings yet, so it rejects a Vertex AI body whose model is in the path. The second transcoder validates against the backend’s schema before sending anyway.

Retries

route:
  cluster: anthropic
  retry_policy: {retry_on: 5xx, num_retries: 1}
A retry with transcoding on the listenerThe AI Protocol Manager on the listener converts the OpenAI Chat request into Anthropic Messages once, before the router. The router buffers the body. Attempt 1 to the anthropic cluster gets a 503. Attempt 2 sends the same bytes, gets a 200, and is the attempt that reports token usage.ClientOpenAI SDKAIPM on the listenerOpenAI Chat → Messagesonce, before the routerrouterbuffers the bodyfor a retryattempt 1 → anthropicPOST /v1/messages✗ 503attempt 2 → anthropicthe same bytes✓ 200, token usage from here
Transcoded once, sent twice. Only the attempt that answered reports token usage.

With AIPM on the listener, transcoding happens once, before the router. The router buffers the transcoded body and sends the same bytes on the retry, and token usage comes from the attempt that answered. There is nothing AI-specific to configure. Retrying on a backend with a different API needs transcoding on each attempt, which is the next section.

Several backends: split, failover, by model

Full config: multi-backend.yaml.

Transcoding split between the listener and each attemptThe AI Protocol Manager on the listener converts the client's API to OpenAI Chat once. The router picks a cluster per attempt: a weighted split, a composite cluster, or a priority group. On each attempt, an AI Protocol Manager on the upstream side converts OpenAI Chat into the API of the backend it reached. In the example, attempt 1 goes to Claude on Vertex AI and fails with 503, and attempt 2 goes to Gemini on Vertex AI and succeeds.Clientany APIAIPM on the listenerclient API → OpenAI Chatonce per requestroutersplit, composite,priority group, retryattempt 1 → vertex_claudeAIPM: OpenAI Chat → Claude.../anthropic/models/…:rawPredict✗ 503attempt 2 → vertex_geminiAIPM: OpenAI Chat → Gemini.../google/models/…:generateContent✓ 200, back to the client's API
Transcode the client's side once, and the backend's side on every attempt. A retry that lands on another API is then converted for that API.

Split the transcoding in two. The AIPM on the listener converts the client’s API into OpenAI Chat. A second AIPM runs on each attempt, after Envoy has picked the cluster and host, and converts OpenAI Chat into whatever that backend speaks. Each endpoint says what it speaks in its metadata:

- endpoint: {address: {socket_address: {address: us-east5-aiplatform.googleapis.com, port_value: 443}}}
  metadata:
    filter_metadata:
      envoy.ai:
        api: GEMINI_GENERATE_CONTENT
        model: gemini-2.5-flash
        binding: "/v1/projects/my-project/locations/us-east5/publishers/google/models/{model}:{generateContent|streamGenerateContent?alt=sse}"

The router’s upstream filters turn that metadata into the AIPM’s instructions on every attempt:

upstream_http_filters:          # every cluster without filters of its own
- name: envoy.filters.http.header_mutation
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.header_mutation.v3.HeaderMutation
    mutations:
      request_mutations:
      - append: {header: {key: x-ai-api, value: "%UPSTREAM_METADATA(envoy.ai:api)%"}, append_action: OVERWRITE_IF_EXISTS_OR_ADD}
      - append: {header: {key: x-ai-binding, value: "%UPSTREAM_METADATA(envoy.ai:binding)%"}, append_action: OVERWRITE_IF_EXISTS_OR_ADD}
      - append: {header: {key: x-ai-model, value: "%UPSTREAM_METADATA(envoy.ai:model)%"}, append_action: OVERWRITE_IF_EXISTS_OR_ADD}
- name: envoy.filters.http.lua
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
    default_source_code:
      inline_string: |
        function envoy_on_request(handle)
          local state = handle:streamInfo():filterState()
          for header, key in pairs({["x-ai-api"] = "envoy.ai.llm_protocol.response",
                                    ["x-ai-binding"] = "envoy.ai.binding.response",
                                    ["x-ai-model"] = "envoy.ai.model.resolved"}) do
            state:set(key, key, handle:headers():get(header))
            handle:headers():remove(header)
          end
        end
- name: envoy.filters.http.ai_protocol_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager
    request_handling: {}
    response_handling: {token_usage: {}}
    filters:
    - name: envoy.http.ai_filters.transcoder
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder
        request_handling: FROM_IR
        response_handling: TO_IR
- name: envoy.filters.http.upstream_codec
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec
Three values relayed from the picked host to the AI Protocol ManagerOn each attempt, the picked host's envoy.ai metadata holds the backend's API, its binding and its model. header_mutation copies them into the x-ai-api, x-ai-binding and x-ai-model headers using UPSTREAM_METADATA. Lua moves each header into filter state: envoy.ai.llm_protocol.response, envoy.ai.binding.response and envoy.ai.model.resolved. The AI Protocol Manager then writes that API, renders that path and sends that model.the backend's APIhow it carries it on HTTPthe model to sendpicked hostenvoy.ai metadataapi: GEMINI_GENERATE_CONTENTbinding: "/v1/projects/…"model: gemini-2.5-flashheader_mutation%UPSTREAM_METADATA%x-ai-apix-ai-bindingx-ai-modelLuaheaders → filter stateenvoy.ai.llm_protocol.responseenvoy.ai.binding.responseenvoy.ai.model.resolvedAI Protocol Manageron each attemptwrites this APIrenders this pathsends this model
On every attempt, three values relay from the host Envoy picked to the AI Protocol Manager. header_mutation can read the host but not set filter state, and Lua the other way round, so they hand off through headers that Lua then removes.
  • Endpoint metadata, not cluster config: a composite cluster runs its own upstream filters for every sub-cluster, so the only thing that identifies the backend on an attempt is the host that was picked. %UPSTREAM_METADATA% reads that host’s metadata.
  • header_mutation, then Lua: set_filter_state can’t run on a cluster yet, and Lua can’t read the host’s metadata. header_mutation can format %UPSTREAM_METADATA%, and Lua can set filter state, so they pass the values along in three headers.
  • Every endpoint sets all three values. The filter state survives across attempts, so a value one endpoint left out would be the previous attempt’s.
  • Token usage on this AIPM reads the backend that answered. Only the winning attempt publishes.

Then the routes. All of them declare request: {llm_protocol: OPENAI_CHAT_COMPLETIONS} (or the client’s API) for both AIPMs.

Split half the traffic between Claude and Gemini:

route:
  weighted_clusters:
    clusters: [{name: vertex_claude, weight: 50}, {name: vertex_gemini, weight: 50}]
  auto_host_rewrite: true

Fail over to Gemini when Claude fails, using a composite cluster (the first attempt uses the first cluster, the retry uses the next):

route: {cluster: vertex_failover, auto_host_rewrite: true, retry_policy: {retry_on: 5xx, num_retries: 1}}
# cluster
- name: vertex_failover
  lb_policy: CLUSTER_PROVIDED
  cluster_type:
    name: envoy.clusters.composite
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.clusters.composite.v3.ClusterConfig
      clusters: [{name: vertex_claude}, {name: vertex_gemini}]

In the test, attempt 1 sends rawPredict with anthropic_version and gets a 503. Attempt 2 sends generateContent with contents. The OpenAI client gets a normal chat completion, and Claude Code, on its own route, gets Anthropic SSE.

Fail over across providers with their own keys, Anthropic first, then Gemini, using the priority group cluster specifier:

route:
  inline_cluster_specifier_plugin:
    extension:
      name: envoy.router.cluster_specifier_plugin.priority_group
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.router.cluster_specifiers.priority_group.v3.PriorityGroupClusterSpecifier
        priority_groups:
        - {name: primary, clusters: [{cluster_name: anthropic, weight: 1}]}
        - {name: fallback, clusters: [{cluster_name: gemini, weight: 1}]}
  auto_host_rewrite: true
  retry_policy: {retry_on: 5xx, num_retries: 1, refresh_cluster_on_retry: true}
  • refresh_cluster_on_retry makes the router ask the specifier again on the retry, and the attempt then runs the new cluster’s filters.
  • Each cluster brings its own key. A cluster’s own http_filters replace the router’s, so it lists a header_mutation that adds its key, followed by the same three filters. It also removes the other provider’s key: upstream filters edit the request in place, so a retry starts with whatever the previous attempt added. The test checks that the Gemini attempt carries no x-api-key.

Route by model: routes can match the model request_info reads from the body:

- match:
    path: /v1/chat/completions
    filter_state: [{key: envoy.ai.model.request, string_match: {prefix: gemini-}}]
  route: {cluster: vertex_gemini, auto_host_rewrite: true}
# ... claude- to vertex_claude, then:
- match: {path: /v1/chat/completions}
  direct_response: {status: 404, body: {inline_string: "no backend serves this model"}}

http_filters:
- name: envoy.filters.http.ai_protocol_manager    # request_info, client-side transcoder
  ...
- name: envoy.filters.http.set_filter_state
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config
    clear_route_cache: true
Routing by the model in the bodyWhen the headers arrive the model is unknown, so the catch-all route matches; it declares request, so the body is parsed. The AI Protocol Manager's request_info reads the model into envoy.ai.model.request. set_filter_state with clear_route_cache drops the picked route, and the routes are matched again: a gemini model goes to vertex_gemini, a claude model to vertex_claude, and anything else gets a 404 without calling a provider.1 headers arriveno model yet, so the catch-allmatches; it declares request:2 AI Protocol Managerrequest_info reads the model intoenvoy.ai.model.request3 set_filter_stateclear_route_cache: truedrops the route Envoy picked4 routes match againgemini-* → vertex_geminiGemini on Vertex AIclaude-* → vertex_claudeClaude on Vertex AIanything else → 404no provider is called
Envoy picks a route before the body arrives. Clearing the route cache after AIPM has read the model makes it pick again, and this time the model can decide.
  • The catch-all route must also declare request:: Envoy picks a route before the body is read, and only an AI endpoint gets its body parsed.
  • clear_route_cache: true, after AIPM, makes Envoy pick the route again once the model is known. A model no backend serves gets the 404 without calling any provider.

Egress through a forward proxy

One route, no per-provider config. The request’s Host picks the destination: clients set HTTP_PROXY=http://localhost:10000 and use an http:// base URL, and Envoy originates TLS. Full config: egress.yaml.

Egress through a dynamic forward proxyA client with HTTP_PROXY set sends plain HTTP to Envoy. The AI Protocol Manager on the listener detects the API from the path, a header or the body, then validates and describes the request. The dynamic forward proxy resolves the request's Host. The AI Protocol Manager on the cluster reads token usage on each attempt, detecting the API from the response. The cluster's TLS takes its SNI and SAN check from the Host, so one route reaches api.openai.com, api.anthropic.com or generativelanguage.googleapis.com.ClientHTTP_PROXY set,http:// base URLEnvoyAIPMdetects the API frompath, header or body,validates, describesDFPresolves therequest's HostAIPM on the clustertoken usage oneach attempt, APIfrom the responsecluster TLS: auto_sni and auto_san_validation follow the Hostthe Hostapi.openai.comapi.anthropic.comgenerativelanguage.googleapis.com
One route for every provider. The Host decides where the request goes, the listener's AIPM works out which API it is, and the cluster's AIPM reads usage from whichever provider answered.
routes:
- match: {prefix: /}
  route: {cluster: egress}
  request_headers_to_remove: [accept-encoding]
  typed_per_filter_config:
    envoy.filters.http.ai_protocol_manager:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManagerPerRoute
      request: {}
# listener: AIPM (schema_validation, request_info, DISABLE) -> dynamic_forward_proxy -> router
# cluster `egress` (dynamic forward proxy), upstream filters:
http_filters:
- name: envoy.filters.http.ai_protocol_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager
    response_handling: {token_usage: {}}
- name: envoy.filters.http.upstream_codec
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec
  • request: {} declares an AI endpoint without naming the API. Schema validation detects it from the path (/chat/completions, /v1/messages, :generateContent), then the anthropic-version header, then the body, and validates against it. Request info reads with the API it found. Every body on the route is parsed strictly, so keep this route for LLM traffic.
  • Token usage on the cluster detects the response’s API from its shape. It runs on each attempt, and only the attempt that answered publishes.
  • auto_sni and auto_san_validation on the cluster use the Host, so TLS names and checks whichever provider the request went to.

Routing decisions outside Envoy

A decision point (PDP) behind ext_proc picks the backend for each request and charges each tenant for the tokens it used. Full config: pdp.yaml, and the PDP is about 150 lines of Go.

Model routing with an external decision pointThe request flows from the client through the AI Protocol Manager on the listener, which describes it, to ext_proc, where the PDP decides the backend. set_filter_state turns the PDP's headers into filter state, and the router with the dynamic forward proxy sends the request to the cluster, where a second AI Protocol Manager transcodes it and reads token usage from the provider's response. The usage record goes back to ext_proc on trailers, and the PDP charges the tenant.Clientmodel: smartAIPMdescribeext_procPDP decidesheaders →filter staterouter+ DFPAIPMon the clusterClaude orGeminisees the model and a token estimate, not the prompttoken usage on the response trailers → the PDP charges the tenantover budget: the next "smart" request goes to Gemini
The PDP decides from a description of the request and settles from the usage Envoy reports. It never handles the prompt or the response body.
http_filters:
# schema_validation, request_info with token_estimation, reserialize_body: DISABLE
- name: envoy.filters.http.ai_protocol_manager
  ...
- name: envoy.filters.http.ext_proc
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
    grpc_service: {envoy_grpc: {cluster_name: pdp}}
    processing_mode:
      request_header_mode: SEND
      response_header_mode: SKIP
      response_trailer_mode: SEND
    metadata_options:
      forwarding_namespaces:
        typed: [envoy.ai.request_info, envoy.ai.token_usage]
- name: envoy.filters.http.set_filter_state
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.set_filter_state.v3.Config
    on_request_headers:
    - object_key: envoy.ai.llm_protocol.response
      format_string: {text_format_source: {inline_string: "%REQ(x-ai-api)%"}}
    - object_key: envoy.ai.binding.response
      format_string: {text_format_source: {inline_string: "%REQ(x-ai-binding)%"}}
    - object_key: envoy.ai.model.resolved
      format_string: {text_format_source: {inline_string: "%REQ(x-ai-model)%"}}
# the route sets host_rewrite_header: x-ai-host
- name: envoy.filters.http.dynamic_forward_proxy
  ...
- name: envoy.filters.http.router
  ...
# cluster `llm` (dynamic forward proxy), upstream filters:
- name: envoy.filters.http.ai_protocol_manager
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ai_protocol_manager.v3.AiProtocolManager
    request_handling: {}
    response_handling:
      token_usage: {usage_signal: SYNTHESIZE_TRAILERS}
    filters:
    - name: envoy.http.ai_filters.transcoder
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.http.ai_filters.transcoder.v3.Transcoder
        request_handling: FROM_IR
        response_handling: TO_IR
- name: envoy.filters.http.upstream_codec
  ...
  • ext_proc comes after the first AIPM. request_info is published before the headers move on, so the PDP gets the model and the token estimate with the request headers. The body is never sent to it.
  • The PDP answers with headers, and set_filter_state turns them into instructions, since ext_proc can’t write filter state. The PDP overwrites all four x-ai-* headers on every request, and the route strips them before the request goes upstream. Otherwise a client could send x-ai-host and pick the host itself; the test sends one and checks it’s ignored.
  • One transcoder instance on the cluster is enough: these clients already speak OpenAI Chat, the common form.
  • usage_signal: SYNTHESIZE_TRAILERS with response_trailer_mode: SEND delivers the usage record at the end of the response, without streaming the body to the PDP.

In the test, the PDP has a 10-token budget. The first smart request goes to Claude and costs 18 tokens. The next smart request goes to Gemini, and an unknown model gets a 400 from the PDP before any provider is called.

Gaps on main

These are the things that didn’t work as I first wrote them. All of them are on main as of this writing.

  • Schema validation doesn’t read bindings. A Vertex AI body names its model in the path, and validation rejects it with missing required field: model. Leave validation out of chains that take Vertex AI clients.
  • set_filter_state doesn’t run on a cluster. It isn’t registered as an upstream filter, so per-attempt instructions go through header_mutation and Lua.
  • auto_sni doesn’t follow auto_host_rewrite. The SNI is taken from the Host before the rewrite applies, so the TLS check fails. Set sni and the SAN check on each cluster, as the configs above do.
  • Upstream filters edit the request in place, and a retry starts from the previous attempt’s edits. That’s why clusters remove other providers’ keys. It also breaks the client-side transcoder when it runs on the cluster: on a retry it reads the path the first attempt rewrote, and a Gemini SDK client gets a 400 (missing required field: model) instead of the retry. Keep the client side on the listener.
  • The Responses API gets usage and request info, but no schema or transcoding yet.

The AI Protocol Manager is alpha, and all of this will move. If one of these configs stops working on a newer Envoy, the tests will tell you which part changed. Issues and ideas are welcome on GitHub.