These are learning notes rather than settled advice. I am working through how trust boundaries should sit in a multi-tenant agent platform, and writing it down is how I find the holes. Some of it will turn out to be wrong, as the first half of this post demonstrates.
I spent a while convinced of a sentence that turned out to be wrong. Working out why settled an architecture question I had been circling separately, about where the L7 proxy should run. The two turned out to be the same question, which is why they are in the same post.
The sentence
The setup is a platform where agent sessions migrate between machines many times a day. A gateway sits in front of them and terminates the agent’s TLS connection so it can read the request, work out where that session currently lives, and forward it there. By any honest definition the gateway is a man in the middle. But it never holds a session’s private key, and on that basis I wrote something close to:
The gateway is trusted with availability, and trusted with nothing about identity.
It does not survive contact with what a terminating proxy actually does.
A proxy that terminates TLS and re-originates is not passing your connection along. It is an endpoint on both sides: facing the agent it behaves exactly like the destination server, and facing the destination it behaves exactly like the agent. In between there is a moment where the request exists as plaintext in that process’s memory. From there the gateway can read everything, rewrite requests, drop or reorder them, fabricate responses and attribute them to the agent, and — unless client identity is separately signed and bound to each request — impersonate clients to the agents themselves.
So a per-session certificate prevents exactly one thing. The gateway cannot go somewhere else and claim to be tenant 42’s agent. That is worth having, but it is a much smaller claim than the one I made. I had conflated “cannot steal the agent’s credential”, which is true, with “cannot impersonate the agent at the application boundary”, which is not.
The version I would defend now:
The gateway is trusted for the confidentiality and integrity of every flow it terminates. Per-session identity keeps the agent’s long-lived private key out of the gateway, but it does not eliminate portable authority: any bearer token, API key, or replayable credential exposed in plaintext can still be stolen and used elsewhere. Compromise controls the traffic that traverses the gateway and any exportable credentials that traffic carries; it does not automatically confer session identities or signing authority that never reached it.
That second clause matters more than it looks, and I had to walk back a stronger version of this paragraph to get it right. My first instinct was that per-session credentials stop in-path authority from becoming portable authority. They do not. They prevent one specific form of in-path authority — possession of the agent’s private key — from becoming portable. Eliminating other portable authority requires sender-constrained credentials, request signing, or credential injection beyond the terminating proxy.
The point lands hardest on the egress path discussed later. A model provider API key is portable authority in its purest form: long-lived, bearer-shaped, valid from anywhere, and useful to an attacker long after the connection that leaked it is gone. If your gateway injects provider credentials — and mine does — then it handles precisely the class of secret that its per-session certificate scoping does nothing to protect. What remains true, and is still worth building for, is narrower: compromise does not hand over identities and signing keys belonging to sessions that never passed through that gateway.
If you want the stronger property, transport security cannot give it to you, because the gateway is an endpoint of that transport. You have to sign above it: the client signs method, target identity, body hash, nonce and expiry; the agent verifies that itself rather than taking the gateway’s word for it; the agent signs its response bound to the request ID. One detail is easy to miss, which is that replay protection has to survive checkpoint and restore, since restoring an older snapshot silently reopens a replay window you thought was closed. This costs latency and SDK surface, and deciding not to pay for it is reasonable. Describing your system as though you had paid for it is not.
Three principals
Part of why the bad sentence was easy to write is vagueness about whose identity is being checked. On the single hop from gateway to worker there are three principals, not one, and they are checked in different directions.
| Principal | Its credential proves | Checked by |
|---|---|---|
| Gateway | “I am an authorized platform hop” | The worker, and the client downstream |
| Agent | “This instance represents session X” | The gateway, and peer agents |
| Client | “This request came from principal Y” | The agent, or an authorization service |
The worker validates the gateway’s identity; the gateway validates that the server it reached presents the expected session identity; the agent separately validates forwarded client authorization. Collapsing these produces sentences like “the worker validates the session certificate”, which has the direction backwards — the session lives on the worker, so it is the thing being authenticated, not the thing authenticating.
While I am correcting myself: I had also described tampering with a checkpointed snapshot as a man-in-the-middle attack on memory. It is a good image but a bad threat model, because it hides the fact that signatures do not fix all of it. An attacker who swaps generation 50 of a snapshot for generation 42 has not forged anything. Every digest verifies, every signature checks out, and the agent wakes into an authentic version of its own past, one where a revoked credential still worked. Integrity was never violated; freshness was. Snapshot manifests therefore have to bind a monotonic generation, checked against an authority that is not the storage itself.
Where this lands architecturally
Everything above is about how much a TLS-terminating component is trusted. That answers a question which looks unrelated: where it should run. The connection is one sentence. If a gateway is fully trusted for the traffic it terminates, then the population whose traffic it terminates is the population that shares its fate.
The pattern worth borrowing is the one sidecar-less meshes converged on, though not for the reason it usually gets cited. People summarize it as one proxy per node instead of one per pod, which misses the part that matters. The important move is the split: a node-local L4 layer handling identity, capture and secure transport, and a separate L7 layer handling TLS termination, protocol policy and routing.
That split is about function, not about which binary you run. Envoy is entirely capable of the L4 role — TCP proxy and network filters, no HTTP connection manager, no termination of application TLS — and using it in both places keeps one data plane, one configuration language and one operational story across the platform, which is worth a lot in practice. Some meshes ship a smaller purpose-built proxy for the node instead, and the argument for that is a reduced attack surface on the component running on every host. It is a reasonable hardening choice, but a second-order one. What actually constrains the blast radius is what the node-local component is permitted to do: give Envoy an L4-only configuration, no interception keys, and no application TLS termination, and it can be configured with the same intended data-plane authority, although a purpose-built proxy may still have a smaller binary and extension attack surface. Configuration scope is the primary control. The binary is a secondary one, not a non-existent one.
The second detail is that the tunnel carries the application stream opaquely. Wrap the workload’s TCP stream in an HTTP/2 CONNECT tunnel protected by mTLS, which is the usual construction, and if the application inside is speaking HTTPS then that inner session stays encrypted straight through the node layer.
Which gives the correction I find myself repeating most often: mTLS is not MITM. Mesh mTLS protects the link and tells you which workload is talking. Enforcing anything about HTTP paths, tool names, model IDs or token budgets requires plaintext, which means something was explicitly authorized to terminate the application’s own TLS. Every “we have mTLS, so we have visibility into agent traffic” claim dies there.
Now the placement argument. A bare-metal node already owns a large failure domain — compromise it and you get every sandbox on it, its local identities, its cached state. That is unavoidable; it is the physics of sharing a host. Authorized TLS termination owns a different failure domain: the plaintext of every tenant whose traffic it decrypts, plus the leaf certificates and private keys it holds to do so. Put a full L7 interception proxy on every node and those two domains merge. Every host compromise becomes a plaintext and key compromise for whichever tenants happened to be scheduled there — and placement is a scheduling decision, not a security decision, which means you have handed your trust boundaries to the scheduler, and it will redraw them on every rebalance.
Ingress and egress are not the same problem
I have been writing “the L7 tier” as though it were one thing. It is two, and almost everything that matters differs between them.
Egress is an agent calling out — a model provider, a tool server, some URL it discovered at runtime. The destination is not yours, frequently is not known until the request arrives, and the client is code you do not trust. This is where interception actually happens: you terminate the agent’s TLS to a third-party hostname by presenting a certificate minted under a private CA that only your own sandbox images trust.
Ingress is the outside calling in — a user’s browser or application reaching an agent session that may currently be asleep. The destination is yours. You terminate TLS for your own domain using an ordinary public certificate, which is not interception at all, because you own the name.
Three things follow. The two paths need separate CA hierarchies, and merging them is a serious mistake. Your ingress certificate is a public one for a name you own and is revocable through ordinary means. Your egress interception root can mint a certificate for any hostname that your sandboxes will accept, which is a far more dangerous capability, and it must never be trusted by anything outside those sandboxes. Keep private interception roots well away from public trust roots.
Only egress is really MITM. Ingress is standard edge termination. That said, the correction at the top of this post applies to both — an ingress gateway that terminates TLS is still fully trusted for that traffic, and still cannot be described as trusted with nothing. What is egress-specific is the interception machinery: a private CA, certificates minted per destination hostname, and the requirement that the client trust a root you control.
The per-node question is really an egress question. Nobody puts ingress on every node; it lives behind a load balancer at the edge by construction. Egress is the one where placement is genuinely open, because that traffic originates on the node and could be terminated there. So everything below is about the egress tier.
The other asymmetries fall out of the same split. Wake-on-request belongs to ingress, along with the routing invariant that a lookup returns an agent identity and a placement epoch and the gateway forwards nothing until the peer proves both. Confused-deputy risk belongs to egress: an agent that asks the proxy to fetch a link-local or metadata address is exploiting the fact that the proxy has network reach the sandbox does not, which is why destination policy has to be applied to resolved addresses rather than hostnames, and re-applied after every resolution.
On the node, or at a remote endpoint
This is about the egress tier specifically. The L4 layer stays on the node either way, and it may well be Envoy. The question is only where the terminating proxy lives — the one that decrypts application TLS and holds interception keys.
Both candidates for that role are multi-tenant. Both run one Envoy process serving many tenants, so shared-versus-dedicated is not the axis people think it is. What differs is who picks the roommates and what the roommates can lose together.
| On the node | At a remote endpoint | |
|---|---|---|
| Who shares the process | Whoever the scheduler placed here | Whoever you assigned |
| Isolation tiers | Coarse and scheduler-coupled | Dedicated shards where needed |
| Private keys | On every host | Only in the L7 tier |
| Host compromise costs | Plaintext and keys for local tenants | Sandboxes only |
| Added latency | Effectively zero | One network hop |
| Failure domain | Aligned with the node | Cuts across nodes; needs replicas |
| Scaling | Proxy count tied to node count | Traffic and compute scale separately |
| Certificate working set | Same cert cached on every node | Cached once per shard |
| Rollout | Bad version reaches every host | Canary and drain one shard |
The row that matters most is the first. With a per-node L7 proxy, the set of tenants sharing a decrypting process is decided by bin-packing and re-decided on every rebalance. You can still express isolation, to be fair — node pools, affinity rules, taints, dedicated hosts — but only at node granularity, and only by coupling a security boundary to the scheduler. Promising a regulated customer that their plaintext never shares a process with an untrusted tenant then means pinning them to their own node pool, which surrenders the density and locality that made per-node attractive in the first place. Remote sharding makes tenant mix an explicit and stable placement decision at a finer granularity than a host, which is what turns a dedicated isolation tier into something you can sell rather than something you configure the scheduler to approximate.
Certificate economics push the same way and are less obvious. Interception needs a leaf per destination hostname, and per node, every host with a tenant calling a given host fetches and caches its own copy — the same certificate held N times across N hosts, with N times the issuance load and a poor hit rate, since each node’s cache is small and starts cold. Concentrate that traffic into a shard serving fifty to a hundred tenants and the certificate is fetched once and reused constantly. Upstream connection pools behave identically: agent egress converges on a handful of model providers and tool servers, so pooling wants concentration, and per-node placement fragments precisely what benefits from being pooled.
I do not want to strawman the other side, because per-node has one real advantage. When a per-node proxy dies it takes egress down for the sandboxes on that node, but those sandboxes already shared that node’s fate, so the failure domain is aligned with one that already existed. A remote shard introduces a domain that cuts across nodes, and you pay for that with replicas, health checks and drain procedures. The reason I still come down on the other side is that the per-node arrangement buys its clean failure property by making every host a leaf-key holder and a plaintext holder, which is a cheap availability win traded for an expensive security loss.
Latency deserves a number rather than a shrug. An extra same-zone hop costs a few hundred microseconds. Against a model provider call taking hundreds of milliseconds that is under one percent and genuinely does not matter; against a 2 ms internal RPC it is a quarter of the budget and matters a great deal. Agent egress, which is the traffic you actually want to inspect and meter, is firmly the first kind, so latency should not be allowed to overrule the security argument here. For latency-critical east-west traffic, keep it on the node and do not intercept it.
Per-node L7 is the right answer when the premise of this post does not hold — when every workload shares one trust domain, when the traffic is latency-critical east-west rather than egress, or at the edge where there is no remote tier to route to. The test I would apply is whether the tenants sharing that process would be acceptable roommates no matter how the scheduler shuffles them. If that sentence makes you uncomfortable, you need the remote tier.
For scale, the shape I would start from is one lightweight L4 component per bare-metal host, ten to twenty L7 shards per failure domain, fifty to a hundred tenants per shard, at least two replicas each, and dedicated shards for outsized or regulated tenants. Those are a load-test hypothesis rather than a capacity claim, and I would size by handshake rate, connection count and stream duration rather than request rate, because agent traffic is dominated by long-lived streams that request-rate dashboards hide. The number is not the point anyway. The point is that the affected population is known before an incident rather than discovered during one.
The principle
One rule generates most of the decisions above. Share mechanisms broadly, share authority narrowly.
Node-local capture, transport machinery, stateless policy binaries and public trust roots should be everywhere; they are mechanism, and duplicating them costs you nothing. Interception CA keys, provider credentials, plaintext access and control-plane mutation rights should be held by as few things as possible; they are authority, and every additional holder is another way to lose them. The same rule explains why a single global interception root is the real blast radius in these systems regardless of how carefully you shard the proxies underneath, and why on-demand certificate selection reduces the working set of private keys in a process without ever removing them.
It also explains why the correction at the top of this post and the placement argument at the bottom are the same argument. Once you accept that a TLS-terminating proxy is fully trusted for the traffic it terminates, where it runs stops being a question about latency and becomes a question about how many tenants you are willing to put inside one trust boundary. For a platform with a thousand tenants the question was never whether one proxy could technically hold a thousand certificates and routes. It probably can. The question is whether one process should own that much authority and that much shared fate.
That is where my thinking currently sits. The part I am least sure of is the revocation story for a workload that migrates faster than any network policy can follow, which I suspect is the next thing I have wrong.
Comments