A Copy That Never Stops Copying

Envoy’s TLS write path copies 16 KB for every 16 KB it sends — not occasionally, but for the life of a connection. This is what causes it and how the fix works.

The line

int rc = SSL_write(rawSsl(), write_buffer.linearize(bytes_to_write), bytes_to_write);

SSL_write() needs contiguous memory. Envoy’s write buffer is a chain of slices. linearize() bridges them: if the front slice already holds enough it returns a pointer for free, otherwise it allocates a slice, copies into it, and returns that.

It reads like a fallback for an awkward case. On a proxied response it is the steady state.

Why it never recovers

Two numbers matter, and they are the same number: the most Envoy hands to one SSL_write() is 16384 bytes, and Slice::default_slice_size_ — the size of each slice the read path reserves — is also 16384.

That is a reservation, not a limit. reservation.commit() only commits what was actually read, and other paths size a slice to whatever was written into it, so slices smaller than 16 KB are ordinary. But a body streamed from a fast upstream fills them right up, and that is the shape this section is about.

The short slice at the front is the response headers. ConnectionImpl::write() moves the codec’s output into the connection buffer wholesale, so a few hundred bytes of headers land as their own slice ahead of the body’s full-size ones. Call that size H.

Here is what linearize(16384) does to that buffer:

The leftover is always exactly H, for any H. The copy takes H bytes off the front and leaves H bytes behind on the next slice, so the offset is conserved rather than worn down. The short slice in the “after” state is a different H bytes — the tail of slice 2, not the original headers.

Call a buffer in this shape — a short slice of H bytes sitting ahead of full-size ones — a misaligned chain. The point is that linearize() does not clear it. It reproduces it, so every write from then on allocates and copies 16 KB.

The fix

When a write would have to copy again right after the last one did, write just the contiguous front slice instead. That is one short TLS record, and from then on the buffer is aligned and the writes that follow cost no copy at all.

The rule, in three cases

Each write asks for up to 16 KB. What happens depends on the buffer:

The first case is not new — linearize() already returns that pointer untouched. The third is the old behaviour. Only the middle one is added:

if (front_slice.len_ >= bytes_to_write) {
  // Already contiguous — linearize() would return this pointer without copying.
  return {front_slice.mem_, bytes_to_write, false};
}

if (avoid_repeated_linearize && linearized_last_write &&
    nextSliceCoversWrite(write_buffer, bytes_to_write)) {
  // Copied last time, about to copy again, and one short record fixes it.
  return {front_slice.mem_, front_slice.len_, false};
}

return {write_buffer.linearize(bytes_to_write), bytes_to_write, true};

All three conditions in the middle case are load-bearing. Dropping either of the last two breaks it in a different way.

Why “the last write had to copy”, and not a size threshold

A natural guard would be “only write the front slice directly if it is large enough to be worth a syscall”. It never fires. A threshold asks whether the front slice is big; the state it needs to act on is the one where the front slice is permanently small — held at H by the conservation shown above. The answer is no on exactly the writes where the fast path would help.

Why the slice behind it has to be checked

Writing the front slice only re-aligns the buffer if the slice behind it can satisfy the next write on its own:

bool nextSliceCoversWrite(Buffer::Instance& write_buffer, uint64_t bytes_to_write) {
  const Buffer::RawSliceVector slices = write_buffer.getRawSlices(/*max_slices=*/2);
  return slices.size() == 2 && slices[1].len_ >= bytes_to_write;
}

Without that check, a chain of equal sub-16 KB slices goes wrong — the slice behind the front one is short too, so nothing gets aligned.

How badly depends on what 16384 leaves over. A linearize consumes whole slices until it needs part of one more, and that slice keeps slice_size − (16384 mod slice_size) bytes. Pick a size just above an exact divisor of 16 KB and that remainder is almost nothing: 4097 is one byte over 16384/4, so it strands 4 bytes; 5462 is just over 16384/3 and strands 2. Those few bytes are all the short record has to write.

These are slice sizes fed to a unit test, not constants in Envoy — a slice holds whatever its producer wrote, so any size occurs. Taking 4097 as the worst case:

Measured in that test, 64 slices of each size:

Nearly double the records and writev syscalls, for identical bytes copied. A 2-byte TLS record still costs about 24 bytes on the wire and one full syscall.

4096 is in the table to show the problem is not about awkward remainders: it divides 16 KB exactly, so nothing is stranded, and the short record writes a whole 4096-byte slice instead. The slice behind it is still 4096, so the next write copies anyway. Any chain of equal sub-16 KB slices fails this way; the odd sizes just make it look worse. Real buffers take that shape whenever framing is interleaved with data — HTTP/1 chunked encoding, small HTTP/2 frames, addBufferFragment chains.

The check also means a write that drains the whole buffer is never split in two, since it can only pass when data remains behind the front slice.

When a write doesn’t go through

If the socket is full, SSL_write() writes nothing and returns SSL_ERROR_WANT_WRITE. Envoy sends the same write again when the socket is ready.

A retry must not re-run the decision above. By then the buffer has already been linearized, so the code would see a contiguous front slice, conclude that no copy was needed, and clear linearized_last_write. When the retry lands and drains, the short remainder is back at the front with no history behind it — and it copies again. On a congested connection that repeats on every write, and the change never takes effect at all.

So the pending write is stored and replayed unchanged: same pointer, same length, and the same record of whether it came from a copy. BoringSSL requires the identical pointer in any case, since Envoy does not set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER.

One ordering detail: that pending write is cleared before write_buffer.drain(), not after. drain() can run callbacks that re-enter doWrite(), and a nested call must not find a write still pending that has already been sent.

Results

Eight alternating paired runs — baseline, fixed, baseline, fixed — so machine interference lands on both arms:

The timing is macOS, where a memcpy is cheap relative to a syscall; the balance differs in production. The deterministic number is the copy counter: 10 copies per iteration → 1.

Rollout

Guarded by envoy.reloadable_features.tls_avoid_repeated_linearize, default on. The flag is read when a TLS connection is created, so disabling it applies to new connections rather than to connections already open.