I have observed that when running coder speedtest on a fast network link TCP packets get dropped between gVisor and tailscale subcomponents.

The cause is the use of a channel.Endpoint to send packets from the gVisor netstack to the wireguard-go device for encryption. The channel.Endpoint has a fixed size queue (set to 512 packets at present), and when the queue is filled, it drops packets.

While dropping packets is nominally expected in networking applications, and is the main way that intermediate network elements can put back-pressure on TCP transmissions to avoid over-congestion, it's pretty disruptive for the packets to get dropped even before leaving Coder. The sending TCP stack doesn't realize there is a problem until one round-trip-time later, which on a fast link can be megabytes of data potentially disrupted. It is also a very crude tool for back-pressure when everything is happening within the same process, dropping throughput by 30% for several seconds.

In my testing on GCP c3d-standard-8 VMs, drops happen around 300 Mbit/s with packet capture turned on.

We'll need to more fully explore the consequences, but hacking gVisor's channel.Endpoint not to drop packets and instead to block allowed throughput to consistently run at 300 Mbit/s, instead of reaching 300 Mbit/s and then quickly dropping back to 200 Mbit/s or so. When I turn off packet capture, this rises to 530 Mbit/s.

related: #13042