Istio ambient performance issues #57769
-
|
Hey folks! I'm currently working on a PoC for istio ambient mode and while testing istio via iperf3 I saw that the bandwidth gets cut in half (from ~10Gbit/s to ~5Gbit/s). I didn't yet test for latency or RPS but I wanted to look into this first as it doesn't seem to be right when looking at other performance tests (like https://istio.io/latest/blog/2025/ambient-performance/ and #50373 (comment)). Any suggestions what to look into? Or maye I'm just hitting a performance limit of the underlying hardware? Searching around the internet didn't give me many ideas. I compiled some more data into https://github.com/xatnight/istio-performance but here's a quick rundown:
Running iperf3 with one connection on cilium without istio gives me 11gbit/s (pod to pod on different nodes). netdebug-client-0:~$ iperf3 -t 20 -c netdebug-server-1.netdebug-server.iperf.svc.cluster.local
Connecting to host netdebug-server-1.netdebug-server.iperf.svc.cluster.local, port 5201
[ 5] local 10.42.4.6 port 55860 connected to 10.42.6.58 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.25 GBytes 10.7 Gbits/sec 372 1.31 MBytes
[ 5] 1.00-2.00 sec 1.34 GBytes 11.5 Gbits/sec 380 1.35 MBytes
[ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 168 1.33 MBytes
[ 5] 3.00-4.00 sec 1.28 GBytes 11.0 Gbits/sec 0 1.62 MBytes
[ 5] 4.00-5.00 sec 1.29 GBytes 11.1 Gbits/sec 68 1.43 MBytes
[ 5] 5.00-6.00 sec 1.28 GBytes 11.0 Gbits/sec 37 1.49 MBytes
[ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 78 1.55 MBytes
[ 5] 7.00-8.00 sec 1.30 GBytes 11.2 Gbits/sec 265 1.61 MBytes
[ 5] 8.00-9.00 sec 1.28 GBytes 11.0 Gbits/sec 83 1.44 MBytes
[ 5] 9.00-10.00 sec 1.28 GBytes 11.0 Gbits/sec 1 1.48 MBytes
[ 5] 10.00-11.00 sec 1.32 GBytes 11.3 Gbits/sec 27 1.63 MBytes
[ 5] 11.00-12.00 sec 1.30 GBytes 11.2 Gbits/sec 214 1.65 MBytes
[ 5] 12.00-13.00 sec 1.28 GBytes 11.0 Gbits/sec 50 1.38 MBytes
[ 5] 13.00-14.00 sec 1.28 GBytes 11.0 Gbits/sec 103 1.37 MBytes
[ 5] 14.00-15.00 sec 1.30 GBytes 11.2 Gbits/sec 59 1.48 MBytes
[ 5] 15.00-16.00 sec 1.24 GBytes 10.7 Gbits/sec 348 1.84 MBytes
[ 5] 16.00-17.00 sec 1.29 GBytes 11.1 Gbits/sec 323 1.29 MBytes
[ 5] 17.00-18.00 sec 1.31 GBytes 11.2 Gbits/sec 0 1.85 MBytes
[ 5] 18.00-19.00 sec 1.28 GBytes 11.0 Gbits/sec 160 1.25 MBytes
[ 5] 19.00-20.00 sec 1.32 GBytes 11.3 Gbits/sec 17 1.28 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 25.8 GBytes 11.1 Gbits/sec 2753 sender
[ 5] 0.00-20.00 sec 25.8 GBytes 11.1 Gbits/sec receiverDoing the same with ambient mode enabled gives me 5,5 gbit/s netdebug-client-0:~$ iperf3 -t 20 -c netdebug-server-1.netdebug-server.iperf.svc.cluster.local
Connecting to host netdebug-server-1.netdebug-server.iperf.svc.cluster.local, port 5201
[ 5] local 10.42.4.155 port 50136 connected to 10.42.6.58 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 615 MBytes 5.15 Gbits/sec 109 785 KBytes
[ 5] 1.00-2.00 sec 616 MBytes 5.17 Gbits/sec 85 627 KBytes
[ 5] 2.00-3.00 sec 656 MBytes 5.50 Gbits/sec 74 558 KBytes
[ 5] 3.00-4.00 sec 623 MBytes 5.23 Gbits/sec 92 709 KBytes
[ 5] 4.00-5.00 sec 622 MBytes 5.22 Gbits/sec 82 486 KBytes
[ 5] 5.00-6.00 sec 620 MBytes 5.21 Gbits/sec 61 564 KBytes
[ 5] 6.00-7.00 sec 653 MBytes 5.48 Gbits/sec 64 598 KBytes
[ 5] 7.00-8.00 sec 608 MBytes 5.10 Gbits/sec 71 666 KBytes
[ 5] 8.00-9.00 sec 633 MBytes 5.31 Gbits/sec 99 658 KBytes
[ 5] 9.00-10.00 sec 614 MBytes 5.15 Gbits/sec 86 673 KBytes
[ 5] 10.00-11.00 sec 640 MBytes 5.37 Gbits/sec 73 419 KBytes
[ 5] 11.00-12.00 sec 634 MBytes 5.32 Gbits/sec 74 597 KBytes
[ 5] 12.00-13.00 sec 615 MBytes 5.16 Gbits/sec 57 613 KBytes
[ 5] 13.00-14.00 sec 640 MBytes 5.37 Gbits/sec 72 590 KBytes
[ 5] 14.00-15.00 sec 640 MBytes 5.36 Gbits/sec 75 481 KBytes
[ 5] 15.00-16.00 sec 642 MBytes 5.38 Gbits/sec 64 576 KBytes
[ 5] 16.00-17.00 sec 759 MBytes 6.37 Gbits/sec 29 440 KBytes
[ 5] 17.00-18.00 sec 670 MBytes 5.62 Gbits/sec 39 504 KBytes
[ 5] 18.00-19.00 sec 723 MBytes 6.06 Gbits/sec 53 562 KBytes
[ 5] 19.00-20.00 sec 716 MBytes 6.01 Gbits/sec 40 538 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 12.6 GBytes 5.43 Gbits/sec 1399 sender
[ 5] 0.00-20.01 sec 12.6 GBytes 5.43 Gbits/sec receiver |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
@howardjohn and @Stevenjin8 have been our data plane perf folks so they probably have some good insight. What I'd say generally though is that the major difference between ambient and raw cilium is the mTLS encryption istio does. That's a massive benefit but it is going to come at a cost. When you enable cilium's encryption for a more apples to apples comparison, Istio outperforms cilium in throughput (at least on our tests): https://istio.io/latest/blog/2025/ambient-performance/ |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick response! Yeah we had the same results running cilium with wireguard. We got around 3gbit/s throughput and we can't run 9000 MTU so there's no real way of improving that further. But based on the results you are linking I wouldn't expect the bandwidth to be cut in half (unless you are running 100gbit/s NICs in these tests? I couldn't find any info about what kind of hardware these were done on, but I assumed 40gbit/s NICs). My expectations was somewhere around the 8gbit/s ballpark when enabling mTLS. |
Beta Was this translation helpful? Give feedback.
-
|
What type of CPUs are you on? Also, whats with all the retransmissions? Those seem alarming (with and without ambient)? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the fast reply! The CPUs are reported as Intel Xeon Gold 6246R. I can ask our hosting partner for confirmation but in the past they said several times that they are using Intel Xeon Gold so that seems correct to me. I will look a bit further into the retransmissions. I didn't pay them much attention as I still achieve almost native performance without istio but maybe that's a point worth looking into. |
Beta Was this translation helpful? Give feedback.
-
|
Ok so that is the equivilent of a AWS C5 instance, 2nd gen Intel (latest is 5th gen). So its pretty likely you are being bottlenecked on CPU? In the past I had benchmarked AWS instances against raw crypto speed: c5: 2GB/s note: GB vs Gb, so the 2GB/s is 16Gb/s. However, this is purely doing TLS encryption operation so doesn't account for any other overhead I also did iperf test (technically, this was not with ztunnel but with a simple TLS proxy, so its not 100% applicable but still relevant): "old" is C5 instance, "new" is c7i. The tests are:
So you can see the TLS case capped out at 4Gb/s, which is a bit less than you saw (perhaps ztunnel is more efficient, etc) |
Beta Was this translation helpful? Give feedback.
-
|
Yeah looks pretty much like it's hardware limitation, thanks for your time and data! I will take it up to our provider and see what we can do. |
Beta Was this translation helpful? Give feedback.
Ok so that is the equivilent of a AWS C5 instance, 2nd gen Intel (latest is 5th gen). So its pretty likely you are being bottlenecked on CPU?
In the past I had benchmarked AWS instances against raw crypto speed:
c5: 2GB/s
c6: 4.8GB/s
c7: 6.4GB/s
note: GB vs Gb, so the 2GB/s is 16Gb/s. However, this is purely doing TLS encryption operation so doesn't account for any other overhead
I also did iperf test (technically, this was not with ztunnel but with a simple TLS proxy, so its not 100% applicable but still relevant):