Difference iPerf results on windows and linux using same hardware

[Munaveed2002]

Hi All,

I am struggling with iperf between windows and linux. When I install Linux on same hardware, I get ~1G bandwidth, however, when I install windows on it I get ~150 Mbps.

I know distance does have a impact when it comes to throughput but why it doesn't have any effect when I install Linux on same hardware?
Would like to know why iperf is sensitive about the distance on windows application but not on Linux?

Stats:

►Test 1:
Version iperf 3.1.7
Operating System: Linux Red Hat (3.10.0-1160.53.1.el7.x86_64)
Latency between Server & client is 12ms

$ ping 10.42.160.10 -c 2
PING 10.42.160.10 (10.42.160.10) 56(84) bytes of data.
64 bytes from 10.42.160.10: icmp_seq=1 ttl=57 time=12.5 ms
64 bytes from 10.42.160.10: icmp_seq=2 ttl=57 time=11.9 ms

--- 10.42.160.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 11.924/12.227/12.531/0.323 ms

►Upload from Client to Server
$ iperf3 -c 10.42.160.10 -p 8443 -b 2G -t 5
Connecting to host 10.42.160.10, port 8443
[ 4] local 10.43.243.204 port 60094 connected to 10.42.160.10 port 8443
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 97.6 MBytes 819 Mbits/sec 0 2.60 MBytes
[ 4] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 2.61 MBytes
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 2.61 MBytes
[ 4] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 0 2.64 MBytes
[ 4] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 2.66 MBytes

---

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 546 MBytes 917 Mbits/sec 0 sender
[ 4] 0.00-5.00 sec 546 MBytes 917 Mbits/sec receiver

iperf Done.
►Download from Server to Client
$ iperf3 -c 10.42.160.10 -p 8443 -b 2G -t 5 -R
Connecting to host 10.42.160.10, port 8443
Reverse mode, remote host 10.42.160.10 is sending
[ 4] local 10.43.243.204 port 60098 connected to 10.42.160.10 port 8443
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 108 MBytes 903 Mbits/sec
[ 4] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
[ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 4] 4.00-5.00 sec 112 MBytes 942 Mbits/sec

---

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 559 MBytes 938 Mbits/sec 0 sender
[ 4] 0.00-5.00 sec 558 MBytes 936 Mbits/sec receiver

►Test 2:
Version iperf 3.1.3
Operating System: Windows 10 64 bit
Latency between Server & client is 12ms

C:\Temp\iperf-3.1.3-win64>ping 10.42.160.10

Pinging 10.42.160.10 with 32 bytes of data:
Reply from 10.42.160.10: bytes=32 time=12ms TTL=62
Reply from 10.42.160.10: bytes=32 time=12ms TTL=62
Reply from 10.42.160.10: bytes=32 time=12ms TTL=62
Reply from 10.42.160.10: bytes=32 time=12ms TTL=62

Ping statistics for 10.42.160.10:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 12ms, Maximum = 12ms, Average = 12ms

C:\Temp\iperf-3.1.3-win64>iperf3 -c 10.42.160.10 -p 8443 -b 2G -t 5
Connecting to host 10.42.160.10, port 8443
[ 4] local 10.43.190.59 port 61578 connected to 10.42.160.10 port 8443
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 17.0 MBytes 143 Mbits/sec
[ 4] 1.00-2.00 sec 18.9 MBytes 158 Mbits/sec
[ 4] 2.00-3.01 sec 18.9 MBytes 157 Mbits/sec
[ 4] 3.01-4.01 sec 18.8 MBytes 158 Mbits/sec
[ 4] 4.01-5.00 sec 18.8 MBytes 158 Mbits/sec

---

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-5.00 sec 92.2 MBytes 155 Mbits/sec sender
[ 4] 0.00-5.00 sec 92.2 MBytes 155 Mbits/sec receiver

iperf Done.

C:\Temp\iperf-3.1.3-win64>iperf3 -c 10.42.160.10 -p 8443 -b 2G -t 5 -R
Connecting to host 10.42.160.10, port 8443
Reverse mode, remote host 10.42.160.10 is sending
[ 4] local 10.43.190.59 port 61588 connected to 10.42.160.10 port 8443
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 15.7 MBytes 132 Mbits/sec
[ 4] 1.00-2.00 sec 15.6 MBytes 131 Mbits/sec
[ 4] 2.00-3.00 sec 15.7 MBytes 132 Mbits/sec
[ 4] 3.00-4.00 sec 15.7 MBytes 132 Mbits/sec
[ 4] 4.00-5.00 sec 15.7 MBytes 132 Mbits/sec

---

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 80.4 MBytes 135 Mbits/sec 0 sender
[ 4] 0.00-5.00 sec 78.9 MBytes 132 Mbits/sec receiver

iperf Done.

[davidBar-On]

Hi, a possible reason is that different buffer sizes are used. The size can be viewed by using the --verbose option, but it is better to try setting the size manually to make sure they are the same. Try first to set the size to a small size in both cases, e.g. -l 512. If the results for Linux and Windows are similar in this case (but probably with reduced throughput), try setting the size to a large value, e.g. -l 128K. (Note that version 3.1.x is very old, although I don't think it has major impact i this case.)

Note that the -b 2G is probably redundant. Without setting bandwidth target, the maximum possible rate is achieved.

[Munaveed2002]

Hi David,

I have dropped -b, and added -l and didn't experienced any different results. Also, one thing I have noticed that in windows the MSS size is picked up as default, however, when I locked it down to 1500, the results didn't changed.

Linux:
$ iperf3 -c 10.42.160.10 -p 8443 -t 5 -V -l 512
iperf 3.1.7
Linux 60246 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64
Control connection MSS 1448
Time: Fri, 04 Mar 2022 19:56:25 GMT
Connecting to host 10.42.160.10, port 8443
Cookie: 60246.1646423
TCP MSS: 1448 (default)
[ 4] local 10.43.243.204 port 36136 connected to 10.42.160.10 port 8443
Starting Test: protocol: TCP, 1 streams, 512 byte blocks, omitting 0 seconds, 5 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 106 MBytes 888 Mbits/sec 0 2.55 MBytes
[ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 2.56 MBytes
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 2.57 MBytes
[ 4] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 0 2.58 MBytes
[ 4] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 2.58 MBytes

---

Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 555 MBytes 931 Mbits/sec 0 sender
[ 4] 0.00-5.00 sec 555 MBytes 931 Mbits/sec receiver
CPU Utilization: local/sender 49.4% (10.1%u/39.5%s), remote/receiver 2.3% (0.7%u/1.5%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

$ iperf3 -c 10.42.160.10 -p 8443 -t 5 -V -l 128K
iperf 3.1.7
Linux 60246 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64
Control connection MSS 1448
Time: Fri, 04 Mar 2022 19:56:34 GMT
Connecting to host 10.42.160.10, port 8443
Cookie: 60246.1646423
TCP MSS: 1448 (default)
[ 4] local 10.43.243.204 port 36140 connected to 10.42.160.10 port 8443
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 5 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 107 MBytes 899 Mbits/sec 0 2.85 MBytes
[ 4] 1.00-2.00 sec 112 MBytes 944 Mbits/sec 0 3.01 MBytes
[ 4] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 3.01 MBytes
[ 4] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 3.01 MBytes
[ 4] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 3.01 MBytes

---

Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 556 MBytes 933 Mbits/sec 0 sender
[ 4] 0.00-5.00 sec 556 MBytes 932 Mbits/sec receiver
CPU Utilization: local/sender 3.9% (0.2%u/3.8%s), remote/receiver 6.5% (0.9%u/5.5%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

Windows:
C:\Temp\iperf-3.1.3-win64>iperf3 -c 10.42.160.10 -p 8443 -t 5 -V -l 512
iperf 3.1.3
CYGWIN_NT-10.0 60030 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Fri, 04 Mar 2022 19:56:47 GMT
Connecting to host 10.42.160.10, port 8443
Cookie: 60030.1646423807.230049.4e329
TCP MSS: 0 (default)
[ 4] local 10.43.155.220 port 53623 connected to 10.42.160.10 port 8443
Starting Test: protocol: TCP, 1 streams, 512 byte blocks, omitting 0 seconds, 5 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 17.8 MBytes 149 Mbits/sec
[ 4] 1.00-2.00 sec 18.0 MBytes 151 Mbits/sec
[ 4] 2.00-3.00 sec 18.3 MBytes 154 Mbits/sec
[ 4] 3.00-4.00 sec 18.3 MBytes 154 Mbits/sec
[ 4] 4.00-5.00 sec 18.3 MBytes 153 Mbits/sec

---

Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-5.00 sec 90.7 MBytes 152 Mbits/sec sender
[ 4] 0.00-5.00 sec 90.7 MBytes 152 Mbits/sec receiver
CPU Utilization: local/sender 34.0% (10.6%u/23.4%s), remote/receiver 8.7% (2.1%u/6.7%s)

iperf Done.

C:\Temp\iperf-3.1.3-win64>iperf3 -c 10.42.160.10 -p 8443 -t 5 -V -l 128K
iperf 3.1.3
CYGWIN_NT-10.0 60030 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Fri, 04 Mar 2022 19:56:56 GMT
Connecting to host 10.42.160.10, port 8443
Cookie: 60030.1646423816.188780.53f97
TCP MSS: 0 (default)
[ 4] local 10.43.155.220 port 53636 connected to 10.42.160.10 port 8443
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 5 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 18.4 MBytes 153 Mbits/sec
[ 4] 1.01-2.00 sec 18.9 MBytes 159 Mbits/sec
[ 4] 2.00-3.01 sec 18.8 MBytes 156 Mbits/sec
[ 4] 3.01-4.00 sec 18.9 MBytes 160 Mbits/sec
[ 4] 4.00-5.01 sec 19.0 MBytes 159 Mbits/sec

---

Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-5.01 sec 93.9 MBytes 157 Mbits/sec sender
[ 4] 0.00-5.01 sec 93.9 MBytes 157 Mbits/sec receiver
CPU Utilization: local/sender 1.8% (0.3%u/1.5%s), remote/receiver 2.4% (0.4%u/2.0%s)

iperf Done.

C:\Temp\iperf-3.1.3-win64>iperf3 -c 10.42.160.10 -p 8443 -t 5 -V -l 128K -M 1500
iperf 3.1.3
CYGWIN_NT-10.0 60030 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Fri, 04 Mar 2022 20:04:22 GMT
Connecting to host 10.42.160.10, port 8443
Cookie: 60030.1646424262.735579.3a7a3
TCP MSS: 1500
[ 4] local 10.43.155.220 port 53875 connected to 10.42.160.10 port 8443
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 5 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 17.8 MBytes 148 Mbits/sec
[ 4] 1.01-2.01 sec 18.2 MBytes 153 Mbits/sec
[ 4] 2.01-3.00 sec 18.1 MBytes 153 Mbits/sec
[ 4] 3.00-4.00 sec 18.2 MBytes 153 Mbits/sec
[ 4] 4.00-5.00 sec 18.1 MBytes 152 Mbits/sec

---

Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-5.00 sec 90.5 MBytes 152 Mbits/sec sender
[ 4] 0.00-5.00 sec 90.5 MBytes 152 Mbits/sec receiver
CPU Utilization: local/sender 2.1% (0.9%u/1.2%s), remote/receiver 0.0% (0.0%u/0.0%s)

iperf Done.

[ChristophKarlHeck]

Hi @Munaveed2002,
I have exactly the same problem did you solve it?
Cheers
Christoph

[Munaveed2002]

the way the Linux deal with windowing is different as compared to Microsoft windows.

Would you be able to share some stats with me?

[ChristophKarlHeck]

Hi,
I have already posted my problem on superuser forum:
https://superuser.com/questions/1720662/bandwidth-from-linux-to-linux-8gbps-greater-than-from-linux-to-windows600mbps
If you need additional information, I can send you the Wireshark analysis.

[davidBar-On]

@ChristophKarlHeck, per the information you posted on the superuser forum, you are using reverse mode (-R). I.e. the Server is sending. Can you try without the -R and see if there is still a problem? Also, can you switch the roles of the client/server and remove the -R and see what happens?

Note that you are using (very) old iperf3 versions on both Linux and Windows. That may be important especially for the reverse mode. I suggest that you will use newer iperf3 versions, to take advantage of the bug fixes over the years.

[davidBar-On]

Hi Munaveed2002,
I don't see anything that may hint to what the problem may be. Some further evaluation possibilities:

1. It may be that there was a problem in iperf3 on Windows that was fixed. Therefore, it can help if you will use a new iperf3 version from (not official site): https://files.budman.pw/ or https://www.neowin.net/forum/topic/1234695-iperf-3101-windows-build/page/3/ (in both download is done from the same site).
2. Wireshark log on Windows can allow to see whether the issue is that for some reasons there are a of TCP retries on Windows.