10G Speeds wildly up and down?

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
I'm checking my speeds, and I get 7+ across most of my network, to and from my Macs, my R720-based Ubuntu VMs, and my QNAP NAS that I store my Plex data on, but when I check my 11.1U7 FreeNAS at the shell I get crazy up and down results (all core network components are Ubiquiti USW-XG and USG-XG with jumbo frames enabled). Enabling or disabling jumbo frames on FreeNAS (running on a R720XD with 192GB of ram, Dual 2.2GHz 8Core E5-2660, X520-T Dual Port 10GBe) doesn't seem to make a difference.

It starts at a high of of 7 and drops 1 1Gbps!? 15.12 is my 5k iMac, and 100.51 is the Plex server. See below that the Plex server does 7.75 reliably to the iMac, so it's not those 2 machines.

Code:
root@freenas:~ # iperf3 -c 192.168.15.12
Connecting to host 192.168.15.12, port 5201
[  5] local 192.168.101.101 port 42846 connected to 192.168.15.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   918 MBytes  7.70 Gbits/sec   11   1.00 MBytes
[  5]   1.00-2.00   sec   796 MBytes  6.68 Gbits/sec   13    546 KBytes
[  5]   2.00-3.00   sec   761 MBytes  6.38 Gbits/sec    6    292 KBytes
[  5]   3.00-4.00   sec   434 MBytes  3.64 Gbits/sec    0    326 KBytes
[  5]   4.00-5.00   sec   534 MBytes  4.48 Gbits/sec    0    355 KBytes
[  5]   5.00-6.00   sec   582 MBytes  4.88 Gbits/sec    0    386 KBytes
[  5]   6.00-7.00   sec   655 MBytes  5.50 Gbits/sec    0    430 KBytes
[  5]   7.00-8.00   sec   668 MBytes  5.60 Gbits/sec    2    239 KBytes
[  5]   8.00-9.00   sec   385 MBytes  3.23 Gbits/sec    0    281 KBytes
[  5]   9.00-10.00  sec   396 MBytes  3.32 Gbits/sec    0    315 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.99 GBytes  5.14 Gbits/sec   32             sender
[  5]   0.00-10.35  sec  5.98 GBytes  4.96 Gbits/sec                  receiver

iperf Done.
root@freenas:~ # iperf3 -c 192.168.15.12
Connecting to host 192.168.15.12, port 5201
[  5] local 192.168.101.101 port 35599 connected to 192.168.15.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   333 MBytes  2.80 Gbits/sec    1    163 KBytes
[  5]   1.00-2.00   sec   300 MBytes  2.52 Gbits/sec    0    211 KBytes
[  5]   2.00-3.00   sec   375 MBytes  3.14 Gbits/sec    1   1.43 KBytes
[  5]   3.00-4.00   sec   295 MBytes  2.48 Gbits/sec    0    194 KBytes
[  5]   4.00-5.00   sec   244 MBytes  2.05 Gbits/sec    1    148 KBytes
[  5]   5.00-6.00   sec   325 MBytes  2.73 Gbits/sec    0    204 KBytes
[  5]   6.00-7.00   sec   395 MBytes  3.32 Gbits/sec    0    248 KBytes
[  5]   7.00-8.00   sec   491 MBytes  4.12 Gbits/sec    0    287 KBytes
[  5]   8.00-9.00   sec   556 MBytes  4.67 Gbits/sec    0    326 KBytes
[  5]   9.00-10.00  sec   601 MBytes  5.04 Gbits/sec    0    378 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.83 GBytes  3.29 Gbits/sec    3             sender
[  5]   0.00-10.35  sec  3.83 GBytes  3.18 Gbits/sec                  receiver

iperf Done.
root@freenas:~ # iperf3 -c 192.168.100.51
Connecting to host 192.168.100.51, port 5201
[  5] local 192.168.101.101 port 55528 connected to 192.168.100.51 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   132 MBytes  1.10 Gbits/sec    3   88.4 KBytes
[  5]   1.00-2.00   sec   129 MBytes  1.08 Gbits/sec    1    103 KBytes
[  5]   2.00-3.00   sec   124 MBytes  1.04 Gbits/sec    2   72.7 KBytes
[  5]   3.00-4.00   sec   111 MBytes   932 Mbits/sec    2   67.0 KBytes
[  5]   4.00-5.00   sec   121 MBytes  1.01 Gbits/sec    1   95.5 KBytes
[  5]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    1    103 KBytes
[  5]   6.00-7.00   sec   131 MBytes  1.10 Gbits/sec    1    105 KBytes
[  5]   7.00-8.00   sec   156 MBytes  1.31 Gbits/sec    1   85.5 KBytes
[  5]   8.00-9.00   sec   134 MBytes  1.12 Gbits/sec    1    103 KBytes
[  5]   9.00-10.00  sec   127 MBytes  1.07 Gbits/sec    2   78.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.26 GBytes  1.08 Gbits/sec   15             sender
[  5]   0.00-10.00  sec  1.26 GBytes  1.08 Gbits/sec                  receiver

iperf Done.


But if the Plex VM is tested to the iMac, it easily does over 7, and the Plex R720 is in the same rack, on the same switch as the FreeNAS R720XD.

Code:
bferrell@plex:~$ iperf3 -c 192.168.15.12
Connecting to host 192.168.15.12, port 5201
[  4] local 192.168.100.51 port 41030 connected to 192.168.15.12 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   878 MBytes  7.36 Gbits/sec  1857    694 KBytes       
[  4]   1.00-2.00   sec   934 MBytes  7.84 Gbits/sec  729    837 KBytes       
[  4]   2.00-3.00   sec   931 MBytes  7.81 Gbits/sec  164    867 KBytes       
[  4]   3.00-4.00   sec   918 MBytes  7.70 Gbits/sec  252    690 KBytes       
[  4]   4.00-5.00   sec   910 MBytes  7.63 Gbits/sec  534    766 KBytes       
[  4]   5.00-6.00   sec   921 MBytes  7.72 Gbits/sec  114    913 KBytes       
[  4]   6.00-7.00   sec   930 MBytes  7.81 Gbits/sec  283    783 KBytes       
[  4]   7.00-8.00   sec   936 MBytes  7.85 Gbits/sec   20   1.04 MBytes       
[  4]   8.00-9.00   sec   922 MBytes  7.74 Gbits/sec  303    933 KBytes       
[  4]   9.00-10.00  sec   956 MBytes  8.03 Gbits/sec    7   1.12 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.02 GBytes  7.75 Gbits/sec  4263             sender
[  4]   0.00-10.00  sec  9.02 GBytes  7.75 Gbits/sec                  receiver

iperf Done.
 

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
So, I've been digging through other posts, and put in some of recommended tunables, which has gotten my Intel 520 into the 3 Gbps range, which I've confirmed with some test file transfers. I'm also going to try the a Chelsio T420 to see if that fares any better.
 

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
So, it doesn't look like these forums really are active... but I keep playing with this and added a bunch of the recommended tunables, but what I'm actually seeing is odd. If I iperf from the FreeNAS to my PVE hosts I get crap speeds, if I from PVE to FreeNAS it does great.

PVE->FreeNAS
Code:
root@freenas:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.101.12, port 40618
[  5] local 192.168.101.101 port 5201 connected to 192.168.101.12 port 40620
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1001 MBytes  8.39 Gbits/sec
[  5]   1.00-2.00   sec   988 MBytes  8.29 Gbits/sec
[  5]   2.00-3.00   sec   983 MBytes  8.24 Gbits/sec
[  5]   3.00-4.00   sec  1018 MBytes  8.54 Gbits/sec
[  5]   4.00-5.00   sec  1.00 GBytes  8.61 Gbits/sec
[  5]   5.00-6.00   sec   902 MBytes  7.55 Gbits/sec
[  5]   6.00-7.00   sec  1001 MBytes  8.42 Gbits/sec
[  5]   7.00-8.00   sec  1.00 GBytes  8.61 Gbits/sec
[  5]   8.00-9.00   sec  1.05 GBytes  9.06 Gbits/sec
[  5]   9.00-10.00  sec   989 MBytes  8.29 Gbits/sec
[  5]  10.00-10.00  sec   235 KBytes  7.92 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  9.78 GBytes  8.40 Gbits/sec                  receiver


FreeNAS -> PVE
Code:
root@freenas:~ # iperf3 -c 192.168.101.12
Connecting to host 192.168.101.12, port 5201
[  5] local 192.168.101.101 port 14462 connected to 192.168.101.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   311 MBytes  2.61 Gbits/sec    0    175 KBytes
[  5]   1.00-2.00   sec   470 MBytes  3.94 Gbits/sec    0    238 KBytes
[  5]   2.00-3.00   sec   668 MBytes  5.60 Gbits/sec    0    291 KBytes
[  5]   3.00-4.00   sec   511 MBytes  4.29 Gbits/sec    0    321 KBytes
[  5]   4.00-5.00   sec   591 MBytes  4.95 Gbits/sec    0    352 KBytes
[  5]   5.00-6.00   sec   625 MBytes  5.24 Gbits/sec    0    379 KBytes
[  5]   6.00-7.00   sec   626 MBytes  5.25 Gbits/sec    0    408 KBytes
[  5]   7.00-8.00   sec   689 MBytes  5.78 Gbits/sec    0    435 KBytes
[  5]   8.00-9.00   sec   694 MBytes  5.81 Gbits/sec    0    459 KBytes
[  5]   9.00-10.00  sec   924 MBytes  7.76 Gbits/sec    0    488 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.97 GBytes  5.12 Gbits/sec    0             sender
[  5]   0.00-10.41  sec  5.97 GBytes  4.92 Gbits/sec                  receiver

iperf Done.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well, the forums are active but after 8 years most of the questions have been answered repeatedly. Most of the posters here are home users and don't have 10G. Those of us who are professionals and/or hobbyists working with 10G are in the vast minority.

Speaking as someone who does this professionally and who wrote the 10G Networking Primer here, my interest fell off when I saw the letters "X520-T" and then saw lots of nonzero under "Retr" because this just screams "crappy cabling" and "failure to use suggested hardware."

You should be confirming that your networking is correct and reliable before you try tuning FreeNAS.
 

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
That's your right, but I have the finest Belden Cat6a cabling and the rest of my Dell stack gets 9Gbps no issues, so it's clearly crappy software and drivers for FreeNAS, but as I mentioned above I did order and now have the recommendation network card so I'll see if that makes FreeNAS happy, but if my cheap QNAP can do 10G on commodity hardware it would be reasonable to assume FreeNAS could so it on enterprise grade hardware... I'll figure it and you can take your disdain elsewhere....
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Intel wrote their own drivers and Sun Microsystems wrote ZFS. I'm sorry if you don't like their drivers and software.

The X520-T is a rather unusual card and is pretty rare. I find rare and unusual hardware also turns out to be problematic in many cases.

10Gbps copper is a crummy technology; we squeezed copper for all it was worth to get from 100Mbps to 1Gbps, adding two pairs, full bidirectionality, and 4D-PAM5 encoding, which meant that we were already pushing limits at 1Gbps. Quite frankly it's amazing they ever managed to come out with 10Gbps copper. I see lots of problems with 10Gbps copper setups. I generally see no problems with 10G SFP+ (fiber) setups.

Slight historical rewind:

We had copper 10Mbps as a standard in 1993, 100Mbps as a standard in 1996, and 1G as a standard in 1999. It was 2006 before 10GBase-T was released as a standard and really it was probably more like 2015 before we saw even mediocre uptake of the technology. 10G on copper is technologically demanding and is hard to get right in practice. By way of comparison, 10G on fiber has been commonplace since the early 2000's and is pretty foolproof.

You're seeing poor performance and errors to one machine. It is not "disdain" if I try to draw some reasonable conclusion based on the stated facts. I would be looking heavily at the switchport for errors and problems, and on the FreeBSD end for driver-reported errors. It screams "look for cabling issues." I don't care if you have Monoprice cables, Belden cables, or if you popped for the AudioQuest Diamond $10000 cables. Copper 10G is extremely finicky.

You complained that you weren't getting any responses and I offered an opinion as to why. As someone who knows quite a bit about networking, and who has provided large amounts of support to 10G users here in these forums, you're free to ignore what I have to say, or to try to take offense at it, but my opinion is "you'd be better off with a Chelsio T520-CR with SFP+ rather than any copper setup", and that opinion is based on lots of experiences both directly and also from trying to support forum users.
 
  • Like
Reactions: Xan

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
I know quite a bit about networking as well, just not much about FreeBSD and FreeNAS, and I have objective data that the network is working pretty well (see below), so yea, I didn't find the feedback very useful. I will take it for what it's worth and keep working the problem, and do appreciate your taking the time to comment, I just feel like some folks do it to appease their ego rather than to try and help folks who are ignorant of a new technology. It often doesn't mean they aren't intelligent, or that they haven't searched the forum, but they didn't find anything they thought was relevant (which might be a mistake). I find it odd, for example, that the speeds are extremely consistent but twice as fast in one direction. I'm sure it's trying to tell me something, but I don't know what it is. I'll try different cabling, and I'll try the Chelsio NIC I bought and see what that does.

My R620s wit the Dell 5810S crush iperf3 on this same switch and cabling, so maybe the 550 is a problem or maybe the one cable is bad. ProxMox cluster communications are great -

Code:
Linux svr-02 5.0.21-1-pve #1 SMP PVE 5.0.21-2 (Wed, 28 Aug 2019 15:12:18 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Sep  7 17:02:20 2019
root@svr-02:~# iperf3 -c 192.168.100.11
Connecting to host 192.168.100.11, port 5201
[  5] local 192.168.100.12 port 42142 connected to 192.168.100.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.50 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.50 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.50 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.58 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.66 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.69 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.69 MBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.69 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.69 MBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.37 Gbits/sec  1311    468 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec  1311             sender
[  5]   0.00-10.04  sec  11.0 GBytes 9.37 Gbits/sec                  receiver

iperf Done.
root@svr-02:~#


And my 2 R720s are good too, one with the X20 and one with a Dell, so you can maybe see why I'm so surprised that FreeNAS is having an issue, I don't typically find problems with my 10GBe network....
Code:
Linux svr-03 5.0.21-1-pve #1 SMP PVE 5.0.21-1 (Tue, 20 Aug 2019 17:16:32 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Sep  6 20:11:54 2019 from 192.168.15.22
root@svr-03:~# iperf3 -c 192.168.101.11
iperf3: error - unable to connect to server: No route to host
root@svr-03:~# iperf3 -c 192.168.100.11
Connecting to host 192.168.100.11, port 5201
[  5] local 192.168.100.13 port 41866 connected to 192.168.100.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec    0   1.44 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.44 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.44 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.44 MBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.44 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.51 MBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec  587   1.21 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.33 MBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec  635   1.01 MBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec  1015   1.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec  2237             sender
[  5]   0.00-10.04  sec  11.0 GBytes 9.37 Gbits/sec                  receiver

iperf Done.
root@svr-03:~#



Code:
Linux svr-04 5.0.21-1-pve #1 SMP PVE 5.0.21-1 (Tue, 20 Aug 2019 17:16:32 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Sep  5 22:23:24 2019
root@svr-04:~# iperf3 -c 192.168.100.11
Connecting to host 192.168.100.11, port 5201
[  5] local 192.168.100.14 port 32950 connected to 192.168.100.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.25 Gbits/sec  553   1.04 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.25 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.26 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.27 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.28 MBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.28 MBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.35 Gbits/sec    0   1.28 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.29 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.31 MBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.41 Gbits/sec    0   1.31 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  553             sender
[  5]   0.00-10.04  sec  10.9 GBytes  9.35 Gbits/sec                  receiver

iperf Done.
root@svr-04:~#


And, as I mentioned, even my QNAP with SFP does over 9Gbps to the PVE cluster, so the odd ball here is clearly the FreeNAS box. Could still be the cable, but it is the only system in the rack that is struggling. Even my iMac does 7.75Gbps, so my home data center runs pretty well.
Code:
root@svr-01:~# iperf3 -c 192.168.15.40
Connecting to host 192.168.15.40, port 5201
[  5] local 192.168.100.11 port 57680 connected to 192.168.15.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.02 GBytes  8.74 Gbits/sec  223    986 KBytes
[  5]   1.00-2.00   sec  1.07 GBytes  9.17 Gbits/sec  146    947 KBytes
[  5]   2.00-3.00   sec  1.05 GBytes  9.01 Gbits/sec   60    680 KBytes
[  5]   3.00-4.00   sec  1.06 GBytes  9.11 Gbits/sec   35    956 KBytes
[  5]   4.00-5.00   sec  1.06 GBytes  9.08 Gbits/sec   41    974 KBytes
[  5]   5.00-6.00   sec  1.06 GBytes  9.12 Gbits/sec  139    919 KBytes
[  5]   6.00-7.00   sec  1.05 GBytes  9.06 Gbits/sec  245   1.02 MBytes
[  5]   7.00-8.00   sec  1.07 GBytes  9.17 Gbits/sec   99    758 KBytes
[  5]   8.00-9.00   sec  1.06 GBytes  9.15 Gbits/sec   73    963 KBytes
[  5]   9.00-10.00  sec  1.08 GBytes  9.24 Gbits/sec  165    936 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.6 GBytes  9.08 Gbits/sec  1226             sender
[  5]   0.00-10.03  sec  10.6 GBytes  9.05 Gbits/sec                  receiver

iperf Done.



Code:
Brett-5k-iMac:etc bferrell$ iperf3 -c 192.168.15.40
Connecting to host 192.168.15.40, port 5201
[  5] local 192.168.15.12 port 49418 connected to 192.168.15.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   893 MBytes  7.49 Gbits/sec                 
[  5]   1.00-2.00   sec   925 MBytes  7.76 Gbits/sec                 
[  5]   2.00-3.00   sec   914 MBytes  7.66 Gbits/sec                 
[  5]   3.00-4.00   sec   933 MBytes  7.82 Gbits/sec                 
[  5]   4.00-5.00   sec   938 MBytes  7.87 Gbits/sec                 
[  5]   5.00-6.00   sec   941 MBytes  7.89 Gbits/sec                 
[  5]   6.00-7.00   sec   922 MBytes  7.74 Gbits/sec                 
[  5]   7.00-8.00   sec   927 MBytes  7.78 Gbits/sec                 
[  5]   8.00-9.00   sec   930 MBytes  7.80 Gbits/sec                 
[  5]   9.00-10.00  sec   931 MBytes  7.81 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  9.04 GBytes  7.76 Gbits/sec                  sender
[  5]   0.00-10.00  sec  9.04 GBytes  7.76 Gbits/sec                  receiver

iperf Done.
 

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
... just a quick update in case anybody comes across this later or cares... I just tried a brand new cable, a different port on the switch (with a different UBNT branded SFP->Ge transciever), and a built in 10Ge port on the switch with no change. I'll try the new NIC as soon as I can schedule the cluster down with the family (the Plex server is currently active).
 

bferrell

Dabbler
Joined
Dec 10, 2018
Messages
15
So... just to close the loop, it wasn't the Intel card in any event, it was the router being a bottle neck. I have FreeNAS on it's own subnet, and had intended to add a SAN interface to the PVE hosts but hadn't. I picked up a Chelsio T420-BT Dual Port 10GbE off eBay and got the same performance, so I added a 10 GB SAN interface to my ProxMox hosts and now can pretty well max out the connection with over 9+Gbps.
 

Attachments

  • freeNAS-speeds.jpg
    freeNAS-speeds.jpg
    310 KB · Views: 428
Top