Hi all,
Before I begin - I feel I know the answer to this is likely going to be "ubiquiti fail" - but wanted to bounce my issue past the experts. The players:
Both the XCP-NG and TrueNas boxes are connected to the switch with trunked ports configured for Aggregation (that's what Ubiquiti calls it) which by default enabled LACP (i.e. 802.3ad). As far as link-status goes, both are online with my Mac Studio being able to connect to both boxes, and both boxes can connect to each other.
The problem, however, is that there is severe inconsistence with speed performance with the TrueNas box. When I have both interfaces up and running, my iperf3 results between TrueNas and XCP, as well as TrueNas and Mac Studio, are severely inconsistent - for example:
I can re-run this test many times, either with the Mac Studio or the XCP box and and the results would be different. Mostly, though, they are on the sub gig side, often under 300 mbit.
If I simply disconnect one of the fiber patch cables from the switch and default TrueNas into a single NIC (with aggregation still enabled), I will consistently get 9+ gbit/s. Doesn't matter which patch cable I use or variation of SFP arrangement, the results are the same -- two NIC enabled = fail; one NIC "disconnected" one "connected" = win. The results, btw, are the same whether the TrueNas box acts as a server or client.
Between the Mac Studio and XCP - I consistently get 8.5-9 gbit/s (running iperf3 in a VM so some overhead) - again, regardless of who is server or client.
I confirmed that LACP is not running in strict mode on TrueNas; I've tried enabling it - but then TrueNas will no longer pickup DHCP address and overall exhibits bizarre behavior.
The one extra bit of info to share is that before I got the 10 gig switch and the mellanox card, TrueNas was connected, via LACP and its on-board gig ports, to my Ubiquiti USW-24-Pro without any issues and consistent speed results.
If anyone has any ideas or suggestions, maybe experiencing a similar issue - I am all ears (and eyes haha). Ultimately I don't mind going single NIC; really the only reason I have LACP is because my rack is a pain to work in... but prefer to aggregate when I can :D
Thanks!
Before I begin - I feel I know the answer to this is likely going to be "ubiquiti fail" - but wanted to bounce my issue past the experts. The players:
- HPE Microserver Gen 8 running TrueNas with a Mellanox-2 dual SFP+ card, with 2x 10 Gig SFPs.
- Ubiquiti USW-AGG 8-port 10 gig aggregation switch with the same 2x10 gig SFPs
- Mac Studio with 10 gig connection, connected to the same switch.
- Xeon based XCP-NG server with 2x RJ-45 10 gig ports connected to the same switch.
Both the XCP-NG and TrueNas boxes are connected to the switch with trunked ports configured for Aggregation (that's what Ubiquiti calls it) which by default enabled LACP (i.e. 802.3ad). As far as link-status goes, both are online with my Mac Studio being able to connect to both boxes, and both boxes can connect to each other.
The problem, however, is that there is severe inconsistence with speed performance with the TrueNas box. When I have both interfaces up and running, my iperf3 results between TrueNas and XCP, as well as TrueNas and Mac Studio, are severely inconsistent - for example:
root@fuzznas:~ # iperf3 -c 10.32.0.10
Connecting to host 10.32.0.10, port 5201
[ 5] local 10.32.0.22 port 18682 connected to 10.32.0.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 23.1 MBytes 191 Mbits/sec 205 1.41 KBytes
[ 5] 1.01-2.00 sec 12.9 MBytes 109 Mbits/sec 128 106 KBytes
[ 5] 2.00-3.01 sec 5.91 MBytes 49.2 Mbits/sec 67 4.28 KBytes
[ 5] 3.01-4.01 sec 10.0 MBytes 84.4 Mbits/sec 100 11.3 KBytes
[ 5] 4.01-5.00 sec 7.14 MBytes 60.1 Mbits/sec 69 7.08 KBytes
[ 5] 5.00-6.01 sec 8.69 MBytes 72.3 Mbits/sec 94 7.11 KBytes
[ 5] 6.01-7.01 sec 8.67 MBytes 73.1 Mbits/sec 83 4.27 KBytes
[ 5] 7.01-8.00 sec 11.1 MBytes 94.1 Mbits/sec 109 14.2 KBytes
[ 5] 8.00-9.00 sec 2.51 MBytes 21.0 Mbits/sec 33 25.7 KBytes
[ 5] 9.00-10.01 sec 6.94 MBytes 57.6 Mbits/sec 51 9.96 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 97.0 MBytes 81.3 Mbits/sec 939 sender
[ 5] 0.00-10.01 sec 96.8 MBytes 81.1 Mbits/sec receiver
iperf Done.
root@fuzznas:~ # iperf3 -c 10.32.0.10
Connecting to host 10.32.0.10, port 5201
[ 5] local 10.32.0.22 port 47229 connected to 10.32.0.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 480 MBytes 4.03 Gbits/sec 68 124 KBytes
[ 5] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 53 242 KBytes
[ 5] 2.00-3.00 sec 354 MBytes 2.97 Gbits/sec 54 80.6 KBytes
[ 5] 3.00-4.00 sec 354 MBytes 2.96 Gbits/sec 68 71.1 KBytes
[ 5] 4.00-5.00 sec 369 MBytes 3.10 Gbits/sec 64 48.3 KBytes
[ 5] 5.00-6.00 sec 244 MBytes 2.05 Gbits/sec 59 111 KBytes
[ 5] 6.00-7.00 sec 335 MBytes 2.81 Gbits/sec 56 92.6 KBytes
[ 5] 7.00-8.00 sec 398 MBytes 3.34 Gbits/sec 69 151 KBytes
[ 5] 8.00-9.00 sec 359 MBytes 3.01 Gbits/sec 52 76.5 KBytes
[ 5] 9.00-10.00 sec 377 MBytes 3.15 Gbits/sec 80 65.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.52 GBytes 3.02 Gbits/sec 623 sender
[ 5] 0.00-10.01 sec 3.52 GBytes 3.02 Gbits/sec receiver
I can re-run this test many times, either with the Mac Studio or the XCP box and and the results would be different. Mostly, though, they are on the sub gig side, often under 300 mbit.
If I simply disconnect one of the fiber patch cables from the switch and default TrueNas into a single NIC (with aggregation still enabled), I will consistently get 9+ gbit/s. Doesn't matter which patch cable I use or variation of SFP arrangement, the results are the same -- two NIC enabled = fail; one NIC "disconnected" one "connected" = win. The results, btw, are the same whether the TrueNas box acts as a server or client.
Between the Mac Studio and XCP - I consistently get 8.5-9 gbit/s (running iperf3 in a VM so some overhead) - again, regardless of who is server or client.
I confirmed that LACP is not running in strict mode on TrueNas; I've tried enabling it - but then TrueNas will no longer pickup DHCP address and overall exhibits bizarre behavior.
The one extra bit of info to share is that before I got the 10 gig switch and the mellanox card, TrueNas was connected, via LACP and its on-board gig ports, to my Ubiquiti USW-24-Pro without any issues and consistent speed results.
If anyone has any ideas or suggestions, maybe experiencing a similar issue - I am all ears (and eyes haha). Ultimately I don't mind going single NIC; really the only reason I have LACP is because my rack is a pain to work in... but prefer to aggregate when I can :D
Thanks!