Check the bandwidth of LACP bond between TrueNAS and PROXMOX Server

[danialwaheed]

i have just configured an bond between two 10Gig ports in Proxmox server and connected them to the TrueNAS. now i want to verify the link between Proxmox server and TrueNAS is 20Gig using IPerf. Can you please help me how to use IPerf in in Proxmox and TrueNAS to verify my links.

Thanks you in anticipation.

 

[jgreco]

Well, you've made a mistake, because you won't get 20Gig. You would have needed a 40Gig link to be able to get 20Gig. You have a 2x10Gig link. Under the very best optimal and somewhat unlikely conditions, you might luck out and be able to get two simultaneous 10Gig streams going.

Resource - LACP ... friend or foe?

Too many LACP discussions lately. Here's what you need to know. 1) Link aggregation groups are an interesting way to boost network throughput. They're typically managed by a protocol called Link Aggregation Control Protocol (LACP). Link...

www.truenas.com

LACP requires that packets be forwarded consistently so that out-of-order delivery does not happen, and because of this, the fastest a connection can be is the speed of one of the underlying physical connections. You typically need several simultaneous traffic sources to be able to make good use of LACP, and because of the way LACP hashing works, it may not take port numbers into account, in which case, traffic from host A to host B will only exercise one of the available links, which would be 10Gig in this case.

 

[danialwaheed]

Well, you've made a mistake, because you won't get 20Gig. You would have needed a 40Gig link to be able to get 20Gig. You have a 2x10Gig link. Under the very best optimal and somewhat unlikely conditions, you might luck out and be able to get two simultaneous 10Gig streams going.

Resource - LACP ... friend or foe?

Too many LACP discussions lately. Here's what you need to know. 1) Link aggregation groups are an interesting way to boost network throughput. They're typically managed by a protocol called Link Aggregation Control Protocol (LACP). Link...

www.truenas.com

LACP requires that packets be forwarded consistently so that out-of-order delivery does not happen, and because of this, the fastest a connection can be is the speed of one of the underlying physical connections. You typically need several simultaneous traffic sources to be able to make good use of LACP, and because of the way LACP hashing works, it may not take port numbers into account, in which case, traffic from host A to host B will only exercise one of the available links, which would be 10Gig in this case.

Thanks for your help, can you please help me, How to check the Bandwidth of the link using IPerf. TrueNAS has an IP address 192.168.0.15 while Proxmox Server has an IP address 192.168.0.50:8006

 

[Heracles]

now i want to verify the link between Proxmox server and TrueNAS is 20Gig using IPerf.

Impossible, because there is no 20G link between the two...

No matter the load balancing is done based on MAC address or IP address, they are both the same when you look at it between the two same servers. So the test will go only on one of the two path.

What you can test is redundancy : start a continuous PING test between the two. After a few success, unplug one of the two 10G. The PING should continue or recover quickly. Replug that one and unplug the second one. Again, the PING should continue or recover quickly. If PING is not disconnected or recover quickly all the time, your LACP is working. If your connection drops, you have a problem.

But No, we can not help you test something that does not exist.

 

[danialwaheed]

Impossible, because there is no 20G link between the two...

No matter the load balancing is done based on MAC address or IP address, they are both the same when you look at it between the two same servers. So the test will go only on one of the two path.

What you can test is redundancy : start a continuous PING test between the two. After a few success, unplug one of the two 10G. The PING should continue or recover quickly. Replug that one and unplug the second one. Again, the PING should continue or recover quickly. If PING is not disconnected or recover quickly all the time, your LACP is working. If your connection drops, you have a problem.

But No, we can not help you test something that does not exist.

but i am getting these results after running IPerf between the two

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 178 KBytes 1.46 Mbits/sec
[ 5] 1.00-2.00 sec 253 KBytes 2.07 Mbits/sec
[ 5] 2.00-3.00 sec 291 KBytes 2.39 Mbits/sec
[ 5] 3.00-4.00 sec 345 KBytes 2.83 Mbits/sec
[ 5] 4.00-5.00 sec 113 KBytes 926 Kbits/sec
[ 5] 5.00-6.00 sec 242 KBytes 1.98 Mbits/sec
[ 5] 6.00-7.00 sec 298 KBytes 2.44 Mbits/sec
[ 5] 7.00-8.00 sec 414 KBytes 3.39 Mbits/sec
[ 5] 8.00-9.00 sec 396 KBytes 3.24 Mbits/sec
[ 5] 9.00-10.00 sec 421 KBytes 3.45 Mbits/sec
[ 5] 10.00-10.24 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.24 sec 2.88 MBytes 2.36 Mbits/sec receiver

 

[danialwaheed]

but i am getting these results after running IPerf between the two

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 178 KBytes 1.46 Mbits/sec
[ 5] 1.00-2.00 sec 253 KBytes 2.07 Mbits/sec
[ 5] 2.00-3.00 sec 291 KBytes 2.39 Mbits/sec
[ 5] 3.00-4.00 sec 345 KBytes 2.83 Mbits/sec
[ 5] 4.00-5.00 sec 113 KBytes 926 Kbits/sec
[ 5] 5.00-6.00 sec 242 KBytes 1.98 Mbits/sec
[ 5] 6.00-7.00 sec 298 KBytes 2.44 Mbits/sec
[ 5] 7.00-8.00 sec 414 KBytes 3.39 Mbits/sec
[ 5] 8.00-9.00 sec 396 KBytes 3.24 Mbits/sec
[ 5] 9.00-10.00 sec 421 KBytes 3.45 Mbits/sec
[ 5] 10.00-10.24 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.24 sec 2.88 MBytes 2.36 Mbits/sec receiver

it is giving an 2.36 Mbits/sec

 

[Samuel Tai]

Did you disable HW offloading on both sides?

 

[Heracles]

So clearly there is something wrong, but not related to LACP.

Network cable ; Speed and Duplex settings on each port ; problematic hardware (Realtek NIC are terrible) ; ...

 

[danialwaheed]

So clearly there is something wrong, but not related to LACP.

Network cable ; Speed and Duplex settings on each port ; problematic hardware (Realtek NIC are terrible) ; ...

Bond1 is linked to TrueNAS, Kindly check if its correctly configured?

 

[danialwaheed]

Bond1 is linked to TrueNAS, Kindly check if its correctly configured?

if not Please tell me how to link bond1 to TrueNAS. i am having a great trouble configuring this.

 

[Heracles]

LACP is illusion, deception, of no benefit without switch redundancy, all of that at the cost of extra management, risks, problems, bugs, etc. As such, I never configured it in any of my TrueNAS server and I will never do. For that, I can not tell you if you did it right or not.

My only recommendation to you is that you should understand what LACP is (redundant layer 2 paths) and what it is not (not adding the bandwidth of multiple NIC). Once you properly understood that, you will realize that it is of no benefit to you and you will just discard it.

Should you have 2 switches, one NIC in each server plugged in each switch plus a link between these switches, then LACP would offer you network redundancy. For that to work, you will need STP to prevent network loops, 2 smart switches and not just switches for home network, and even more. It will then protect you against the failure of a single network component. Just know that these network components are the most reliable components in your global setup. It is non-sense to increase their already very high availability when you do not have server-level redundancy (single Proxmox server, single TrueNAS server), no data-level redundancy (perfect backups, multiple copies, ...) or application-level redundancy.

Just forget about LACP....

 

[Patrick M. Hausen]

For that to work, you will need STP to prevent network loops

With multi-chassis LACP STP does not come into play. Both links are active, that's the point. But you need "stackable" switches or VSS or some other proprietary technology to get multi-chassis LACP.

 

[Heracles]

But you need "stackable" switches or VSS or some other proprietary technology to get multi-chassis LACP.

Indeed but these are even more expensive....

As I said, better just forget about LACP...

 

[Elliot Dierksen]

if not Please tell me how to link bond1 to TrueNAS. i am having a great trouble configuring this.

I would start with a single link. Once you are getting around 95% of the available hardware speed (950Mbps on a gigabit link), then you can feel good that the network config is solid. As several others have noted, you are never going to get more than the speed of a single link with a LAGG interfaces. It load balances to give you effective bandwidth of multiple physical links, but no one conversation will get more than the speed of a single link.