SMB multi-channel crippled by TrueNAS NIC config

[Sawtaytoes]

I have 4 NICs in my NAS. I want to use all 4 for SMB multichannel, but as soon as I try to add a second one on the same network, it complains:

You're not supposed to use LAGG, so what do I do?

As a TrueNAS user, it's frustrating getting conflicting info on arbitrary (and unnecessary) restrictions on my own freakin' NAS versus the capabilities of the OS itself.

My personal computer runs Windows and has 4 NICs. 3 of them are on this network, 1 of them is assigned to a VLAN. Why is that bad? Windows prioritizes sending data out of the 25Gb fiber links rather than the one slower Ethernet link I use as a backup. I can't imagine why TrueNAS wouldn't also do this.

 

[ragametal]

OK, I'm not an expert on networking and have no experience with SMB multichannel but i do have 2 NICs connected into the same network without LAGG and have no issues.

The trick is to disable static IP and DHCP on the secondary NIC. That way both NICS can be physically connected to the network but no traffic would be flowing on the second NIC unless you specifically say so.

If not, your TrueNAS system may end up with multiple IPs (one for each NIC) and your router will not know which IP to use to communicate with TrueNAS.

The following are the settings for my secondary NIC.

I hope this helps.

 

[Patrick M. Hausen]

One cannot have two NICs with different IP addresses in the same network. Period. IP does not work that way.

 

[somethingweird]

More details why -

One cannot have two NICs with different IP addresses in the same network. Period. IP does not work that way.

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

truenas.com

 

[HoneyBadger]

Try putting the interfaces from the TrueNAS side into a LAGG, and assign multiple IP aliases in the same subnet to the bonded interface.

 

[jgreco]

As a TrueNAS user, it's frustrating getting conflicting info on arbitrary (and unnecessary) restrictions on my own freakin' NAS versus the capabilities of the OS itself.

You seem to be a bit misinformed. You can do just about anything with TrueNAS that is legitimate in FreeBSD or Linux. Two network interfaces on the same IP subnet is not legitimate. While you can "do" it with both FreeBSD and Linux, it will not work the way most people would expect, and it almost certainly won't work in a useful manner. I would submit that this means that "the capabilities of the OS itself" does not include janky broken multiple interface configurations.

Set up link aggregation and do your networking correctly, as mentioned above and in the linked article.

 

[Sawtaytoes]

It's not me who came up with this non-LAGG idea, it's TrueNAS:

I used to use LAGG, but it says SMB3 Multichannel doesn't work with it, so I turned it off to benefit from imporved SMB speeds.

Someone posted last year that my ConnectX-6 cards have a way of combining both adapters into 1 (not the same as LAGG), to make it look like 1 interface. Not sure what that is or if it's event relevant to me. Maybe LAGG is the only way.

 

[jgreco]

So the apparent conclusion is that if IP networking doesn't work correctly with multiple network interfaces on a single network (so use LAGG), and that SMB multichannel doesn't function if you use LAGG, then what you need is network interfaces on separate networks. This sounds familiar from previous discussions of SMB multichannel: use more than one network. I'm not a Samba guy but it's what you HAVE to do for protocols like iSCSI in many cases.

If you can get your ConnectX-6 cards to present multiple physical interfaces as a single logical interface of some sort, that's probably just as good. The issue is with how the IP stack interacts with the network. You cannot have two logical interfaces on a single IP network as it breaks the proper functioning of ARP and other stuff as outlined in the article linked above. If it only appears as a single logical interface, you should be good to go.

 

[SteveinSea]

To actually implement SMB Multichannel, are people using VLANs or physically separate router/switch ports? It's been ~20 years since I read the IP spec, but my memory seems to recall that (without LAGG) you couldn't even put 2 NICs in one machine connected to the same physical network device (a switch with VLANs) and comply with the spec.

For example, I have Chelsio 6225-CR dual 25 Gbps NICs in several machines at home. I have these machines setup with DAC direct attach to a backup server and the other interface plugged into a 25 Gbps switch port with VLANs. If I wanted to enable SMB Multichannel for one of these systems, can I plug the 2nd Chelsio port into another 25 Gbps switch port (same switch) on a new VLAN? If that doesn't work, will it work in the same physical network device if the ports are routed instead of switched? Or, for a clean install, do I need to run 1 wire from the Chelsio to switch 1 VLAN 1, a second wire from Chelsio to switch 2, VLAN 2, and then let network routing direct a client's dual NICs in the same way?

 

[jgreco]

It's been ~20 years since I read the IP spec, but my memory seems to recall that (without LAGG) you couldn't even put 2 NICs in one machine connected to the same physical network device (a switch with VLANs) and comply with the spec.

This is correct! And a point often disputed fruitlessly by those who feel they "know better". We have an entire resource on the topic, and I get tired of debating those who feel they know better without ever having researched the topic. So I don't really have anything to add here other than to be amazed at seeing this gem of wisdom in a post by a new user as part of his second post. Welcome to the forums, I feel like you will do well here.