Network card binding

[czerwiec]

Hi,

I built my first FreeNAS 11.3U5 on VMware. I configured IP address, SMB shares and all is working fine.
I want to add iSCSI MPIO so I added two more NICs in VMware.
When I boot up freenas I noticed that my NICs has been reordered. VMX0 keep the IP configuration but MAC address has changed.
The IP configuration is now on Network Adapter 3 in VMware, not NIC 1 in VMware as it was with single NIC config.

Is it possible to bind network cards so they always be in the same place? Or be in the same order as in VMware?

 

[Aterra]

Same issue here.
I don't understand how the NIC ordering is happening.
If I install TrueNAS directly on my Dell R720 the nic order is whacked.
Onboard NIC 1 = eth1 in TrueNAS
Onboard NIC 2 = eth0 in TrueNAS
Onboard NIC 3 = eth4 in TrueNAS
Onboard NIC 4 = eth3 in TrueNAS

When I install TrueNAS into a VM running on ESXi 7.0 with a single nic I get
vmx0 = vmx0 in TrueNAS << all good here

When I add another NIC to teh VM I get:
vmx0 = vmx0 in TrueNAS << all good here
vmx1 = vmx1 in TrueNAS << all good here

Now when I add a 3rd NIC to the VM I get this whacked order:
vmx0 = vmx1 in TrueNAS << bad
vmx1 = vmx0 in TrueNAS << bad
vmx2 = vmx2 in TrueNAS

How do I force specific VMware VMXNet3 NICs to specific interfaces in TrueNAS?

 

[jgreco]

This is a known VMware bug.

https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2047927

It has to do with the manner in which the (virtual) PCI devices are presented to the VM. There's discussion of this on the pfSense forums and the FreeBSD bugtracker. Since it isn't technically a FreeBSD bug, your best bet is to either try one of the workarounds, or complain to VMware.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198406

One of the responses in there describes how to pin PCI ethernet interfaces into a desired order by editing the .vmx file for your VM.

 

[Aterra]

jgreco, that was helpful for the VMware VM. I could not get the vmx changes to stick so I finally fixed it by doing the following:

1. Remove the SCSI controllers & NICS from the VM from within the GUI
2. Remove the scsi0 device and all ethernetX.pciSlotNumber from the vmx.
3. Boot VM once quickly then power off
4. Add ALL the NICS (4 total) I need, boot VM once quickly then power off. The vmx should look like this:
ethernet0.pciSlotNumber = "160"
ethernet1.pciSlotNumber = "192"
ethernet2.pciSlotNumber = "224"
ethernet3.pciSlotNumber = "256"
5. Add all the SCSI controllers back and including the disk(s)

Still weird how TrueNAS installed bare metal directly on my R720 orders the nics.

Thanks!

 

[jgreco]

Still weird how TrueNAS installed bare metal directly on my R720 orders the nics.

That might be something that "lspci" would help with figuring out. No promises. This stuff is sometimes frustrating.

 

[czerwiec]

Sorry for lack of update but I lost hope on answers here. So I was playing with it on my own in the meantime.
As a test, I started a linux VM (centos and ubuntu) and I noticed the same problem with NIC order. I was looking on answers and came across the same links as jgreco.

I thought that pcislotnumber are "static" based on the 2nd link provided by jgreco but it would be too easy. I had some different values.
I tried to do some maths on VMware pci mappings but not much success. So I did very similar as Aterra.
I added the required number of NICs to freenas in VMware. Boot it up, check VMX config file in VMware for ethernetX.pciSlotNumber. Checked the NIC order (based on MAC) in freenas. Power off freenas, unregistered it from VMware (not deleted).
Edit VMX file and change values for ethernetX.pciSlotNumber. Register the VM back to vmware and booted up. All NICs now perfectly matched with VMware.
I did not have to remove any devices from VMware, just simply change values in VMX. It is a bit time consuming but it is working.