Forcing Nic ip by mac address or some other method so True doesn't change it

[universal4]

I had been preparing a Truenas install for a while and was finally confident the server was ready to take to the data center. It is a minimum 4 to 5 hour drive round trip, usually turns in to a much longer trip then that so I most often try to have the majority of setup done in advance.

Last thing I had done was change the ip from the 192.168.1.x ip to the true public, add a 3rd ip on port #2 of the 4 available and reboot to make sure all was ok.

Before the lecture starts about not assigning a public ip, if I felt the install was so insecure that it was such a bad thing for the public ip binding, I could easily disable the switch port and only enable it when needed. I like to have public ip's accessible to enable if I feel the need.

I later discovered that the reason the web interface would no longer answer was that the ip's that I (the administrator) assigned to specific nic's have been reordered, and therefore the ip's went to different ports. (now suddenly bge0 is actually bge2) and (bg1 is different and switched to bge3) and (bge 2 is now bge1) and finally (bge3 is bge0) or something similar to that. Thus if in fact the port 0 (first nic) was disabled in the switch, TrueNas might decide it knows a better ip set then I do and reorder the nic's and all of them would be on the wrong vlans talking to the wrong subnets.

Suggesting that different ports be used is not a logical solution for a number of reasons. That would immediately mean that all consistency on the backside of the rack is gone, and the first port on one server is completely different then all the other servers. Any basic logic can easily see that is FAR more risky and much more prone to making cabling mistakes.

Honestly, how many data centers have you ever been in where 5 out of 10 servers have the left nic public and the other 5 the right nic is public?

Also, If there is no way to guarantee port x is ip=x and port=y is ip=y, any reboot could easily mean that port=y connects first and it will be assigned ip=x and port=x will get the remaining ip=y.

Is there any guaranteed method to assign ip's to the ports I WANT, so the ports defined by administrators are always plugged in to the proper vlan's so each of the assigned ip's can reach their desired destinations in the correct subnets? Maybe by mac address? Nic alias?

 

[Chris Moore]

I later discovered that the reason the web interface would no longer answer was that the ip's that I (the administrator) assigned to specific nic's have been reordered, and therefore the ip's went to different ports. (now suddenly bge0 is actually bge2) and (bg1 is different and switched to bge3) and (bge 2 is now bge1) and finally (bge3 is bge0) or something similar to that. Thus if in fact the port 0 (first nic) was disabled in the switch, TrueNas might decide it knows a better ip set then I do and reorder the nic's and all of them would be on the wrong vlans talking to the wrong subnets.

If an IP configuration is statically assigned to a specific NIC, I have never seen it change in one of my systems. How did you make these assignments, in the GUI or from the command line?

 

[garm]

If an IP configuration is statically assigned to a specific NIC, I have never seen it change in one of my systems. How did you make these assignments, in the GUI or from the command line?

I second this, all my IP assignments are static and they never change. You set this up by command line didn’t you?

 

[Samuel Tai]

Actually, TrueNAS may not be at fault here. The BIOS may have reordered the ports before reporting these to the kernel on boot.

 

[universal4]

I assigned the ip's in the text based portion.

What happens, is if any ports are not active at boot, is that the active ports become bge0 and bge1 and the inactive ports become bge 2 and bge 3 even though they are NOT correct in that order.

The bios does not re-order the ports, and they stay in the correct order by bios as well as in the idrac

Based upon some of what I have been reading since I discovered this phenomena, I have seen reports of this happening within FreeBSD. I have never seen this happening with any other OS.

I absolutely need port 1 to be my public, and 2 and 3 to have specific functions on specific vlans so they match the rest of the rack.

I have recreated the issue many times, just by simply unplugging a random nic and rebooting, the inactive port always moves to the end of the list. Method is adding a 192 address to nic one (text based setup) other 3 plugged in to 10 dot, booting, all things ok, then changing ip on nic one to something else, moving the 192 dot to nic 4 and rebooting with nic 1 unplugged, the nic order changes

This install originally started out in the mid range of version 11.2 Release U1, then was upgraded in to the 12's then truenas, so I am not sure if that has any bearing on the discussion or not

 

[universal4]

I downloaded the latest Dell firmware and driver repository from Dell today and ran it to make sure the idrac, lifecycle, bios, Nic etc all had the latest firmware.

I then reset the network configuration back to defaults.

After this process and I booted to verify no ip's were bound I did an ifconfig and verified no nic's were active and none had ip's bound. And just as I had suspected the nic's were in the wrong order, they were 3,4,1,2

So in order to verify that the iDrac card is report all ports and mac addresses correct, I plugged an active cable in one at a time and refreshed the drac to verify the correct order. My ports 1,2,3,4 FreeBSD Ports 3,4,1,2

I am researching if there is a way to define the ports by mac address so I can preserve the correct nic order

 

[no_connection]

Could pcie structure change? Like it's turning off pcie slots for unplugged cables or something?

 

[universal4]

Rebooting with all ports unplugged, or all ports active produces the same results.

I have read about ways that the ports can be named and controlled at boot by mac address, but so far my attempts have been unsuccessful.

 

[universal4]

Ok this all suddenly became more complicated on the hardware side, and something I have never seen happen previously.
It seems that the order of the network cards are in 2 different orders. Those familiar with the iDracs, can see the ports listed under:
Server > System Inventory > Firmware Inventory
Hardware > Network Devices > Integrated Nic 1

They do NOT Match. I was trusting what the Hardware > Network Devices > told me because it lists the cards as (example)
Nic 1 Port 1 Partition 1 - C8:1F:66:F7:31:1F
Nic 1 Port 2 Partition 1 - C8:1F:66:F7:31:20
The above matches exactly physical ports with active link testing.

However Server > System Inventory > Firmware Inventory is the order ifconfig sees at boot, so neither the OS nor TrueNas is changing or manipulating this. (this matches ifconfig)
Broadcom Gigabit Ethernet BCM5720 - C8:1F:66:F7:31:21
Broadcom Gigabit Ethernet BCM5720 - C8:1F:66:F7:31:22

I will have to live with the fact that port 3 and 4 are bge0 and bge1 and bge2 and 3 are ports 1 and 2 on the card. I can obviously provide a friendly name on them so the web interface will be slightly less confusing, since in my opinion that is the lesser of two evils, as the cabling in the data center is the most important portion to keep the consistency.

I have never seen anything like this happen, and have always had 100% accuracy in card names and the order in the drac's have never been different then reality.

I thank everyone who has responded and offered suggestions along the way.

With permission, I will later update this thread if there is a method to adjust the two ordered lists in firmware, in case anyone ever sees such weirdness again. I am awaiting further information from Dell about this.

 

[elvisimprsntr]

BACKGROUND

I apologize in advance for hijacking this thread, but I have a similar issue.

I have a NAS with 4 physical Ethernet ports. When TrueNAS CORE boots up, it appears to configure the Ethernet port names in order of PCI enumeration, which does not match the port number order engraved on the Ethernet ports and MAC address order.

Code:

> lcpci -d 8086:1533

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
09:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0a:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

> pciconf -l

igb0@pci0:5:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb1@pci0:6:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb2@pci0:9:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb3@pci0:10:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00

I was able to add entries to the end of the /boot/device.hints file to align the names with the port number order engraved on the Ethernet ports, which persists on reboot.

Code:

hint.igb.0.at="pci0:6:0:0"
hint.igb.1.at="pci0:5:0:0"
hint.igb.2.at="pci0:10:0:0"
hint.igb.3.at="pci0:9:0:0"

> pciconf -l

igb1@pci0:5:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb0@pci0:6:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb3@pci0:9:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
igb2@pci0:10:0:0:    class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00

There doesn’t seem to be any BIOS setting that affects PCI bus enumeration as I have seen suggested on some forums.


OBSERVATIONS

I noticed the Dashboard still lists the Ethernet ports in order of PCI enumeration.




QUESTIONS

1. Will the tweaks to the device.hints file persist on firmware updates? (I've seen the work on GitHub for FreeBSD, but wasn't sure if it would work on TrueNAS CORE. https://github.com/eborisch/ethname)
2. Is there a better way to force the Ethernet interface names that would persist on firmware updates and/or for the Dashboard to sort on name vs PCI enumeration?


UPDATE 20210701

1. I was not able to figure out how to change the order the interfaces are listed on the dashboard, but I did add more lines to /boot/device.hints to at least remove the unused interfaces from the dashboard.

Code:

hint.igb.2.disabled="1"
hint.igb.3.disabled="1"