Having trouble with lagg after upgrading to TN

[Myriad]

Hey folks:

Got a problem with a perfectly working lagg setup in v11 that just will not work in v12 and looking for some clues here. When I upgraded to TN the upgrade was successful - but no network. So I destroyed the lagg moved the cat5 to a non-lacp port and added one nic through the cli and I was up. When I configure the lagg again through the gui it works, but when I reboot again - no network. I have a Supermicro X11SSL-F, but when I upgraded the nic names have been changed to igb0 and em0 as I see that the X11SSL-F uses two different intel nics (?!) with two different drivers. My question is: does the difference in Intel nic ports mess up the lagg functionality on Truenas? It worked in Freenas fine for years but something has changed and I'm damned if I can figure out what? Any and all advice gratefully received.

 

[Spearfoot]

Hey folks:

Got a problem with a perfectly working lagg setup in v11 that just will not work in v12 and looking for some clues here. When I upgraded to TN the upgrade was successful - but no network. So I destroyed the lagg moved the cat5 to a non-lacp port and added one nic through the cli and I was up. When I configure the lagg again through the gui it works, but when I reboot again - no network. I have a Supermicro X11SSL-F, but when I upgraded the nic names have been changed to igb0 and em0 as I see that the X11SSL-F uses two different intel nics (?!) with two different drivers. My question is: does the difference in Intel nic ports mess up the lagg functionality on Truenas? It worked in Freenas fine for years but something has changed and I'm damned if I can figure out what? Any and all advice gratefully received.

Odd that you're seeing two different Intel NICs -- SuperMicro's website shows that the X11SSL-F has two Intel I210-AT ethernet ports plus a separate BMC/IPMI ethernet port:

X11SSL-F | Motherboards | Products | Supermicro

www.supermicro.com

So you should see igb0 and igb1, and possibly a separate BMC/IPMI ethernet port. What devices do you see when you run ifconfig?

 

[Myriad]

Here is the output of ifconfig (I have omitted the vnet data):

Code:

em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 0c:c4:7a:c5:17:49
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether 0c:c4:7a:c5:17:48
        inet6 fe80::ec4:7aff:fec5:1748%igb0 prefixlen 64 scopeid 0x2
        inet6 2607:f2c0:f00e:2d00::d:3 prefixlen 64
        inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

It's weird that the upgrade seems to have changed the net names.

 

[Spearfoot]

Here is the output of ifconfig (I have omitted the vnet data):

Code:

em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 0c:c4:7a:c5:17:49
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether 0c:c4:7a:c5:17:48
        inet6 fe80::ec4:7aff:fec5:1748%igb0 prefixlen 64 scopeid 0x2
        inet6 2607:f2c0:f00e:2d00::d:3 prefixlen 64
        inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

It's weird that the upgrade seems to have changed the net names.

The MAC addresses of the em0 and igb0 devices sure look two sequential ports on the i210-AT. No idea why they'd be assigned different device types.

It shouldn't matter... but it could be that having the two different device types is what's breaking your LAGG setup.

Are you running v12.0-U2.1?

Have you checked your BIOS settings to ensure the Intel NICs are configured identically? (Example: if option ROM is turned off for one, make sure it's turned off for the other; etc.)

 

[Myriad]

I am running TrueNAS-12.0-U2.1. Option ROM is off but I will try toggling it to on to see if it makes a difference. There are no other settings in the bios for the nics that I can see. Thanks for helping me with this. After a few months of trying everything I can think of I think the mis-matched NIC theory is the culprit. I have another client system at a different location with a Supermicro board and the exact same issue, so if I can solve this it will make my Sunday.

 

[Myriad]

Got this message on startup:

Code:

Mar 28 14:15:25 LARWOOD 1 2021-03-28T14:15:25.139844-04:00 LARWOOD.WORKGROUP snmpd 1830 - - Name of an interface changed. Such interfaces will keep its old name in IF-MIB.

Changing nic settings to option ROM didn't do anything (as expected). I will trying updating the bios next...

 

[Spearfoot]

According to the FreeBSD 12.0 release notes:

FreeBSD 12.0-RELEASE Hardware Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

Both ports on your i210-AT NIC should be assigned the em driver, if I understand things correctly.

There may be a boot-time tunable you can use to force the system to assign the same NIC driver to both i210-AT interfaces.

Also, check your BMC/IPMI setup and make sure it's using its own dedicated ethernet port, not sharing w/ the i210.

Other than that... I'm all out out ideas. Perhaps someone with more knowledge will stop by and help out.

Good luck!

 

[Myriad]

Here is the output of pciconf -lv:

Code:

# pciconf -lv
hostb0@pci0:0:0:0:      class=0x060000 card=0x080115d9 chip=0x0c088086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 v3 Processor DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
xhci0@pci0:0:20:0:      class=0x0c0330 card=0x080115d9 chip=0x8c318086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB xHCI'
    class      = serial bus
    subclass   = USB
em0@pci0:0:25:0:        class=0x020000 card=0x153a15d9 chip=0x153a8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I217-LM'
    class      = network
    subclass   = ethernet
ehci0@pci0:0:26:0:      class=0x0c0320 card=0x080115d9 chip=0x8c2d8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
pcib1@pci0:0:28:0:      class=0x060400 card=0x080115d9 chip=0x8c108086 rev=0xd5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:1:      class=0x060400 card=0x080115d9 chip=0x8c128086 rev=0xd5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:      class=0x0c0320 card=0x080115d9 chip=0x8c268086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:      class=0x060100 card=0x080115d9 chip=0x8c528086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'C222 Series Chipset Family Server Essential SKU LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:      class=0x010601 card=0x080115d9 chip=0x8c028086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]'
    class      = mass storage
    subclass   = SATA
none0@pci0:0:31:3:      class=0x0c0500 card=0x080115d9 chip=0x8c228086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
none1@pci0:0:31:6:      class=0x118000 card=0x080115d9 chip=0x8c248086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series Chipset Family Thermal Management Controller'
    class      = dasp
pcib2@pci0:1:0:0:       class=0x060400 card=0x080115d9 chip=0x11501a03 rev=0x03 hdr=0x01
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'AST1150 PCI-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:2:0:0:     class=0x030000 card=0x080115d9 chip=0x20001a03 rev=0x30 hdr=0x00
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
igb0@pci0:3:0:0:        class=0x020000 card=0x153315d9 chip=0x15338086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Seems there is an issue with v12 as it sees the same nic as a different type (igb vs em) though they do have different model #s. Maybe I should file a bug report? This bug could be related.

 

[Myriad]

Update: After installing an Intel dual-nic card and disabling the Supermicro on board cards I see the same behavior noted above (i.e., no connectivity after a reboot). To be sure I also installed the card in a separate test system and installed the latest version of Truenas from scratch and after enabling Lagg and rebooting it works fine. However, when you create a jail - connectivity dies. IMHO, this seems to be a problem with the software not the hardware. I would like to know if ANYONE running Truenas actually has a working system using lagg and if so, what's the secret recipe???