How to reset network?

Dunuin · Jul 26, 2021

Hi,

I installed my NAS as FreeNAS-11.2, upgraded it to 11.3 and now I'm on TrueNAS12.0U4.1.
I've got 3 NICs, 12 VLAN interfaces and 9 bridges. I need to add another bridge but I can't change any interfaces because when testing it it always get this error:

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 747, in commit
    await self.sync()
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 1833, in sync
    await self.middleware.call('interface.bridge_setup', bridge)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1241, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1209, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1113, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 53, in bridge_setup
    member_iface.mtu = mtu
  File "netif.pyx", line 773, in netif.NetworkInterface.mtu.__set__
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1198, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 750, in commit
    await self.rollback()
  File "/usr/local/lib/python3.9/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 53, in bridge_setup
    member_iface.mtu = mtu
  File "netif.pyx", line 773, in netif.NetworkInterface.mtu.__set__
OSError: [Errno 22] Invalid argument

Then I wanted to delete all interfaces except for the NICs themselve to add everything again later but I can'T even delete the existing interfaces...
Then I tried to boot into 11.3U3.2 (where I know that changing Interfaces worked because I createm them there) to delete all bridges and vlan interfaces. But if I try to boot into 11.3U3.2 there are no bridges or vlan interfaces at all. Just my 3 NICs and a 4th "unknown" NIC with the IP that one of the other NICs should have. So I can'T delete the bridges/vlans in 11.3 if they don'T exist there.
Then I booted into 12.0U4.1 again and all the bridges/vlans are still there again but I can't delete/add anything...

So is there a way to reset all interfaces without reinstalling the complete TrueNAS or loading it with factory defaults? I don'T want to restore factory defaults because I created like 50 users with permissions, VMs, configured alot of services, around 50 shares, ...

Spearfoot · Jul 26, 2021

Dunuin said:
Hi,

I installed my NAS as FreeNAS-11.2, upgraded it to 11.3 and now I'm on TrueNAS12.0U4.1.
I've got 3 NICs, 12 VLAN interfaces and 9 bridges. I need to add another bridge but I can't change any interfaces because when testing it it always get this error:

Code:
Error: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 747, in commit await self.sync() File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 1833, in sync await self.middleware.call('interface.bridge_setup', bridge) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1241, in call return await self._call( File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1209, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1113, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.9/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 53, in bridge_setup member_iface.mtu = mtu File "netif.pyx", line 773, in netif.NetworkInterface.mtu.__set__ OSError: [Errno 22] Invalid argument During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 137, in call_method result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self, File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1198, in _call return await methodobj(*prepared_call.args) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 973, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 750, in commit await self.rollback() File "/usr/local/lib/python3.9/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 53, in bridge_setup member_iface.mtu = mtu File "netif.pyx", line 773, in netif.NetworkInterface.mtu.__set__ OSError: [Errno 22] Invalid argument

Then I wanted to delete all interfaces except for the NICs themselve to add everything again later but I can'T even delete the existing interfaces...
Then I tried to boot into 11.3U3.2 (where I know that changing Interfaces worked because I createm them there) to delete all bridges and vlan interfaces. But if I try to boot into 11.3U3.2 there are no bridges or vlan interfaces at all. Just my 3 NICs and a 4th "unknown" NIC with the IP that one of the other NICs should have. So I can'T delete the bridges/vlans in 11.3 if they don'T exist there.
Then I booted into 12.0U4.1 again and all the bridges/vlans are still there again but I can't delete/add anything...

So is there a way to reset all interfaces without reinstalling the complete TrueNAS or loading it with factory defaults? I don'T want to restore factory defaults because I created like 50 users with permissions, VMs, configured alot of services, around 50 shares, ...

The member_iface.mtu = mtu error may be a clue: do you have a custom MTU setting for one of your interfaces?

Dunuin · Jul 26, 2021

I managed to remove all network interfaces by deleting all vlans/bridges and also clicking in the "Reset Configuration" button on all 3 NICs.
Then I tried to add all interfaces again. Adding VLAN interfaces to all 3 NICs works fine. There is one ConnectX-3 NIC set to 9000 MTU (VLANs attached to that NIC are MTU 9000 too) and two Intel NICs (i220AT and i217LM) with default 1500 MTU (and VLAN attached to them also set to 1500 MTU). I am able to add bridges (9000 MTU) to the VLANs attached to the Connectx-3 NIC but not to both Intel NICs. As soon as I try to add a bridge to one of the VLANs I get this error. I tried empty MTU and 1500 MTU for NIC/vlan/bridge but still the same error. Adding a bridge directly to one of the Intel NICs works fine. Just looks like there is a middleware bug that doesn't allow me to add bridges to vlan interfaces that are attached to the Intel NICs.

Spearfoot said:
The member_iface.mtu = mtu error may be a clue: do you have a custom MTU setting for one of your interfaces?

I tried both empty MTU fields and changing nothing (so 1500 MTU is used). Always the same error.

Someone knows if it is ok to attach a virtio NIC (from a bhyve VM) to a vlan interface instead of a bridge that is attached to a vlan interace? Looks like that is working because I did it once by mistake. But I always though VM interfaces should only be bridged to bridges. Would that be a workaround if I can' create bridges?

Spearfoot · Jul 26, 2021

Dunuin said:
I managed to remove all network interfaces by deleting all vlans/bridges and also clicking in the "Reset Configuration" button on all 3 NICs.
Then I tried to add all interfaces again. Adding VLAN interfaces to all 3 NICs works fine. There is one ConnectX-3 NIC set to 9000 MTU (VLANs attached to that NIC are MTU 9000 too) and two Intel NICs (i220AT and i217LM) with default 1500 MTU (and VLAN attached to them also set to 1500 MTU). I am able to add bridges (9000 MTU) to the VLANs attached to the Connectx-3 NIC but not to both Intel NICs. As soon as I try to add a bridge to one of the VLANs I get this error. I tried empty MTU and 1500 MTU for NIC/vlan/bridge but still the same error. Adding a bridge directly to one of the Intel NICs works fine. Just looks like there is a middleware bug that doesn't allow me to add bridges to vlan interfaces that are attached to the Intel NICs.

I tried both empty MTU fields and changing nothing (so 1500 MTU is used). Always the same error.

Someone knows if it is ok to attach a virtio NIC (from a bhyve VM) to a vlan interface instead of a bridge that is attached to a vlan interace? Looks like that is working because I did it once by mistake. But I always though VM interfaces should only be bridged to bridges. Would that be a workaround if I can' create bridges?

Glad you got past the initial problem!

Your network is more complicated than mine. All of my VLAN setup is on pfSense; my FreeNAS servers have a pair of interfaces on two separate VLANS, with no explicit VLAN setup or bridging involved on FreeNAS itself.

Perhaps someone with more experience can help you with your bridge setup.

Dunuin · Jul 26, 2021

My setup is so complicated because I got a OPNsense VM routing between 9 VLANs. And another OPNsense VM is running on a Proxmox server and both OPNsense VMs are working together in HA mode so both OPNsenses need the same NIC names for all VLANs. FreeBSD bridges can't work with vlan tagged traffic but I need virtual NICs so the interfaces names match on both servers. So the only way I got that to work was to create one VLAN interface for each VLAN and one bridge for each VLAN interface on the TrueNAS host so the OPNsense VM only needs to work with untagged traffic.
Because both hypervisors are using KVM with virtio NICs the NIC names are matching for all 9 virtual NICs.

Patrick M. Hausen · Jul 26, 2021

If you want to create the bridges manually - as I recommend - you need to boot the system in a state where no automatically created bridge is present. So disable autostart for all jails, plugins and VMs. Reboot, then go ahead with your configuration. Make sure to move all IP address configuration from the VLAN interface to the bridge interface if there is a bridge on that VLAN. This is not optional.

Once that works, re-assign and start your jails, VMs, etc.

Dunuin · Jul 26, 2021

Patrick M. Hausen said:
If you want to create the bridges manually - as I recommend - you need to boot the system in a state where no automatically created bridge is present. So disable autostart for all jails, plugins and VMs. Reboot, then go ahead with your configuration. Make sure to move all IP address configuration from the VLAN interface to the bridge interface if there is a bridge on that VLAN. This is not optional.

Once that works, re-assign and start your jails, VMs, etc.

How would I do that?
Creating a rc.conf tunable in the WebUI like this for a bridge named bridge43 that is attached to a vlan interface named vlan43?

Code:

cloned_interfaces="bridge43"
ifconfig_bridge43="addm vlan43 up"

One of the bridges also would need to be my gateway for the server? How would I tell TrueNAS to use one of the bridges to reach my gateway? In my case it would be bridge43 with IP 192.168.43.10 that needs to use 192.168.43.1 as the gateway.
Do I also need to manually create the vlan interfaces because maybe rc.conf tunables are initialized before the GUI adds the vlan interfaces?

Samuel Tai · Jul 26, 2021

No, you create bridges and VLANs under Networks->Interfaces.

Dunuin · Jul 26, 2021

Samuel Tai said:
No, you create bridges and VLANs under Networks->Interfaces.

Thats excatly whats not working because I always get that error if I want to save any changes and there is a bridge attached to any vlan present...

Patrick M. Hausen · Jul 26, 2021

That is probably because there already are automatically created bridges. That's why I recommended rebooting with all jails, plugins and VMs off, to get rid of them.

If you don't run any of these virtualisation features, I might be on the wrong track. I would still like you to check with ifconfig if there is any bridge alreay there.

Dunuin · Jul 26, 2021

I already disabled all VMs and I'm not using any jails/plugins and rebooted several times.

ifconfig:

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: SYNC NIC
        options=802000<WOL_MAGIC>
        ether XX:XX:XX:XX:XX:XX
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: 1G NIC
        options=800020<JUMBO_MTU>
        ether XX:XX:XX:XX:XX:XX
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
mlxen0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether XX:XX:XX:XX:XX:XX
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan51: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: FFLANNAS vlan
        options=80000<LINKSTATE>
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 51 vlanpcp: 1 parent interface: mlxen0
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan49: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: LANNAS vlan
        options=80000<LINKSTATE>
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 49 vlanpcp: 1 parent interface: mlxen0
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan48: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: TORNAS vlan
        options=80000<LINKSTATE>
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 48 vlanpcp: 1 parent interface: mlxen0
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan45: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: DMZNAS
        options=80000<LINKSTATE>
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 45 vlanpcp: 1 parent interface: mlxen0
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan43: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN vlan
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.43.10 netmask 0xffffff00 broadcast 192.168.43.255
        groups: vlan
        vlan: 43 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN vlan
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 2 vlanpcp: 3 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan41: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: RETRO vlan
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.41.4 netmask 0xffffff00 broadcast 192.168.41.255
        groups: vlan
        vlan: 41 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan42: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: DMZ vlan
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.42.9 netmask 0xffffff00 broadcast 192.168.42.255
        groups: vlan
        vlan: 42 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan44: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: MNGT vlan
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 44 vlanpcp: 3 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan46: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: TOR vlan
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 46 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan47: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: FFLAN vlan
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 47 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vlan50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IOT vlan
        ether XX:XX:XX:XX:XX:XX
        groups: vlan
        vlan: 50 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge51: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: FFLANNAS bridge
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.51.4 netmask 0xffffff00 broadcast 192.168.51.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan51 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge49: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: LANNAS bridge
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.49.4 netmask 0xffffff00 broadcast 192.168.49.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan49 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge48: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: TORNAS bridge
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.48.4 netmask 0xffffff00 broadcast 192.168.48.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan48 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge45: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: DMZNAS bridge
        ether XX:XX:XX:XX:XX:XX
        inet 192.168.45.4 netmask 0xffffff00 broadcast 192.168.45.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan45 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: SYNC bridge
        ether XX:XX:XX:XX:XX:XX
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

em0 and igb0 are the Intel NICs where I'm not able to add any bridges to VLAN interfaces attached to them.
bridge4 works because I directly bridge it the the NIC without a vlan interface.
bridge45, bridge48. bridge49 and bridge51 are working attached to vlan interfaces because they use the Mellanox NIC with MTU 9000.
bridge2, bridge41, bridge42, bridge43, bridge44, bridge46, bridge47 and bridge50 I can not create because the middleware is crashing while saving it.

Patrick M. Hausen · Jul 27, 2021

Please use code tags like so - your output is really hard to read:

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: SYNC NIC
options=802000<WOL_MAGIC>
ether XX:XX:XX:XX:XX:XX
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
[...]

em0 is already part of bridge4, you cannot add VLANs if that is the case:

Code:

bridge4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: SYNC bridge
ether XX:XX:XX:XX:XX:XX
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 55
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

Not quite sure about the igb0 case. You are trying to create bridges with the VLANs as members, correct? Not igb0 itself?

Have you disabled hardware offloading for all physical interfaces that carry VLANs and/or bridges?

Dunuin · Jul 27, 2021

Patrick M. Hausen said:
Please use code tags like so - your output is really hard to read:

Sorry, that was supposed to be a code and not a quote block.

Patrick M. Hausen said:

I know. em0 actually doesn't need tagged vlan and untagged vlan is working fine there. Just adding bridges directly to the physical NICs is working. The problem is adding bridges to vlan interfaces because that always results in a middleware crash then trying to test the config and without a successful test and can't save any changes. So "em0 -> bridge2" is working but not "em0 -> vlan2 -> bridge2". Same for igb0.

Patrick M. Hausen said:
Not quite sure about the igb0 case. You are trying to create bridges with the VLANs as members, correct? Not igb0 itself?

Thats right. Like "igb0 -> vlan41 -> bridge41".

Patrick M. Hausen said:
Have you disabled hardware offloading for all physical interfaces that carry VLANs and/or bridges?

Yes I tried both. With and without an enabled hardware offloading checkbox. I also tried it with and without "-rxcsum -txcsum -rxcsum6 -txcsum6 -tso -lro -vlanmtu -vlanhwtag -vlanhwfilter -vlanhwtso -vlanhwcsum" as additional options to really disable all HW offloading.

I've got a modified middleware file from the support and after replacing the 12.0U4.1 version with it, it actually worked to create these bridges.
Now I need to check if the network is still working. And I hope they tell me what to do so I can test a little bit more so that bug can be fixed with the U5 update.

Patrick M. Hausen · Jul 27, 2021

Glad you are making progress. Definitely looks like a bug. Works - and always did - in my machine

Make sure to move all IP configuration to the bridge interfaces like in my screenshot.

Bildschirmfoto 2021-07-27 um 19.28.42.png

Dunuin · Jul 27, 2021

Patrick M. Hausen said:
Glad you are making progress. Definitely looks like a bug. Works - and always did - in my machine
Make sure to move all IP configuration to the bridge interfaces like in my screenshot.

Thanks. I did that.
Until now it looks like services and VMs are working.

Still no clue what caused the error. I looked into that edited middleware file and it only should add some additional debug information and not fix anything. So no idea why it worked this time. I didn't changed anything except for replacing the middleware file. And I wasn't able to find a workaround for a whole week and I rolled back to 11.3 and 12.0U4.1 again, disabled all services, disabled all VMs/jails/plugins, set all NICs to defaults, deleted and added all vlans/bridges, rebooted a dozen times...all without fixing it. So I hope we find the problem so it can be fixed.

Important Announcement for The TrueNAS Community.

How to reset network?

Dunuin

Contributor

Spearfoot

He of the long foot

Dunuin

Contributor

Spearfoot

He of the long foot

Dunuin

Contributor

Patrick M. Hausen

Hall of Famer

Dunuin

Contributor

Samuel Tai

Never underestimate your own stupidity

Dunuin

Contributor

Patrick M. Hausen

Hall of Famer

Dunuin

Contributor

Patrick M. Hausen

Hall of Famer

Dunuin

Contributor

Patrick M. Hausen

Hall of Famer

Dunuin

Contributor

Similar threads

Important Announcement for The TrueNAS Community.