Network Bridge not working on 22.02 - RC.2

[Astrodonkey]

This is still severely broken. Can't manage changing the interface or adding a bridge cause it says the IP is already in use. When I test, I lose connectivity. There should be a revert that's automatic if there's no UI response to confirm anymore otherwise what's the point of testing. Have to reboot the node.

I concur. I wasn't able to get any of the above solutions to work on 22.02.4. Disappointing to see the original bug was closed over a year ago with no action taken. I have multiple NICs, wondering if this is somehow related.

In any case, the only way I was able to get this to work was to bypass the GUI entirely by configuring the networking changes using the IPMI interface. Then I had to reboot the node and everything came up as expected.

 

[19norant]

Wanted to throw my experiences out there for anyone who is still fighting this. Because, yes, things are ridiculously painful in the Web GUI.

I am on TrueNAS SCALE 22.12.0

Both approaches us a dual NIC setup.

Slightly successful approach #1:

1. Set one to a static IP.
2. Set one to DHCP.
3. Do everything from the web interface of the DHCP-assigned address. This seems to work as it avoids being affected by any network changes on the NIC with your statically defined IP address.
4. Pour a cocktail.

Horribly unsuccessful approach #2:

1. Be at the terminal. No Web GUI.
2. Using option #1: Configure network interfaces
3. Remove all IP addresses from your two NICs.
4. Create a link aggregation interface: bond0 (I did this with LOAD_BALANCE aggregation)
5. Assign it with your static IP and verify successful network access.
6. A reboot doesn't hurt.
7. Delete the IP from bond0.
8. Create a bridge interface: br0, and assign it your static IP address.
9. Verify success.
10. Pour a cocktail, because you can't deal with the constant messages:
    Code:

    br0: received packet on bond0 with own address as source address

Going to search around a bit on that error and possibly post a new thread to discuss it.

​

---

EDIT: Seems the VM doesn't have outside network access, presumably because bond0 and br0 are too busy yapping at each other and/or dropping packets?​

​

Odd ping results from the VM, too.​

• Ping the internet: No
• Ping the host on br0 IP: Yes
• Ping my laptop on same local subnet: Sporadically (i.e. there is about 20-30% packet loss)
• Ping the local network's gateway: No (presumably related to the first bullet)

The setup from approach #2 is unusable. Probably bond0 + br0 needs a lot of love.​

 

[pcnerd]

I am on TrueNAS SCALE 22.12.0

same here

Slightly successful approach #1:

1. Set one to a static IP.
2. Set one to DHCP.
3. Do everything from the web interface of the DHCP-assigned address. This seems to work as it avoids being affected by any network changes on the NIC with your statically defined IP address.
4. Pour a cocktail.

Not for me, details follow.

Horribly unsuccessful approach #2:

1. Be at the terminal. No Web GUI.
2. Using option #1: Configure network interfaces
3. Remove all IP addresses from your two NICs.
4. Create a link aggregation interface: bond0 (I did this with LOAD_BALANCE aggregation)
5. Assign it with your static IP and verify successful network access.
6. A reboot doesn't hurt.
7. Delete the IP from bond0.
8. Create a bridge interface: br0, and assign it your static IP address.
9. Verify success.
10. Pour a cocktail, because you can't deal with the constant messages:
    Code:

    br0: received packet on bond0 with own address as source address

No cocktails either, sorry!

Odd ping results from the VM, too.

• Ping the internet: No
• Ping the host on br0 IP: Yes
• Ping my laptop on same local subnet: Sporadically (i.e. there is about 20-30% packet loss)
• Ping the local network's gateway: No (presumably related to the first bullet)

The setup from approach #2 is unusable. Probably bond0 + br0 needs a lot of love.​

That's exactly what I observe. Furthermore, my whole VLAN subnet seems to be flooded by TrueNAS SCALE. Network connectivity of all members goes down or at least is heavily affected! However, like you described, ICMP echo requests make it through every now and then, like 5% of pings make it back.

Action protocol so far:
1. create empty bridge with the GUI. Apply. Persist. Remove IP from primary NIC. Add same (or other) IP to the bridge. Apply. --> Connection lost and network flooded.
2. same as No. 1 but through the CLI. Same result.
3. same as No. 2 but with reboots between each step. --> No joy, same result.

I have to physically unplug the network cable from the managed switch. The moment that happens, the network returns to normal operations. Yes, I have STP enabled, but even waiting 5 minutes doesn't change the outcome. Multiple reboots and manual 'ip link set' within the CLI didn't help.

 

[pcnerd]

Addendum: The main managed switch turns off the whole trunk port due to a storm caused by TrueNAS SCALE. I also can see kernel messages written to stdout of TrueNAS SCALE that a broadcast storm / loop is underway.

For clarification: Main managed HP Layer 2 switch ---> smaller HP managed switch ---> Access Point & TrueNAS SCALE
After enabling the bridge: Main managed HP Layer 2 switch >xxxxxx< smaller HP managed switch ---> Access Point & TrueNAS SCALE

 

[jgreco]

Both approaches us a dual NIC setup.

Slightly successful approach #1:

1. Set one to a static IP.
2. Set one to DHCP.

And you had a network storm? Shocking. Absolutely shocking. Why WOULDN'T this result in a storm?

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

You need to only have one connection between a bridge and a switch. You can do this with LACP if your switch supports it, otherwise the use of multiple interfaces connected to a bridge is very likely to result in a storm or other network badness.

 

[pcnerd]

Hi jgreco,
you're replying to another guy.

I don't use LACP (yet) on here. I just want to attach a single NIC to the bridge, which itself has an IP address. It doesn't matter which GBit port I use, same behaviour, meaning a hardware failure is very unlikely.

In the meantime I also tested with and without VLAN (trunk / access port), same result.

For clarification: Once the single NIC isn't member of the bridge but rather logically attached directly (same subnet), nothing out of the ordinary happens.

 

[petervg]

https://jira.ixsystems.com/browse/NAS-114462

I'm also experiencing issues with this.

Using Truenas Scale 22.12.0 and 2 NIC's. One is the onboard realtek (used to connect truenas to the network using a static IP) and the other one is an intel I350 used in a virtual machine running pfSense.

I need to create a bridge to allow other VM's on truenas store data on the truenas SMB shared pools so I followed the official manual:

Setting Up a Network Bridge

Provides instructions on setting up a network bridge interface.

www.truenas.com

I configured the bridge like this:
Type: Bridge
Name: br0
Description: Bridge for VM
DHCP/IPv6 not checked
Bridge members: enp6s0 (the onboard realtek - VM's are also using this one)
Alias IP address: 192.168.111.205/24

When I click on "save", the only thing that happens is the save button being disabled (the text on the button is greyed out and when I move the mouse over the button the mousepointer goes from an arrow to a round circle with a line crossed through) and the only thing I can do is close the settings window. See attached pictures

Since this is clearly a bug I took a look at the jira ticket to see the progress. And I may be wrong here, but as far as I can see, this ticket is closed without changes because of "user configuration error"?

 

[pcnerd]

Since this is clearly a bug I took a look at the jira ticket to see the progress. And I may be wrong here, but as far as I can see, this ticket is closed without changes because of "user configuration error"?

I'm not sure we're talking about the same issue. In your scenario you aren't able to click save, which I believe is caused by the fact that you try to use the same subnet (I could be wrong and I cannot test the theory right now). Try a different subnet for the bridge OR try to remove the IP of your main interface within the same subnet first, then report back.

 

[Ceaser]

I made an account just to bump and add some information to this thread.

I discovered this issue while attempting to transition to TrueNAS Scale. I use core because I have older hardware and the small performance gain I get from core over scale was within a reasonable range for me. I recently purchased a 10gb SFP+ NIC and a set of SSDs to replace my 10+ year old drives. Even though I've gotten used to using Jails over Docker at this point I wanted to go back because Docker has more applicable containers. So initially I setup the 10gb NIC on Core. Everything worked flawlessly, so I started my transition to Scale then I started having problems.

**NOTE**
This configuration works flawlessly on Core as intended but not on Scale
10gb NIC and Bridge serve my computer internet.

Dual 1gb Eth NIC in a Link Aggregation configuration. *Connected to Switch*
10gb SFP+ NIC *Connected to my computer*
The issues start when I bridge Lagg0/Bond0 and SFP+ together.

Lagg0/Bond0 - Static IP
Bridge0/Br0 - DHCP (yes)

Can't access web UI/SSH/SMB or ping TrueNAS with Static or DHCP IP.
**********

Now I've been trying to get this to work on Scale for about 2 weeks and finally gave up and started putting together a different box strictly for Docker. I don't want docker in a VM because the performance is terrible for me.

 

[fpc]

I'm having issues bridging as well. I tried bridging from the Network settings, opening a second window and erasing the IP, then bridging the two interfaces. It just disconnects me until a reboot. I've even tried using a QNAP QSW-M408-2C with bridging built in but it just wont connect either. Does anyone know how to figure this out? Would love to see how dual 10GBS works.

Here is the log file error:

syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:02.2: PCI bridge to [bus 0a]
syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:02.2: bridge window [mem 0xfcf00000-0xfcffffff]
syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:08.1: PCI bridge to [bus 0b]
syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:08.1: bridge window [io 0xe000-0xefff]
syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:08.1: bridge window [mem 0xfcb00000-0xfcefffff]
syslog:Jul 6 20:39:16 truenas kernel: pci 0000:00:08.1: bridge window [mem 0xd0000000-0xe01fffff 64bit pref]
syslog:Jul 6 20:39:36 truenas kernel: audit: type=1400 audit(1688701176.311:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=3162 comm="apparmor_parser"
syslog:Jul 6 20:39:51 truenas kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 1(veth700ea812) entered blocking state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 1(veth700ea812) entered disabled state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 1(veth700ea812) entered blocking state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 1(veth700ea812) entered forwarding state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 2(veth77af8510) entered blocking state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 2(veth77af8510) entered disabled state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 2(veth77af8510) entered blocking state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 2(veth77af8510) entered forwarding state
syslog:Jul 6 20:40:08 truenas kernel: kube-bridge: port 3(veth62ed9202) entered blocking state

 

[Kieros]

31 July 2023 and still this is a pain in the @ss. After running the NAS for over a year without problems I am here again due to some changes in my system I am here just to find out the GUI still sucks at making settings to NICS and bridges with reboots etc... unable to save settings no loading page, no automatic fall back oif the page is not loading and luckely I am not the only one still struggling to get this to work. I lost connection to my VM's. Probably my own doing but trying to get it to work again with making bridges and everything it is a PAIN! I think a NAS should have a proper GUI for setting up networks IP addresses but at truenas this is not important. but probably everyone is stupid. You need to fix it in shell or something while they want you to use the GUI. Make a working GUI then. Instead of this buggy system.

So my 2 cents to this post. Will update when I fixed it.

 

[hallgeir]

I have been struggling with this exact same issue all evening: Whenever removing the IP from the physical interface, adding it to the bridge, and hitting Test changes, it'd just wait until the timeout and revert.

What finally fixed it for me, was to
1. Shut down my VM (I only have one currently)
2. Remove the NIC from the VM (MAY not be needed, but I'm pretty sure I tried just shutting down the VM earlier as well)

And then repeat the process according to the documentation. Then it actually worked. I could then re-add the NIC, using my new bridge, and it's all good.

I hope this can be of any help to anyone, for what it's worth.

 

[Kieros]

I got it solved. As well but dunno how I did it.

 

[AndBu]

I have been struggling with this exact same issue all evening: Whenever removing the IP from the physical interface, adding it to the bridge, and hitting Test changes, it'd just wait until the timeout and revert.

What finally fixed it for me, was to
1. Shut down my VM (I only have one currently)
2. Remove the NIC from the VM (MAY not be needed, but I'm pretty sure I tried just shutting down the VM earlier as well)

And then repeat the process according to the documentation. Then it actually worked. I could then re-add the NIC, using my new bridge, and it's all good.

I hope this can be of any help to anyone, for what it's worth.

This is literally the only solution! THANKS!

 

[Kris Moore]

Glad you got it worked out. It is on our list to create some auto-bridging solution for VMs, but we haven't confirmed when that will land.

 

[AndBu]

Glad you got it worked out. It is on our list to create some auto-bridging solution for VMs, but we haven't confirmed when that will land.

Hey Kris, thanks for your friendly reply. This would be be a real quality of life improvement. In the meantime, it could already be a huge step for everyone trying this to put a big fat warning on the official guide that you have to spin down your VMs and remove their NICs before following the guide.

 

[ABain]

Hey Kris, thanks for your friendly reply. This would be be a real quality of life improvement. In the meantime, it could already be a huge step for everyone trying this to put a big fat warning on the official guide that you have to spin down your VMs and remove their NICs before following the guide.

I'll get this feedback to the docs team; if you have feedback for them in the future, there is a feedback link on the docs pages.

 

[mutenroid]

hi guys,

Yesterday I had the same problem to create a bridge. When I finally created it, I noticed that the k3s cluster apps were without connectivity to internet (impossible to refresh apps catalog, impossible to update apps, etc....)

Logging into truenas via ssh, and launching sudo k3s get pods -A showed that coredns pod was in endless crash-restart-crash-restart.

I had connectivity from the VM, and I could also connect to my shares, but the k3s cluster (and official truenas apps) could not resolve anything. They had no internet access.

Can you think of a way to fix this?

Anyway, it was a nightmare to remove the bridge and leave things as they were.

Thanks in advance
Regards