Trouble with 10Gb bridge setup using TrueNAS GUI

[ZiggyGT]

I could really use some help. I setup a bridge using the GUI. I have 3 Mellanox X2 dual port cards. System B connected to my switch with a fiber cable. System B is connected to my server. That server has Mellanox X3 dual port card and one port connected to an Aruba switch and the other to System B. System B performs perfectly when connected but does not work after I create the bridge on system B. Attached is a copy of the GUI screen and the ifconfig showing that the bridge is setup. I cannot ping the 192.160.0.60 address from any remote system. I can talk to the gui on address 192.168.1.94 from a system on that subnet.

I always have the issue that I cannot configure the bridge using the GUI to. When the network interface gui goes into test mode, the network reconfigures, and I cannot press the save button because it is no longer on that address. I cannot configure two interfaces to the same subnet. I connect to the GUI on the x.x.1.x subnet. This does not get reconfigured and I can save the config. The bridge is then persistent. I can configure the address I want through the 192.168.1.94 address. I have had this setup working in the past but I reloaded SW and now I cannot get it to connect.
I am about ready to give up on setting the bridge via the gui and go back to a script or buy a darn switch. I would really like to get this to get setuo reliably. I was hoping for some help.

 

[sretalla]

I think it's going to be necessary to see some kind of diagram of what you're trying to achieve in order to help you.

You're connected over the 192.168.1.x network from your client and you're connecting to the remote system using the bridge/mellanox cards on the 192.168.0.x network, so I don't see why you wouldn't be able to continue connection on the 192.168.1.x network after any changes (unless you're creating a network loop/storm with the bridge).

 

[ZiggyGT]

I think it's going to be necessary to see some kind of diagram of what you're trying to achieve in order to help you.

You're connected over the 192.168.1.x network from your client and you're connecting to the remote system using the bridge/mellanox cards on the 192.168.0.x network, so I don't see why you wouldn't be able to continue connection on the 192.168.1.x network after any changes (unless you're creating a network loop/storm with the bridge).

Thanks for responding. Here is my diagram. I am connecting from my windows client 192.168.1.78 to the 192.168.1.94 1Gbs port on my server to create the bridge on my working network 192.168.0.x . When I do it this way I am always connected so I can respond after the network changes are tested. I set the bridge created in the gui to fixed IP 192.168.0.60 because I don't want my shared drives to change. the problem is that I could not access the network from the 10Gbs clients on the bridged ports.

I tried to diagnose the problem. I connected the clients directly to the main server 10Gbs port. everything was fine.
I then decided to manually set up the bridge with the shell commands. This is not permanent but I thought might help. I examined the script I used to use in FREENAS 11. It included specifically declaring each port in the bridge as up. "ifconfig mlxen0 up" ifconfig mlxen1 up" etc
Once I did this the bridge worked perfectly.
On each reboot I have to repeat this using shell or create a script. I don't know if the bridge will be created before the script runs. That is a project for tomorrow. I had tried to setup the bridge on some solarflare cards. that did not work. perhaps it was the same issue. I'll try that again because the solarflare physically works better.

It is wierd that I do not have this issue with a bridge created with a single mellanox x-3 card (partly why I was stumped) but I do have it with a bridge created with 3 mellanox x-2 cards. My problem is solved once I add the script. I think it is likely a bug that the multiple X2 cards are not "up" after the bridge creation. Is it because they are X-2 or is it because there are 3 of them? I'll find out tomorrow after some testing.

On the windows clients the network has to be disabled/enabled to see the network if the windows machine is up first.

This is the script info. https://www.truenas.com/community/threads/bridge-between-network-interfaces.41566/page-2#post-570639
One of the comments in this thread seems to say I can add the "up" to the bridge creation. I'll give that try as well

Update: I added the up command in each of the mlxen0-mlxen5 ports in the Interfaces section of the Truenas GUI. Now the bridge works properly. The system running the bridge SW should be up before the clients to connect properly. if it is not, I had to enable/ disable the port on my Windows client. On a connected Truenas machine I had to reboot it. I did some brief performance testing, looks like I have some work to do.

 

[sretalla]

One of the comments in this thread seems to say I can add the "up" to the bridge creation. I'll give that try as well

You can put "up" in the Options on the bridge interface in the GUI if that's somehow not happening automatically.

 

[ZiggyGT]

You can put "up" in the Options on the bridge interface in the GUI if that's somehow not happening automatically.

That was not sufficient. The bridge does start in the up state.
I added the up command in each of the mlxen0-mlxen5 ports in the Interfaces section of the Truenas GUI. Now the bridge works properly. I had no trouble with a bridge on a different machine which is why this was puzzling. I am wondering if it is related to the fact that I am using multiple network cards in this system and that system has a single dual channel network card.

The system running the bridge SW should be up before the clients to connect properly. on a different Truenas system. if it is not, I had to enable/ disable the port on my Windows client to get it connected. On a connected "client"Truenas machine I had to reboot it. I did some brief performance testing, looks like I have some work to do. The Windows machine has a dog of a disk. I was only able to get 115MB/s transferring a big file

 

[jgreco]

I am about ready to give up on setting the bridge via the gui and go back to a script or buy a darn switch. I would really like to get this to get setuo reliably. I was hoping for some help.

You're going about this the wrong way.

Set up an "administrative" interface on one of the igbX interfaces. Set it at an IP range OUTSIDE your existing ranges; let's just suggest 10.0.0.1/24 for the NAS and 10.0.0.2/24 for a laptop or PC.

Now log in and set up your bridge stuff and other interfaces. This makes it a lot easier to work with the system and commit the changes you intended to make.

You will want to log in and keep an eye on the interface states. You may need to add "up" to the ifconfig flags for interfaces, which you seem to have some grasp of.

Also be aware that the ConnectX-2 cards are not recommended.

 

[ZiggyGT]

My administrative port is 192.168.1.94/24 connecting to 192.168.1.78/24 so it is on a different subnet, and ideally related to the assigned static IP of the node. Everything else I have is 192.169.0.x
What is different with the 10.0.0.1/24 and 10.0.0.2/24 combo?
I purchased the Mellanox x2 cards some time ago. What is the bad rap for the Mellanox cards? At one time it seemed to be the card people were using with Freenas. Chelsio and Intel seem to be the current in favor cards.

 

[jgreco]

What is different with the 10.0.0.1/24 and 10.0.0.2/24 combo?

When you test changes through the GUI, the system unconfigures the existing network (by running deconfiguration commands in the reverse order) which is supposed to return the system to a state where there is not a live network configuration. Then the system configures the system with the new configuration being tested. As you noted, if you change IP addresses or bridge configurations, you only have a limited time to confirm the new config, and this can be difficult to do if you have to move cables or adjust your PC/laptop network configuration.

If, instead, you have (as an example) 10.0.0.1/24 set up on your NAS on igb1 and hook that up to a PC numbered at 10.0.0.2/24, when you make all your changes to the 10G bridging stuff and commit it, it will deconfigure igb1 but then immediately reconfigure igb1 with the same address, so that when you hit the button to confirm the change, your laptop or PC has no trouble reaching the NAS and confirming the GUI prompt.

There's nothing "different" or magic about it except that you're using a configuration strategy that allows you to maintain network connectivity to the NAS.

I purchased the Mellanox x2 cards some time ago. What is the bad rap for the Mellanox cards?

They're PCIe 2.0 cards, which means that a dual port card does not have sufficient PCIe bandwidth. Power consumption for a dual isn't too terrible at about 7 watts, but the chipset support is really designed as a dual role card that is primarily focused on Infiniband and has Ethernet thrown in as a bonus. The silicon put out by Intel and Chelsio is really optimized for ethernet with acceleration features to match. Some people have reported problems using bridging on the Mellanox, which may be due to how the driver handles vlan tagging or other stuff. I don't have a specific defect list because my suggestion is normally just to use a card known to do all the stuff correctly. Finally, well, y'know, Nvidia, eugh.

I generally refer to Mellanox as the Realtek of the 10G world. Yes, it is often sufficient for baseline needs, but it isn't a real good card compared to more recent developments. Yes I realize that you probably got your cards for $5 a pop which is usually why people end up with these cards here.

At one time it seemed to be the card people were using with Freenas

Weird that I do not remember such a time. I wrote the 10 Gig Networking Primer in 2014 and at the time it was slanted very heavily in Chelsio's favor because Intel had a driver issue on FreeBSD 9 that prevented it from being stable.

Mellanox had dropped official support for the ConnectX-2 by 2015 (see the specific post at https://www.truenas.com/community/threads/10-gig-networking-primer.25749/page-7#post-275533 and my admittedly critical comments in the next few posts) or if you don't mind going right to the source, Archive.org has conveniently archived it:

Wayback Machine

1.3 Unsupported Functionality/Features
The following are the unsupported functionalities/features in FreeBSD Rev 2.1.6:
• ConnectX®-2 Adapter cards

In FreeBSD, there was some hackery done to support the ConnectX-2 cards under the mlx4en driver (really for the ConnectX-3 cards) but I don't recall the specifics any longer. I remember thinking it seemed bad at the time.

Chelsio and Intel seem to be the current in favor cards.

Since around 2011 (for Chelsio) or a few years later (for Intel). The Intel support is top notch due to Intel's driver team who polished that thing for years. That might no longer be the case for newer chipsets; Intel's had some financial issues in recent years. Still, for the cards we're discussing, you could go either way. If you need budget cards, I would also consider Solarflare SFN6122.

But the other thing is, in the last ten years, prices on 10G switchgear has dropped. You are not going to get great performance out of your NAS abused as a switch. Why not just buy a small 10G switch? The Mikrotik CRS309-1G-8S+IN switch is usually about $250 and used gear on eBay is sometimes even cheaper.

 

[ZiggyGT]

Thanks for the complete response with references. I am reading through a lot of history.

When you test changes through the GUI, the system unconfigures the existing network (by running deconfiguration commands in the reverse order) which is supposed to return the system to a state where there is not a live network configuration. Then the system configures the system with the new configuration being tested.
If, instead, you have (as an example) 10.0.0.1/24 set up on your NAS on igb1 and hook that up to a PC numbered at 10.0.0.2/24, when you make all your changes to the 10G bridging stuff and commit it, it will deconfigure igb1 but then immediately reconfigure igb1 with the same address, so that when you hit the button to confirm the change, your laptop or PC has no trouble reaching the NAS and confirming the GUI prompt.

There's nothing "different" or magic about it except that you're using a configuration strategy that allows you to maintain network connectivity to the NAS.

That why I am using a maintenance remote PC @192.168.1.74 --> igb1 @192.168.1.74 when my bridge is on @192.168.0.60. I am really trying to understand. This is the same as your example just different addresses?.

They're PCIe 2.0 cards, which means that a dual port card does not have sufficient PCIe bandwidth. Power consumption for a dual isn't too terrible at about 7 watts, but the chipset support is really designed as a dual role card that is primarily focused on Infiniband and has Ethernet thrown in as a bonus.

The cards I am using are not those multi-purpose cards IB/Ethernet cards. They are the dual SPF+ type, ethernet only. These are all X2 cards revisions are bizarre.
MNPH29C-XTR X2 Dual port
MNPH29C-XTR X5 Dual port
MNPH29C-XTR X5 Dual Port
I see from the docs that they are 5.0GT/s PCIe 2.0. I did not know how big a transfer was. You stated that this is 4Gbit/sec. What is the translation?

Solarflare SFN6122F spec sheet says: PCI Express PCIe Gen 2.0 compliant @ 5.0 GT/s for full, 40 Gbps bi-directional bandwidth

http://www.moderntech.com.hk/sites/default/files/datasheet/Solarflare_Onload_SFN6122F_10GbE_Adapter_Brief.pdf

So I don't see the limitation on bandwidth that you stated. A 8X PCIe 2.0 slot should be able to handle 10Gb/s speeds.

.I generally refer to Mellanox as the Realtek of the 10G world. Yes, it is often sufficient for baseline needs, but it isn't a real good card compared to more recent developments.

Weird that I do not remember such a time. I wrote the 10 Gig Networking Primer in 2014 and at the time it was slanted very heavily in Chelsio's favor because Intel had a driver issue on FreeBSD 9 that prevented it from being stable.

There is a lot to study, perhaps I miss read or my memory is poor. My original intention was to connect the two servers together with direct connect to do backups. My goals evolved over the last 5 years.

Mellanox had dropped official support for the ConnectX-2 by 2015 (see the specific post at https://www.truenas.com/community/threads/10-gig-networking-primer.25749/page-7#post-275533 and my admittedly critical comments in the next few posts) or if you don't mind going right to the source, Archive.org has conveniently archived it:

Wayback Machine

1.3 Unsupported Functionality/Features
The following are the unsupported functionalities/features in FreeBSD Rev 2.1.6:
• ConnectX®-2 Adapter cards

that is pretty damning evidence.

I would also consider Solarflare SFN6122.

I reviewed the threads and there is a lot of good discussion and evidence the Solarflare SFN6122 is a pretty good option. I purchased some of those for my windows clients a couple years ago as the windows support was pretty good. I installed 3 cards in my test server. It would not boot. with two cards it's fine and the bridge works perfectly. but with 3 cards there is a problem. Not sure if it is a power supply issue or a bus issue. I'll play musical chairs with the cards tomorrow and perhaps swap the power supply.

But the other thing is, in the last ten years, prices on 10G switchgear has dropped. You are not going to get great performance out of your NAS abused as a switch. Why not just buy a small 10G switch? The Mikrotik CRS309-1G-8S+IN switch is usually about $250 and used gear on eBay is sometimes even cheaper.

I have been looking at the 8 port Mikrotik it's quiet and affordable. I have pretty much decided that the NetApp CN1610 NAE-1101 is too loud and power hungry. My original decision to go with a bridge was because the server is already there and it is much more quiet than the available IT switches. The bridge will be in my backup server.

Mikrotik pricing and performance changes things. Instead of buying 3 x new cards for use in a bridge, a $250 switch does not look expensive. If I can get the Solarflare cards to work, I might stay on the bridge path.

Thanks for the help.

 

[jgreco]

That why I am using a maintenance remote PC @192.168.1.74 --> igb1 @192.168.1.74 when my bridge is on @192.168.0.60. I am really trying to understand. This is the same as your example just different addresses?.

You had previously indicated a loss of connectivity ....

When the network interface gui goes into test mode, the network reconfigures, and I cannot press the save button because it is no longer on that address.

... and this typically means that igb1 didn't get reinstantiated. Quite frankly this is annoying to try to puzzle out from someone's forum postings and so I just assumed something wasn't quite as described. This could be as simple as a netmask being off, so it is easiest to just "go big" and try something that should clearly work.

The cards I am using are not those multi-purpose cards IB/Ethernet cards.

But the driver's the same and I'd bet you the silicon is either the same or at least very similar, possibly differing by something as little as a single byte flag in the firmware.

So I don't see the limitation on bandwidth that you stated. A 8X PCIe 2.0 slot should be able to handle 10Gb/s speeds.

I do the math on these things on the fly in my head so maybe I was wrong. Sorry. I might have mixed up part of it. But even so, the PCIe 2 stuff is less desirable than the PCIe 3 cards. A PCIe lane's performance is complicated in other ways too and it is best not to be expecting to get theoretical max performance out of it.

Mikrotik pricing and performance changes things. Instead of buying 3 x new cards for use in a bridge, a $250 switch does not look expensive. If I can get the Solarflare cards to work, I might stay on the bridge path.

If it does what you need, it might be fine. Just don't expect miracles.

 

[Patrick M. Hausen]

@jgreco Kristof Provost assured me in private email that the if_bridge(4) code is not the bottleneck anymore in the packet forwarding path. FreeBSD is still not a proper switch but might prove good enough, finally.

Folks are working on passing tagged frames across a bridge interface, too.

 

[ZiggyGT]

@jgreco Kristof Provost assured me in private email that the if_bridge(4) code is not the bottleneck anymore in the packet forwarding path. FreeBSD is still not a proper switch but might prove good enough, finally.

Folks are working on passing tagged frames across a bridge interface, too.

How best to test the performance of the bridge? I have some systems connected directly to an aruba 2500 10Gbs switch and some connected to each other through a bridge. I tested using the default transfer size using iperf3 between Windows 10/11 machines. I got pretty poor performance at 2.6Gbs with direct connection and through a bridge. I really saw no difference but that not saying much at this performance difference. I also did not change the MTU yet. In the description of iperf3 there is the ability to create multiple threads. Is there a recommendation for how best to improve performance (short of the airplane drop @jreco has suggested for some solutions) I am rechecking the firmware and driver revisions to make sure I have the latest. I also will check the speeds TrueNAS to TrusNAS with and without the bridge. I'll test the file transfer rates later as some systems have SSD and different disk configurations. I am puzzled because I saw 7Gps on FreeNAS, but that was ages ago and not sure of the exact config. I will document the configurations this time.

 

[Patrick M. Hausen]

@ZiggyGT Glad you asked

From that email:

Benchmark results will vary wildly depending on setup and hardware. There is a reason the universal advice about performance is “benchmark your workload”.

My old numbers seem to suggest ~18 million packets per second, or about 8Gbps in 64-byte packets. It should easily saturate a 10Gb link with a more realistic packet size mix.

[...]

Secondly, if_bridge is sufficiently fast that using iperf3 you’re no longer measuring what if_bridge can do, but what the socket layer can do. iperf is the wrong tool for high-performance network benchmarking. It spends most of its time copying data in and out of userspace. It’s simply unable to generate sufficient load to produce useful results. Use pkt-gen/netmap or DPDK or some hardware packet generator.

HTH,
Patrick

 

[jgreco]

iperf is the wrong tool for high-performance network benchmarking. It spends most of its time copying data in and out of userspace.

This doesn't make any sense. FreeBSD bridging doesn't involve userspace. DPDK does, but then again its focus is a bit different. Bridging traffic for iperf3 should perform the same as most other types of traffic, unless we define bridging to include non-bridging activities such as sourcing or sinking traffic in userland. That's not bridging by any conventional definition I am aware of; a bridge moves stuff from one MAC address on one layer 2 network to another MAC address on a different layer 2 network. But I guess I'm kind of a Stevens networking kind of guy.

The concerns I would have about bridging using a FreeBSD host are mostly performance based, noting that I've been doing routing (layer 3) on FreeBSD since the '90's and I'm aware of the limitations. I've been through the Luigi Rizzo stuff with packet forwarding in the oughties and then the netmap stuff a decade ago. I do look forward to being able to handle more networking in software but I'm skeptical it is competitive with a cheap switch.

 

[Patrick M. Hausen]

Kristofs argument is that you are measuring the performance of iperf3 on the end nodes and not the performance of the intermediate system acting as a bridge.

 

[jgreco]

Kristofs argument is that you are measuring the performance of iperf3 on the end nodes and not the performance of the intermediate system acting as a bridge.

That's a dumb argument. You use as many systems as you need when measuring forwarding (or bridging) performance. You keep adding load until you see a plateau and/or packet loss. The thing that is going to kill is overall packet per second performance; this will plateau at some point. Luigi Rizzo had some good work demonstrating this back around 4.5R with the introduction of the device polling code, where he managed to squeeze 185Kpps out of an Athlon 750 system. We have an additional 20 years of improvements including netmap and other stuff, much of which is Luigi's work. Even then, however, I don't believe I've seen anything capable of more than ~~100Mpps (even DPDK) whereas that runs out quickly with 10Gbps ports against a physical switch.

 

[Patrick M. Hausen]

He as well as me above was answering to someone measuring with a single pair of machines.

 

[jgreco]

Can't fix broken I guess. I just don't see a lot of value to abusing the host as a switch. It performs poorly compared to a real 10G switch, and real switches have gotten cheap in the last ten years.

 

[ZiggyGT]

I purged my Mellanox x2 cards from my Truenas servers. I have them in some of my less important clients now. I pondered the Microtik but it did not have enough ports, 4 vs the 6 I need. Surplus switches are far too loud to be local.

I upgraded from the Solarflare SFN6122F to the newer PCIE 3.0 SFN7120 cards and the network performance went up significantly. Instead of peaking at 7-8Gb/s. I was able to achieve faster. Here is the performance chart and graphic. With multiple threads the full bandwidth is achieved. Seems the bridged bight have more variance when heavily loaded but surely adequate for me. I plan to add 4 more clients to the node 6 server. I hope this helps other understand better the performance capability of the bridge method of creating a switch. We'll see how well this bubble gum solution holds together over time. The cost during iperf testing with two different paths with 16 threads wis about 8% CPU. I don't think the server felt too abused.

 

[ZiggyGT]

I finally decided to buy a real 10GB/s switch to replace the bridge in my truenas backup server that I had created with 4 - Solarflare dual port 10Gb Cards. This setup was working fine. The biggest issue was that the backup server was too loud for my work area. It was driving me nuts even after all my attempts to make it more quiet. I also found that the X9 server with the drives was 196 watts, with no drives 117 watts. I had hoped to Spin down the drives when not doing a backup but was having trouble figuring out how to do that.

There were some price drops on surplus equipment that made them look attractive. For example the NetApp NAE-1101 16-Port @ $100 on eBay. Investigating this I found they are very loud and still about 90watts. New 10Gb consumer switches are also looking attractive. they are: TP-Link tl-sx3008f and the microtik CRS309-1G-8S-S. I purchased the TP-Link. It is 17-25watts. This means that at 13cents per Kwatt/hr I can pretty much save the cost of the switch in a year. It is whisper quiet and my work area is now peaceful.
I wanted to test the difference between the performance of the bridge and 10Gb switch. I could not see a difference in the iperf testing. I then used large file disk transfer to test SMB network drives. There was a lot of variation. I repeated the tests with the Tp-Link 10Gb switch. I saw similar variations but overall higher performance. I then used Crystal Disk to test local and network drives. The G drive is a nVME drive so I knew if I tested transfers from that drive the network would be the bottleneck not the disk drive. The surprise was that the single drive SSD pool performance was the same as the Truenas spinner pool. That must mean that the 415MB/Sec (Read). 500MB/sec (Write) is the network bottleneck.

My Takeaways-
- A truenas bridge can be used as an effective 10Gb switch. Performance is not as high as a real switch but it is pretty good. uses lless that 3% of cpu. Cost for 8 ports $160
- If you only need 8 SFP+ ports a new TP-link or Mikrotik Switch will be more economical than a Bridge $225-250
- Surplus switches are power hungry and noisy compared to a modern switch.

.