Bridge between network interfaces

[ZiggyGT]

Yes 10GB Links for iSCSI to VMware.
I setup the Bridge because I was too cheap to buy a Switch, this allowed me to hook 2 separate VMware hosts to Freenas Via 10GB links and keep them on same 10GB Subnet.
My pool is all SSD drives for VMWare my vm's boot up in seconds.
combination of Chelsio and Mellonox.
Create Bridge is not native to the GUI (yet) which is why we need the script. Also if Bridge0 exists for some reason your script will fail.
Performance is outstanding using the 10GB for iSCSI and also moving SMB files
The Bridge did work, but I found better stability using a 10GB switch (Ebay $75 netapp switch) Then I just set unique IPs to each 10GB card for Multipathing in VMware.
test your script from the shell prompt verify it setups the Bridge, once the bridge is active it will show up in the GUI under network.

I purchased a 1Gb switch with 2x10Gb ports. I use it to connect the many 1Gb systems into the 10Gb network and test all my 10Gb hardware. It is so loud I had to build a room for it under my stairs. I thought about adding a 12x10Gb switch but I imagine that it also is designed to be deep in an IT department lab. How is your netapp switch? I have been looking at a Fujitsu switch because the price was becoming in range. I made my "muffler room" adequate to accommodate another 1U switch. I hope the bridge works so I don't need to deal with the potential noise.

 

[ZiggyGT]

Yes 10GB Links for iSCSI to VMware.
I setup the Bridge because I was too cheap to buy a Switch, this allowed me to hook 2 separate VMware hosts to Freenas Via 10GB links and keep them on same 10GB Subnet.
My pool is all SSD drives for VMWare my vm's boot up in seconds.
combination of Chelsio and Mellonox.
Create Bridge is not native to the GUI (yet) which is why we need the script. Also if Bridge0 exists for some reason your script will fail.
Performance is outstanding using the 10GB for iSCSI and also moving SMB files
The Bridge did work, but I found better stability using a 10GB switch (Ebay $75 netapp switch) Then I just set unique IPs to each 10GB card for Multipathing in VMware.
test your script from the shell prompt verify it setups the Bridge, once the bridge is active it will show up in the GUI under network.

I am puzzled still. I Have the script named correctly now and it runs ( I see the echo commands) but gets errors.
errors I get are:
ifconfig bridge create -- error: interface bridge does not exist
ifconfig bridge0 addm mlxen0 addm etc... -- error: interface bridge0 does not exist
ifconfig mlxen0 up -- error: bad valuep
same error for other ports
ifconfig bridge0 inet 192.168.0.9/24 -- error bridge0 does not exist
ifconfig bridge0 --error bridge0 does not exist
if I type each of the commands into the shell It works propoerly
If I type ./bridgescript.sh -- I get the same errors
The default shell is zsh. does this matter if these are csh commands. sorry for the dumb questions. Script would not run with #!/bin/csh as first line.
Please help me resolve this. when working (commands manually entered) performance is a little strange. I have 3gb/sec drives on Dual Xeon 771 w/8GB motherboard (test system). Speed is about 1.5Gb/sec then falls off. If I delete files and resend it hits 5.5Gb/sec. I assume it is cached at that point. It falls off quickly as you can see in the plot. Target is a slow drive so might be the bottleneck. Just trying to get it working before migrating to real hardware. Looking for any success though

 

[ZiggyGT]

What does
#!/bin/csh
do?
It gave me errors, I commented it out of the script. but I am guessing that it tells the script that what shell to use. I have trouble that the script runs but gives errors

I found that the issue was that I was using a windows editor to create the script. The cr/lf vs lf was causing the failures. I used the windows editor ConTEXT to create a unix formatted file. Once I did that and got the path correct the script worked well. I can communicate across the network.
Thanks for your help. this was a nightmare.

 

[blueether]

I found that the issue was that I was using a windows editor to create the script. The cr/lf vs lf was causing the failures. I used the windows editor ConTEXT to create a unix formatted file.

Notepad++ is my windows goto, my IT department are even kind enough to install it on my work laptop

 

[ZiggyGT]

The experiment with the bridge is working pretty well now. I have tried to transfer the hardware from the HP XW8400 I was testing it on to the HP Z400 that is my Freenas production server. I had 2xMelanox 2 port X2 cards and 1xMelanox 2 port X3 card to provide 6 10Gb ports. this was working well in the HP XW8400. I had to use a PCI video card to fit them all into that motherboard. I installed all the cards in the HP Z400. Removing the PCIe Graphics card that was there previously. To my surprise a error appeared on boot telling me the Ram in the last 2 slots was bad (it is not). I removed the Melanox X3 card and the system booted without errors. I replaced that card with a Melanox 1 port X2 card. Now I have a bridge with 5x10Gb ports. I have seen this before on a different Z400 motherboard. If it does not like something on the expansion bus it complains the last two slots of ram are bad. I thought I might be running out of PCIe lanes but I get the same result with the PCI video card if the Mellanox X3 card is installed. Any suggestions as to what the problem might be. The XW8400 is a much older motherboard than the Z400 and the Mellanox X3 card worked well. The Z400 has both PCIe and PCie 2.0 slots. Neither worked. Anyone else have trouble with Mellanox X3 cards in HP motherboards.

 

[ZiggyGT]

With 11.3 there is a bridge option so no script is needed. Has anyone tried to use the bridge option in 11.3?

 

[Patrick M. Hausen]

With 11.3 there is a bridge option so no script is needed. Has anyone tried to use the bridge option in 11.3?

Yes. Works as designed. What is your question?

Kind regards,
Patrick

 

[ZiggyGT]

I was looking for some confirmation that someone has used the new bridge option to connect multiple cards together as my script did. I made a lot of dumb errors in setting up the script to create a bridge for the 3 Mellanox cards I use to create my switchless 10GB network. I probably will make some more using the new option. I am happy to see it integrated

I am having some other trouble right now with the PLEX plugin being seen as running. I am a little apprehensive to go to 11.3 and toss the bridge script for fear I may never get it working again if I need to go back to 11.2 for some reason. I have disabled the bridge script for now and I am running just with 1Gb cards and still have some trouble. Contemplating going for broke with 11.3. The server is seen fine on that loaded on a nearly identical machine. It is physically to far to connect to my 10Gb network right no. SO my question was. Has anyone used the new bridge option to create a switchless 10Gb network? I am to shy to be the first. How are you using it?

 

[Patrick M. Hausen]

Whatever you do in your bridge script will probably continue to work in 11.3 - mine did. There are no fundamental changes in that area in FreeBSD from 11.2 to 11.3. After upgrading to 11.3 and learning about the new network features I configured

• all physical interfaces without IP address but "up" - this is important!
• you might want to add the
• a bridge with all interfaces as members
• IP configuration on the bridge

Works as it has been on dozens of plain FreeBSD servers in our data centre for years and is documented here:

Chapter 34. Advanced Networking

Advanced networking in FreeBSD: basics of gateways and routes, CARP, how to configure multiple VLANs on FreeBSD, etc

www.freebsd.org

HTH,
Patrick

 

[ZiggyGT]

Thanks. If there is no difference in the OS the old script should run. Here it is
"#!/bin/zsh"
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
export PATH
PAUSE="8"
echo "Here goes the Bridge Post Init Hope this works"
echo "Getting Ready to Create Bridge and Add Members set to UP and asign IP addresses"
PAUSE="4"
ifconfig bridge create
ifconfig bridge0 addm mlxen0 addm mlxen1 addm mlxen2 addm mlxen3 addm mlxen4 addm mlxen5
ifconfig mlxen0 up
ifconfig mlxen1 up
ifconfig mlxen2 up
ifconfig mlxen3 up
ifconfig mlxen4 up
ifconfig mlxen5 up
ifconfig bridge0 inet 192.168.0.9/24
ifconfig bge0 inet 192.168.10.1/24
echo "set IP address"
ifconfig mlxen0 mtu 9000
ifconfig mlxen1 mtu 9000
ifconfig mlxen2 mtu 9000
ifconfig mlxen3 mtu 9000
ifconfig mlxen4 mtu 9000
ifconfig mlxen5 mtu 9000
ifconfig bridge0 mtu 9000
echo "Report config"
ifconfig bridge0

From what you said the creation of the bridge part in the script will be taken care of by what the new UI does but the last part will need to be in the script. but I can just leave as is and it should work. I hope I got the fright. thanks for the insight

 

[Patrick M. Hausen]

Why not put the MTU into the "MTU" field of the UI? And the "up" into the "Options" field? I do not see any need for a separate script for what you are doing.

HTH,
Patrick

 

[ZiggyGT]

Why not put the MTU into the "MTU" field of the UI? And the "up" into the "Options" field? I do not see any need for a separate script for what you are doing.

HTH,
Patrick

I see I'll look at the 11.3 interface again. It would be nice not to deal with the script

 

[ZiggyGT]

Why not put the MTU into the "MTU" field of the UI? And the "up" into the "Options" field? I do not see any need for a separate script for what you are doing.

HTH,
Patrick

Patrick,

I took the plunge and updated to 11.3. Took out my bridge script and relied on the new bridge mechanism to set it up. I had to avoid getting the 1Gb port that was on DHCP to not fight with the bridge. I want the bridge on my main network 192.168.0.x and the 1Gb port was already there. The system set them both on the 192,168.0.x subnet so I could not access the GUI after I set it up. I used the commend console to change the address of the 1Gb port to a different subnet and I was off sailing. There are a few errors I am chasing. I think the MTU is set wrong on one of the Win 10 clients of the firmware needs updating. Performance was as high as 739MB/s on Port 1, 450MB/s on Port 2, 520MB/sec on Port 3, and 110MB/s on Port 4. (I think the MTU might be buggered)
I used to see some trailing off of the speed in 11.2 but 11.3 really rocks. Steady performance through the whole transfer,

Thanks for the help and encouragement to take the plunge. The new interface to setup the bridge was so easy. this really will make this easy to go to switchless 10GB networking.

Bob

 

[ZiggyGT]

My 10Gb/s bridge setup in TRUENAS works!. I had a few issues that set me off from this project for a while. I tried a few times with the GUI interface in TRUENAS. I could get the bridge setup but it would never survive a reboot. I had declared victory too early.
The GUI has a feature (that you cannot turn off) that enables you to test the network configuration before it permanently saving it. This does not work well if you are setting up the bridge from a node that will be part of the bridge because the node gets disconnected. That means you cannot save the state and after 60 seconds the bridge evaporates.

To get around this I setup a different computer to connect to the GUI on a different subnet @192.168.1.60 . I do all of my editing of the bridge network configuration by accessing it on a dedicated 1Gb/s port on my server @192.168.1.61. The 6 x 10Gb/s ports are all assigned to a bridge in the GUI to @192.168.0.60. I used 3x dual port Mellanox X-2 cards for this. Note: I could not get Intel X520 cards to do this. Since the maintenance node I connect with is not using DHCP I can easily setup the switch to use DHCP. This makes the setup and any changes easy to do on the bridge.

Why do this? Saves the cost of a switch. The computer makes less noise than most of the switches I have. I have two servers. The noisy one is in the closet with the Aruba 2500 for port switch and this server with the bridge is near many of the target workstations. I have not seen any performance differences between being connected directly to the Aruba switch or through the bridge, but I have not run organized performance benchmarking yet.