Bridge between network interfaces

[jagter_freenas]

Hi,

I have a Freenas system with 4 x 10GB NIC's that are connected point to point to to 2 servers, each server has 2 x 10GB NIC's. Reason for this is that I don't have a 10GB switch at the moment. The two servers are VMware hosts and the Freenas acts as a SAN via iSCSI protocol over the 10GB links.

Each physical connection is a separate sub-net.

The 2 x servers and the Freeness are also connected via a normal 1GB network for the normal VMware Management and vMotion traffic.

The problem is that the 1GB network is relatively slow for the vMotion traffic and I would prefer if the vMotion traffic can also flow over the 10GB links. However, because of the separate sub-nets, the two servers can only communicate via the 10GB links if I bridge the 10GB traffic in the Freeness box between the 10GB NIC's.

I have created a bridge in the Freeness box and added routes to the the servers, but the servers still can't reach each other.

My setup is as follows:

Server 1 Freenas
10.11.242.3 < ------------------------------> 10.11.242.2 (ql2)
static route
net 10.11.243.0/24 gw = 10.11.242.2

Server 2
10.11.243.3 <-----------------------------> 10.11.243.2 (ql3)
static route
net 10.11.242.0/24 gw = 10.11.243.2


In Freenas, I created the bridge as follows:
ifconfig bridge create
bridge0
ifconfig bridge0 add ql2 addm ql3 up

What am I doing wrong or don't I understand?

 

[Mirfster]

The 2 x servers and the Freeness are also connected via a normal 1GB network for the normal VMware Management and vMotion traffic.

Are you using the same subnet "/24" for the other 1 GB NICs? If so, try changing the subnet on the 10GB NICs to something else.

Don't think you need to make a Bridge for this since you are direct connecting only two boxes. Just set the statics (with a subnet that is different from the one you have for the 1 GB NICs.)

 

[depasseg]

If the goal is to bridge the interfaces in FreeNAS (to mimic a switch), why would you use different network subnets? Put them all in the same network.

 

[Mirfster]

If the goal is to bridge the interfaces in FreeNAS (to mimic a switch), why would you use different network subnets? Put them all in the same network.

You are correct. I didn't read the OP's post carefully enough.

 

[depasseg]

My comment was directed to the OP. Since he configured each link as different networks.

Think of the bridge as creating switchports (which is all a switch is).

 

[jagter_freenas]

@depasseg

The reason for the different sub-nets is to prevent multiple NIC's in the Freenas box with different ip addresses on the same sub-net.

Therefore each of the four 10GB NIC's in the Freenas box is on a separate sub-net.

The 1GB sub-net is also on a separate sub-net.

 

[jagter_freenas]

Are you using the same subnet "/24" for the other 1 GB NICs? If so, try changing the subnet on the 10GB NICs to something else.

Don't think you need to make a Bridge for this since you are direct connecting only two boxes. Just set the statics (with a subnet that is different from the one you have for the 1 GB NICs.)

The 1GB network is also on a separate sub-net.

You can't set the static ip's for the 10GB point to point links so that they are all on the same sub-net, as it would violate the rule of only one NIC (one ip address) per subnet for the Freenas box.

I therefore do need separate sub-nets for each 10GB link and I need to bridge the sub-nets in the Freenas box.

 

[depasseg]

You are mixing up so many issues it's becoming difficult to follow.

You are bridging 4 different interfaces but are running 4 separate IP networks above that without full routing.

Put all 4 ports in a bridge, configure 1 IP address on Freenas, and the same IP space on your other servers.

If you had a 4 port switch, would you configure every link with a different IP network, and expect is to work?

 

[depasseg]

Why are you configuring multiple IP addresses in freenas, if you are putting them in a bridge behind the scenes? If nothing else, try link aggregation.

 

[Mlovelace]

Hi,

I have a Freenas system with 4 x 10GB NIC's that are connected point to point to to 2 servers, each server has 2 x 10GB NIC's. Reason for this is that I don't have a 10GB switch at the moment. The two servers are VMware hosts and the Freenas acts as a SAN via iSCSI protocol over the 10GB links.

Each physical connection is a separate sub-net.

The 2 x servers and the Freeness are also connected via a normal 1GB network for the normal VMware Management and vMotion traffic.

The problem is that the 1GB network is relatively slow for the vMotion traffic and I would prefer if the vMotion traffic can also flow over the 10GB links. However, because of the separate sub-nets, the two servers can only communicate via the 10GB links if I bridge the 10GB traffic in the Freeness box between the 10GB NIC's.

I have created a bridge in the Freeness box and added routes to the the servers, but the servers still can't reach each other.

My setup is as follows:

Server 1 Freenas
10.11.242.3 < ------------------------------> 10.11.242.2 (ql2)
static route
net 10.11.243.0/24 gw = 10.11.242.2

Server 2
10.11.243.3 <-----------------------------> 10.11.243.2 (ql3)
static route
net 10.11.242.0/24 gw = 10.11.243.2


In Freenas, I created the bridge as follows:
ifconfig bridge create
bridge0
ifconfig bridge0 add ql2 addm ql3 up

What am I doing wrong or don't I understand?

Did you re-configure the hosts to use the iscsi VMKernel ports for vmotion? If you didn't change the host config then they are still set to pass vmotion traffic over the 1Gb links

 

[jagter_freenas]

Hi,

Thanks everybody for your help and advice.

Turned out that I did not understand 100% how a bridge works and how to configure it. The problem was that I allocated individual ip addresses to the NIC's of the bridge instead of a single ip for the bridge.

Everything is working 100% now.

 

[jagter_freenas]

HI,

One more question - how do I make the bridge creation survive a reboot?

I have added the commands to the /etc/rc.conf file using the Shell in the GUI, but after a reboot, the commands are not in the file /etc/rc.conf.

I presume it is because Freenas is running from a RAM image and I am therefore only changing the file in RAM and not the actual file on the USB boot stick.

Any advice please?

 

[Ludovic Lamarre]

You could add a post init script:
Tasks -> Init/Shutdown Scripts -> Add Init/Shutdown Script
Type: script
Path: path to a script you wrote on persistent storage, aka a hard drive
When: Post Init

Ok, reboot

 

[Patrick M. Hausen]

Hi, all,

I just got a new Supermicro system that looks suspiciously similar to this https://www.ixsystems.com/freenas-mini/ ;-)
The board features 4 Gb-interfaces. And since some of my other devices at home don't work without the FreeNAS, anyway, I decided to get rid of that extra small switch I had in the closet.

I run a jail with mineos, so the bridge0 interface is already created by FreeNAS. I only needed to add the remaining 3 Gb-interfaces to the same bridge. So I created a postinit script. Didn't work at the first attempt, because the bridge0 obviously is created after the postinit scripts are run.

So, here's my script that takes care of this problem - just wanted to share:

Code:

#! /bin/sh

PAUSE="30"
BRIDGE="bridge0"
INTERFACES="igb1 igb2 igb3"

(
    until /sbin/ifconfig | /usr/bin/grep -q "^${BRIDGE}:"; do sleep 2; done
    sleep ${PAUSE}

    for if in ${INTERFACES}
    do
        /sbin/ifconfig ${if} up
        /sbin/ifconfig ${BRIDGE} addm ${if}
    done
) &

exit 0

Take care,
Patrick

 

[Dudleydogg]

Patrick good Script how would you incorporate assigning the IP address in there ? and MTU of 9000 for the Bridge?

 

[Patrick M. Hausen]

Code:

/sbin/ifconfig ${BRIDGE} inet 1.2.3.4/24

After the last loop before the closing parenthesis? Same for the MTU ...

HTH,
Patrick

 

[ZiggyGT]

I can successfully create a bridge using the command below. I have typed in in the command line from the GUI. How do I create a script that runs post init. I see the menu items to do it but my script does not execute, Tedious to type this on each reboot. All 4 external computers do see the Server and each other. I have not run performance tests yet

Please help with step by step for the script installation. what is the best name and location for the script. I assume there is a file name or location that is causing my post init script not to execute:

ifconfig bridge create
ifconfig bridge0 addm mlxen0 addm mlxen1 addm mlxen2 addm mlxen3
ifconfig mlxen0 up
ifconfig mlxen1 up
ifconfig mlxen2 up
ifconfig mlxen3 up
ifconfig bridge0 inet 192.168.0.9/24
ifconfig mlxen0 mtu 9000
ifconfig mlxen1 mtu 9000
ifconfig mlxen2 mtu 9000
ifconfig mlxen3 mtu 9000
ifconfig bridge0 mtu 9000

I have 2 dual port Mellanox cards installed.
I called the file "bridge script.sc" and put it in a director I created called Init. I point the execution there from the GUI inputs.
Thanks for any help you can give this frenass/unix newbie

 

[Dudleydogg]

Ziggy your Script looks just fine but it should be called bridgescript.sh (or something with no spaces) also file extension is .sh
make it a Pos tinit so its the last thing the system runs. I put my scripts in my home Folder make a folder called scripts. /



#!/bin/csh
PAUSE="8"
echo "Here goes the Bridge Post Init Hope this works"
echo "Getting Ready to Create Bridge and Add Members set to UP and asign IP addresses"
ifconfig bridge create
ifconfig bridge0 addm cxgb0 addm cxgb1 addm cxgb2 addm cxgb3
ifconfig cxgb0 up
ifconfig cxgb1 up
ifconfig cxgb2 up
ifconfig cxgb3 up
ifconfig bridge0 inet 192.168.104.1/24
ifconfig cxgb0 mtu 9000
ifconfig cxgb1 mtu 9000
ifconfig cxgb2 mtu 9000
ifconfig cxgb3 mtu 9000
ifconfig bridge0 mtu 9000

Location:
stephen@san:~/scripts % pwd
/mnt/FS4RAIDZ/home/stephen/scripts/mountiscsi.sh
stephen@san:~/scripts %

 

[ZiggyGT]

Ziggy your Script looks just fine but it should be called bridgescript.sh (or something with no spaces) also file extension is .sh
make it a Pos tinit so its the last thing the system runs. I put my scripts in my home Folder make a folder called scripts. /



#!/bin/csh
PAUSE="8"
echo "Here goes the Bridge Post Init Hope this works"
echo "Getting Ready to Create Bridge and Add Members set to UP and asign IP addresses"
ifconfig bridge create
ifconfig bridge0 addm cxgb0 addm cxgb1 addm cxgb2 addm cxgb3
ifconfig cxgb0 up
ifconfig cxgb1 up
ifconfig cxgb2 up
ifconfig cxgb3 up
ifconfig bridge0 inet 192.168.104.1/24
ifconfig cxgb0 mtu 9000
ifconfig cxgb1 mtu 9000
ifconfig cxgb2 mtu 9000
ifconfig cxgb3 mtu 9000
ifconfig bridge0 mtu 9000

Location:
stephen@san:~/scripts % pwd
/mnt/FS4RAIDZ/home/stephen/scripts/mountiscsi.sh
stephen@san:~/scripts %

Dudleydogg,
Thanks for the help. I can;t wait to try it. Clearly you must be using this kind of setup to get 10Gb access to the freenas server. I am going to incorporate the instrumentation you suggested to see if it executes. I am not familiar with Unix/Linux so I make some dumb mistakes for sure.
I am using Mellanox cards in the Freenas server, but SolarFlare in the windows 10 machines as the drivers are easy and cards cheap. Are you using Chelsio?
Do you use all 4 devices to connect to your freenas server?
Why do you think there are so many freenas experts so down on this configuration?
What kind of performance are you seeing?
Have you had any stability issues caused by the bridge?

 

[Dudleydogg]

Yes 10GB Links for iSCSI to VMware.
I setup the Bridge because I was too cheap to buy a Switch, this allowed me to hook 2 separate VMware hosts to Freenas Via 10GB links and keep them on same 10GB Subnet.
My pool is all SSD drives for VMWare my vm's boot up in seconds.
combination of Chelsio and Mellonox.
Create Bridge is not native to the GUI (yet) which is why we need the script. Also if Bridge0 exists for some reason your script will fail.
Performance is outstanding using the 10GB for iSCSI and also moving SMB files
The Bridge did work, but I found better stability using a 10GB switch (Ebay $75 netapp switch) Then I just set unique IPs to each 10GB card for Multipathing in VMware.
test your script from the shell prompt verify it setups the Bridge, once the bridge is active it will show up in the GUI under network.