Bridging 10GbE with Chelsio T420-CR

[Jon K]

Hi all - I am going to be attempting to bridge two ESXi hosts together with my FreeNAS box which has a Chelsio T420-CR (dual port) installed without using a switch. Each ESXi host has a Mellanox ConnectX-2 (single port) card. Right now the ESXi hosts are attached with a direct attach cable and working at 10 GbE for vMotion. Their NFS VMK IPs are 172.16.100.100 and 172.16.100.101 both with /29 netmask - my plan is to cable host 1 to port 1 of the FreeNAS box and host 2 to port 2 of the FreeNAS box. Everything would be on the same /29 subnet so I'd make the FreeNAS box 172.16.100.102/29, ideally.

Any pointers on bridging this adapter to accomplish this on the FreeNAS side?

Thanks in advance!

 

[jgreco]

https://forums.freenas.org/index.ph...-without-10gb-switch.25259/page-3#post-161894

 

[Jon K]

Simple enough :)

I'll have the FreeNAS box racked and cabled on Sunday w/ new UPS and let you know how it goes! Thanks!

 

[Jon K]

https://forums.freenas.org/index.ph...-without-10gb-switch.25259/page-3#post-161894

So I did as you recommended but substituted my interfaces:

ifconfig bridge0 addm cxgbe0 addm cxgbe1 up

I see:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:ce:b6:3a:14:00
nd6 options=9<PERFORMNUD,IFDISABLED>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: cxgbe1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: cxgbe0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000


Can I assign the bridge an IP?

 

[jgreco]

You could, but you're probably better off using the address on cxgbe0 as long as that works. FreeNAS will understand that, but may not grok the bridge or where it came from.

 

[Jon K]

You could, but you're probably better off using the address on cxgbe0 as long as that works. FreeNAS will understand that, but may not grok the bridge or where it came from.

I follow. Should I have set an address on cxgbe0 prior to having built the bridge?

 

[Jon K]

So to answer my own question - no, the address doesn't need to be set. What I found in FreeBSD manual on bridging is that I should run:

ifconfig bridge create (creates bridge0)
ifconfig bridge0 addm cxgbe0 addm cxgbe1 up
ifconfig cxgbe0 up
ifconfig cxgbe1 up
ifconfig bridge0 inet 172.16.100.102/29

The result of "ifconfig bridge0" then shows the address as required and it can ping itself.

A reboot results in the loss of the bridge configuration, so it looks like I'll need to edit /etc/rc.conf and add:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm cxgbe0 addm cxgbe1 up"
ifconfig_cxgbe0="up"
ifconfig_cxgbe1="up"
ifconfig_bridge0="inet 172.16.100.102/29"

Sound about right? I'll probably want to set mtu to 9000 also.

 

[jgreco]

I don't think that's a good idea. Making a bridge under FreeBSD is different, because there isn't FreeNAS middleware that's designed to work a certain way to worry about.

I'd use FreeNAS to configure up cxgbe0. This way the system is aware of the interface and the IP address, and is very likely to work fully-correctly. Then add that to a bridge as previously described.

Also, jumbo frames aren't recommended. I advise against them unless you really really need them. There's a sticky around here somewhere...

 

[Jon K]

I don't think that's a good idea. Making a bridge under FreeBSD is different, because there isn't FreeNAS middleware that's designed to work a certain way to worry about.

I'd use FreeNAS to configure up cxgbe0. This way the system is aware of the interface and the IP address, and is very likely to work fully-correctly. Then add that to a bridge as previously described.

Also, jumbo frames aren't recommended. I advise against them unless you really really need them. There's a sticky around here somewhere...

Fair enough. So set the IP on cxgbe0 - then that IP will also respond on cxgbe1? Should I set the IP in the gui via "add interface" and then make it part of the bridge? That way, the IP config is persistent through reboots and I just establish the bridge via post-init script?

 

[jgreco]

The IP won't "respond on cxgbe1." cxgbe1 is bridged and isn't responding to anything, much like when you ping your FreeNAS host through a switch, it isn't your switch port "responding." The host is still responding via cxgbe0.

Should I set the IP in the gui via "add interface" and then make it part of the bridge? That way, the IP config is persistent through reboots and I just establish the bridge via post-init script?

Yes, that was the point. If you work as close to possible within the proper FreeNAS framework, then things mostly just continue to work, and you only have to worry about the parts you're doing that FreeNAS doesn't support directly.

This has apparently worked fine for others who've followed my suggestion. I've done a lot of work with bridging and other FreeBSD networking over the years, but I don't think I've actually tried this specific trick on FreeBSD 10 yet. Still, should probably work fine. If not, obviously there are less elegant routes to making it work.

 

[Jon K]

The IP won't "respond on cxgbe1." cxgbe1 is bridged and isn't responding to anything, much like when you ping your FreeNAS host through a switch, it isn't your switch port "responding." The host is still responding via cxgbe0.



Yes, that was the point. If you work as close to possible within the proper FreeNAS framework, then things mostly just continue to work, and you only have to worry about the parts you're doing that FreeNAS doesn't support directly.

This has apparently worked fine for others who've followed my suggestion. I've done a lot of work with bridging and other FreeBSD networking over the years, but I don't think I've actually tried this specific trick on FreeBSD 10 yet. Still, should probably work fine. If not, obviously there are less elegant routes to making it work.

Ok I understand.

I guess what I want to clarify is that the FreeNAS will be 172.16.100.102 on cxgbe0 which will be in a bridge with cxgbe1. Host 1 (172.16.100.100) and Host 2 (172.16.100.101) will both be able to contact 172.16.100.102, right? And, subsequently, both Host 1 and Host 2 can reach each other through the bridge (all within the same subnet)?

 

[jgreco]

That's the general idea of a bridge, yes.

 

[Jon K]

That's the general idea of a bridge, yes.

LOL yes I was just making sure there wasn't some ulterior operation within FreeBSD

 

[Jon K]

jgreco - thanks for your help as always. Spent this afternoon racking and stacking, got everything cabled, IP'd, and then bridged. All worked - I could ping 172.16.100.100 and 172.16.100. Rebooted, no work... as expected. Bridge0 is missing from ifconfig which is to be expected without a post-init script. Cxgbe0 was still IP'd 172.16.100.102/29, it can ping 172.16.100.100. I then did the same steps as prior, ifconfig bridge create, ifconfig bridge0 addm cxgbe0 addm cxgbe1 up, ifconfig cxgbe0 up, ifconfig cxgbe1 up, but the bridge is not working now - you can see that it did work:

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@krcesx1:~] vmkping -I vmk4 172.16.100.100 (self)
PING 172.16.100.100 (172.16.100.100): 56 data bytes
64 bytes from 172.16.100.100: icmp_seq=0 ttl=64 time=0.209 ms

--- 172.16.100.100 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.209/0.209/0.209 ms
[root@krcesx1:~] vmkping -I vmk4 172.16.100.101 (host 2)
PING 172.16.100.101 (172.16.100.101): 56 data bytes

--- 172.16.100.101 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
[root@krcesx1:~] vmkping -I vmk4 172.16.100.102 (FreeNAS)
PING 172.16.100.102 (172.16.100.102): 56 data bytes
64 bytes from 172.16.100.102: icmp_seq=0 ttl=64 time=0.586 ms
64 bytes from 172.16.100.102: icmp_seq=1 ttl=64 time=0.458 ms

--- 172.16.100.102 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.458/0.522/0.586 ms
[root@krcesx1:~] vmkping -I vmk4 172.16.100.102
PING 172.16.100.102 (172.16.100.102): 56 data bytes
64 bytes from 172.16.100.102: icmp_seq=0 ttl=64 time=0.273 ms
64 bytes from 172.16.100.102: icmp_seq=1 ttl=64 time=0.263 ms

--- 172.16.100.102 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.263/0.268/0.273 ms
[root@krcesx1:~] vmkping -I vmk4 172.16.100.101 (host 2 after original bridge)
PING 172.16.100.101 (172.16.100.101): 56 data bytes
64 bytes from 172.16.100.101: icmp_seq=0 ttl=64 time=0.793 ms
64 bytes from 172.16.100.101: icmp_seq=1 ttl=64 time=0.418 ms

Any thoughts?

Thanks!


Update: So, just for S&G's, since I am up with nothing to do, I reinstalled FreeNAS, booted up, gave cxgbe0 172.16.100.102/29, did ifconfig bridge create, ifconfig bridge0 addm cxgbe0 addm cxgbe1 up, ifconfig cxgbe0 up, ifconfig cxgbe1 up, and I can ping:

[root@krcsan1] ~# ping 172.16.100.100
PING 172.16.100.100 (172.16.100.100): 56 data bytes
64 bytes from 172.16.100.100: icmp_seq=0 ttl=64 time=0.284 ms
^C
--- 172.16.100.100 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.284/0.284/0.284/0.000 ms
[root@krcsan1] ~# ping 172.16.100.101
PING 172.16.100.101 (172.16.100.101): 56 data bytes
64 bytes from 172.16.100.101: icmp_seq=0 ttl=64 time=0.226 ms
^C
--- 172.16.100.101 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.226/0.226/0.226/0.000 ms
[root@krcsan1] ~# ping 172.16.100.102
PING 172.16.100.102 (172.16.100.102): 56 data bytes
64 bytes from 172.16.100.102: icmp_seq=0 ttl=64 time=0.022 ms
^C
--- 172.16.100.102 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.022/0.022/0.022/0.000 ms
[root@krcsan1] ~#

So not sure why it won't come back up as a bridge after a reboot.

 

[jgreco]

I reinstalled FreeNAS

should be unnecessary

, booted up, gave cxgbe0 172.16.100.102/29, did ifconfig bridge create, ifconfig bridge0 addm cxgbe0 addm cxgbe1 up, ifconfig cxgbe0 up, ifconfig cxgbe1 up,
So not sure why it won't come back up as a bridge after a reboot.

Do the "ifconfig cxgbe1 up" first. May not matter, but many "problems" I've found with FreeBSD networking tend not to be actual problems but rather jusr a failure to be pedantic in configuration.

 

[Jon K]

should be unnecessary



Do the "ifconfig cxgbe1 up" first. May not matter, but many "problems" I've found with FreeBSD networking tend not to be actual problems but rather jusr a failure to be pedantic in configuration.

I understand the reinstall was unnecessary but I couldn't get anything to communicate but for the one NIC so I said let's start this over.

Sorry to be verbose, but can you order/describe the config process/commands you'd recommend I try? It's currently working but I have not rebooted.

Edit - nevermind, jgreco for the win!

Rebooted, bridge gone, cxgbe0 already up, so steps followed:

ifconfig cxgbe1 up
ifconfig bridge create
ifconfig bridge0 addm cxgbe0 addm cxgbe1 up

That is all that is needed to restore the bridge. Going to try and put that in as a post-init script and see if it's automatic. Confirmed - post-init script with the above, semi-colon separated, works properly at reboot. Excellent!

One last question - I know you're not a huge advocate of jumbo frames, but I want to do some testing. I know how to do it from the ESXi side, but where do I set mtu to 9000 on a bridge?

 

[jgreco]

That's the order I'd expect would work (best).

 

[Jon K]

Thanks - it does! Also it looks like the bridge will use the mtu of the first member, so if I set cxgbe0 to 9000 then the bridge should get created at 9000.

 

[jgreco]

Thanks - it does! Also it looks like the bridge will use the mtu of the first member, so if I set cxgbe0 to 9000 then the bridge should get created at 9000.

Not something I've looked at. In general I find jumbo packets not to be worth the trouble.

 

[Jon K]

Not something I've looked at. In general I find jumbo packets not to be worth the trouble.

Completely understand - just in case anyone is curious in the future, I set mtu to 9000 on the first member of the bridge, but then the second member didn't function at reboot. So, my post-init script now looks like:

"ifconfig cxgbe1 mtu 9000; ifconfig cxgbe1 up; ifconfig bridge create; ifconfig bridge0 addm cxgbe0 addm cxgbe1 up"

And all good :) 10 GbE w/ jumbo frames bridged so two ESXi hosts can reach themselves and the FreeNAS box!