Bridging 10GbE with Chelsio T420-CR

[Jon K]

Hey jgreco - so, this worked for a LONG time. Finally came time to upgrade ESXi and I ran into issues because the IP was defined on cxgbe0 and if I rebooted the host cabled to that port, the port would go down and thus the NFS server was unreachable from the other, up host. I posted the fix here:

https://forums.freenas.org/index.ph...-drops-when-client-reboots.53052/#post-366851

 

[jgreco]

I think the main problem would be that the FreeNAS middleware may not recognize that this IP interface is there, and that may break some functionality. Does it show up in the system interface list when you have the IP configured on the bridge? I don't have the time or interest to try, but the information may be helpful to other people trying this.

Normally on a typical FreeBSD system, what you describe is the way you'd ideally want to configure it, for the reason you figured out. Congrats on persevering and solving that, it puts you in the clearly-capable-of-this-sort-of-hackery club.

 

[Jon K]

Thanks jgreco - I do confess that I am blind to the middleware issues that may be lurking only because I use the FreeNAS system as a true "NAS/SAN" for VMs so I am really only leveraging the ZFS functionality itself, snapshooting, reporting features out of FreeNAS, and of course the networking flexibility. In hindsight this fix makes complete sense but as you say, I don't/won't really know if FreeNAS likes it. To answer, no, with this setup the GUI does not see the bridge device or IP configuration on it unfortunately.

 

[jgreco]

So you're probably losing out on at least some statistics, plus if you try more complicated configurations, I'm thinking especially within the iSCSI administration, the system may not be aware of the interface and may not offer it as an option.

Mostly I think you just need to be extremely aware of the sharp pointy bits you might be encouraging to tear your belly open during an upgrade. Make sure you test thoroughly after any updates, It seems clear that you are able to figure out how to resolve any issues manually, but you may also wish to make sure that you inspect the interface from the CLI manually both before and after any updates. If you're using it for SAN storage, and aren't frequently updating the system, this is much less of an issue, right up to the point where you do an update and then find yourself mysteriously broken. In such a case, being familiar with what it looked like right before the upgrade is a big win, so I'm trying to encourage you to make sure you do that as part of your process.

I can't believe it's 2017 and we still don't have cheap and decent 10G switches.

 

[CookiesLikeWhoa]

There are some more affordable 10G switches out there now. Ubiquiti has their US-16-XG and you can usually land a used Netgear/Arista/Cisco 10G switch for around the same money. That said, I don't know if I would use the US-16-XG for production. I have it at home and it's wonderful, it is lacking in areas that my Netgear switches have and can be difficult to work into an existing network, but overall a pretty good unit.

 

[jgreco]

I'm aware of the Ubiquiti switch, I wish they'd get their act together and do a nice EdgeRouter in the 10G variety...

Perhaps you could explain the shortcomings of the UBNT switch, before I get grumpy and start crumbling cookies. ;-)

 

[CookiesLikeWhoa]

They actually are working on a 10G edge router. The beta product just came out a few days ago "edge router infinity".

The shortcomings...well there are a lot of them, some of them to do with security, others to do with hardware compatibility, I think I'd sum it up by saying don't use it in production. Hope it will get their some day.

 

[Jon K]

So you're probably losing out on at least some statistics, plus if you try more complicated configurations, I'm thinking especially within the iSCSI administration, the system may not be aware of the interface and may not offer it as an option.

Mostly I think you just need to be extremely aware of the sharp pointy bits you might be encouraging to tear your belly open during an upgrade. Make sure you test thoroughly after any updates, It seems clear that you are able to figure out how to resolve any issues manually, but you may also wish to make sure that you inspect the interface from the CLI manually both before and after any updates. If you're using it for SAN storage, and aren't frequently updating the system, this is much less of an issue, right up to the point where you do an update and then find yourself mysteriously broken. In such a case, being familiar with what it looked like right before the upgrade is a big win, so I'm trying to encourage you to make sure you do that as part of your process.

I can't believe it's 2017 and we still don't have cheap and decent 10G switches.

Absolutely anticipate upgrades not going well. In fact, I had migrated ~5TB of VMs over to a (slow) large Synology device and upgraded FreeNAS to Corral. I found in about 4 minutes that it doesn't support post-init or iSCSI config I needed so I had to flip it back just for that since my 10GbE is hinged on that. Fortunately, I am doing my iSCSI traffic over 1GbE interfaces w/ multi-pathing, etc. so that's pretty vanilla. My 10 Gbps is just used for vMotion/svMotion/NFS.

 

[Jon K]

I'm aware of the Ubiquiti switch, I wish they'd get their act together and do a nice EdgeRouter in the 10G variety...

Perhaps you could explain the shortcomings of the UBNT switch, before I get grumpy and start crumbling cookies. ;-)

The Ubiquiti 16-XG basically supper no SFP+ modules lol

 

[jgreco]

Absolutely anticipate upgrades not going well.

So, that's all good then. I have no real objections to cool hacks like this as long as the victim implementer is aware of the risks and able to cope, and I'm always pleased when they actually learn more fun stuff in the process. You've definitely earned your networking driver's license. ;-)