CARP and Supermicro switches

[dhj]

Was curious if anyone ran into an issue of getting duplicate multicast packets causing issues with failovers when you have an X20 configured for HA? We're running a pair of Supermicro SSE-X3548 switches mlag'd together. The each of the two controllers are attached over a portchannel back to the pair. Looked at using a PIM (protocol independent multicast) on the switches to try and prevent this but it doesn't seem that PIM can be setup with portchannels....just physical ports.

Anyone else run into anything like this?

 

[Samuel Tai]

Are you using a regular port-channel or a multi-chassis port-channel?

 

[dhj]

We tried both. Initially we were using multichassis port-channel. Ran into the duplicate multicast packet issue (found from snifs using 'tcpdump -vv -i cxl0 -T carp carp' and 'tcpdump -vv -i cxl1 -T carp carp'). Then we recabled and had each controller on a separate port channel to a separate switch. That allowed us to do manual failovers (via the gui) but if we killed one of the switches to simulate a network failure the connected controller would reboot but the cluster would hang (gui would never come back).

 

[Samuel Tai]

Try putting both X20s on the same switch. CARP needs to be visible on both systems.

 

[jgreco]

Probably worth noting that almost nobody on the forums has actual iX TrueNAS appliances with HA features, and your question is a bit technical on both the networking and the HA side of things.

Have you reached out to iX for support on this?

 

[dhj]

I have but they felt it was more of a switch issue and the onus is on us to fix it. Wasn't finding x20 documentation covering troubleshooting CARP issues with switches and so started going through the forums here...then started asking questions.

I'll just keep looking elsewhere if this isn't appropriate for the Forums. Tnx jgreco.

 

[jgreco]

It's not that it isn't appropriate, it's just that you're in waters most people don't travel in, and I wanted to make sure you had reached out to iX.

Having fought stupid issues like this myself, I appreciate how it can be frustrating. CARP is especially annoying because while it is superficially similar to VRRP, it is basically something that gets sneered at by the router vendors (who also tend to be switchgear vendors). I had similar troubles many years ago with ESXi in a multiple uplink scenario that required the use of the (at the time undocumented) ReversePathFwdCheckPromisc tunable to "fix" it.

Sorry I don't have anything concrete for you about your specific issue. I think I'd try to figure out whose silicon is powering the thing, and then look at other products with that silicon. Supermicro tends to be focused on datacenter racks at scale, and often their products aren't quite as polished with respect to obscure features as your typical high end servers, or, in this case, switchgear.

 

[santapaz]

We tried both. Initially we were using multichassis port-channel. Ran into the duplicate multicast packet issue (found from snifs using 'tcpdump -vv -i cxl0 -T carp carp' and 'tcpdump -vv -i cxl1 -T carp carp'). Then we recabled and had each controller on a separate port channel to a separate switch. That allowed us to do manual failovers (via the gui) but if we killed one of the switches to simulate a network failure the connected controller would reboot but the cluster would hang (gui would never come back).

Hi, Have you ha any luck ?

We have the same X20 HA with dual controller too each connected to 2 switches .

I'm trying to find documentation about the network cabling and switch configuration best practices. Anyone?

Thanks

 

[dhj]

Yeah IXSystems doesn't actually provide any documentation for a recommended switch topology to use. We had to get each of the two topologies we tried vetted by one of their escallation engineers who would give it a "thumbs up" during our initial engagement where a support engineer was setting up the x20 while we watched.

At first we used a typical LACP setup where we had each controller connected to each of two peer switches using LACP:

This is generally what they recommend. However our switches...when the truenas controllers failed over...started spewing duplicate CARP multicast traffic and it confused the TrueNAS and the cluster would refuse to come back up until we "bounced" our portchannels. This got rid of the duplicate CARP packets and the cluster would come up. This wasn't a truenas issue....it was a supermicro switch issue (as far as we can tell at this point).

I managed to get around that problem by recabling to this configuration:

This allowed us to do manual failovers from the TrueNAS management gui without issue. HOWEVER if we went and physically shutdown one of the switches (or pull the cables associated with a portchannel) the duplicate mac address issue would come back again. Again The IXSystems engineers reviewed this config and blessed it (mostly because it made the problem mostly go away).

This topology has a big caveat. If the switch on the current master controller goes down it will always cause a controller failover....this will cause an interruption to storage connectivity for a minute or so...not good if you have sensitive workloads. You have been warned ;')

I'll stress at this point its not a problem with the TrueNAS's. Its a switch multicast issue. Hopefully you didn't buy Supermicro switches ;')

If you have a support contract you should definitely engage IXSystems to assist you with this. This is just a recount of my own experience. Your mileage may vary.

 

[jgreco]

Its a switch multicast issue. Hopefully you didn't buy Supermicro switches

I've been watching your issue here with some interest. This is the kind of thing you can't easily debug/diagnose on a forum, and really need to lab up with some wireshark and all the gear involved.

I had been sorta hesitant to lay blame where I wanted to, which is the Supermicro switchgear, but evidence now favoring that, I thought I'd comment.

Many vendors do not make their own switchgear, especially at the low end, and rely on OEM's and rebadging, usually with custom firmware. My best example of this was the early-2000's Accton ES4624, which I have on good authority (hey look, it's me) was also sold by SMC, 3Com, Dell, Foundry, and others. One of the interesting bits from this was the sheer difference in the firmware quality. Features like VRRP/CARP are typically implemented by the switch's control plane CPU, so can be dramatically affected by the vendor's attention to detail. SMC and 3Com, which sold relatively few of the ES4624's, did not bother customizing, rebadging, pumping out every firmware update that was made available by Accton, while others, such as Foundry, actually had some of their own above and beyond.

Supermicro has always been a great source for the basic components of a data center at a low price. However, their products are rarely the sort of "in depth" polished products that you get from Dell or HP, where things like iDRAC+LifecycleController are highly integrated, deep functionality, compared to Supermicro's comparatively superficial IPMI implementation.

I kinda expected that this would resolve with fingers pointing at the Supermicro switchgear, because it's unlikely that a company whose focus is not actually networking would invest much time in supporting this kind of functionality. I think in this case, the best you can hope for is that it comes from the upstream vendor, whether that's a silicon manufacturer or a board-level OEM, with fully functional firmware that implements everything correctly. Guessing that in this case, your "complex" setup exceeded that.

 

[Ericloewe]

I love how easy enterprise switching is. Vendors are so clear and up-front about what sort of platform their switches use and who makes them. /s

In all seriousness, some cursory digging doesn't reveal much about the switch's internals (not surprising, even for something like Dell switches it's surprisingly hard), but it may be worth investigating whether an alternative OS would fit your needs better. If you're lucky, the switch is a true badge-engineered reference model from someone whose stuff is supported by SONiC, which I gather is the most popular option for people not up for replacing all their stuff with Nvidia/Mellanox gear.

 

[jgreco]

Dell switches

With Dell's acquisition of Force10, they seem to have stopped picking random OEM's for much of their lineup and may actually have turned into a halfway decent switchgear vendor.

Even before that, they seemed to have better-than-average gear. I'm sure I've got a PowerConnect 5012 rattling around here somewhere (mainly to sneer at), and some of the 5324's I purchased many years ago are still in service at customer sites. These are low end switches, to be sure, but it was pretty impressive how reliable they've been. I can't think of many other networking devices like them; I've got at least a dozen under management, they're like sixteen years old, and I'm not recalling any failures, though there might have been one. After all the crap in the early 2000's with $#!+ty switches, the advent of these around 2005 kinda raised the bar for "quiet competence". And that's BEFORE Force10.

The PowerConnect 8024F and 8132F have been doing core switching here for many years, and my biggest issue with them is simply that the layer 3 features aren't particularly robust, but this is really true of all switch vendors.

You can get some visibility into the silicon used by various vendors over at the Cumulus Linux HCL

Network Hardware Compatibility List (HCL)

Find hardware compatible with NVIDIA Cumulus Linux.

www.nvidia.com

which isn't helpful, of course, for many other closed platforms.

 

[Ericloewe]

This model from Supermicro isn’t listed there or in SONiC’s docs, which often means the platform is from some niche player and/or turned out to be a dead end.
Given the feature set, I would not be surprised to find a Cavium/Marvell XPliant under the hood. It’s good enough to be relevant among reputable manufacturers, but never caught on quite enough to be supported by network OSes. See also the Dell S51xx series: https://www.servethehome.com/dell-s5148f-on-48-port-25gbe-6-port-100gbe-switch-hands-on/