Broadcom P411W-32P NVMe Switch Adapter

[pclausen]

I'm looking at picking up this guy:

https://docs.broadcom.com/doc/P411W-32P-UG

Per page 4 from the above document, I see this:

Operating System and Driver Support

The adapter does not require a driver for operation or communication to attached NVMe drives. The drives are presented natively to the host and use the native NVMe drivers.

Use the inbox Fusion-MPT™ SAS driver for firmware updates. The following operating systems support the inbox Fusion-MPT SAS driver. See Firmware Download Operation for firmware update steps.

• Microsoft Windows
• VMware vSphere/ESXi
• Red Hat Enterprise Linux
• SuSE Linux
• Ubuntu Linux
• Citrix XenServer
• CentOS Linux
• Debian Linux
• Oracle Enterprise Linux
• Fedora
• FreeBSD

I plan to use the adapter with a SMC CSE-116 1U server with a 10 bay NVMe backplane. The backplane model is BPN-SAS3-116A-N10. Note the N10 at the end. That makes this a unique backplane in that in addition to supporting SAS3/SATA, it also has 10 direct NVMe connectors.

This is the server:

Supermicro CSE-116 X10SRW-F LGA2011 Motherboard 10 Bay SFF 1U Server 2x PSU | eBay

Find many great new & used options and get the best deals for Supermicro CSE-116 X10SRW-F LGA2011 Motherboard 10 Bay SFF 1U Server 2x PSU at the best online prices at eBay! Free shipping for many products!

www.ebay.com

So that hardware should work with TrueNAS, right?

 

[Ericloewe]

Do you need to sell a kidney to buy that thing? I'm thinking you could get away with something cheaper and better overall:

• Two adapter cards for PCIe x16 to 4x SFF-8643/2x Oculink/nx whatever the cool new connector is
  • Gives you the same 8 disks, but with double the bandwidth
  • Much lower cost, most likely
• This leaves a x8 slot available for networking

The adapter cards are available in a variety of types. Passive; with redrivers; with retimers; with a PCIe switch - in that order of increasing cost. That last one is basically what you're looking at, but with overprovisioning.

 

[pclausen]

I can get the P411W-32P for $375 (new open box). I know it retails for $2,500, so that's a great deal it would seem.

The CSE-116 server has a 32x PCIe riser with one 16x slot and 2 8x slots. So dual PCIe x16 to 4x SFF-8643 won't work. Hence why I'm looking at a NVMe switch adapter so that I can mux 32 NVMe PCIe lanes to 16 PCIe lanes.

But yeah, I need one 8x slot for my NIC, so that leaves just the 16x slot and one 8x. Without muxing, I'd be stuck with only being able to run 6 NVMe drives using passive PCIe adapters, which I know are way cheaper. They require Bifurcation as well from what I understand, but I think the X10SRW-F supports that with the latest BIOS. I know my X10SRL-F mobo's do.

The X10SRW-F does have a 8x PCIe slot on the mobo in addition to the 32x riser. However, in this chassis, that slot is blocked, so there are only 32 PCIe lanes to work with.

But I can see picking up a 8X to 2x SFF-8643 passive adapter to go with the P411W-23P to get all 10 NVMe slots working.

EDIT: I stand corrected. That server appears to have 2 risers in it, so there are indeed two 16x slots and one 8x slot and I confirmed it does support Bifurcation. Reason I was unsure is that I don't have the server in hand yet (being delivered tomorrow). So yeah, a pair of passive x16 to SFF-8643 would be way cheaper at about 25 bucks a pop.

 

[jgreco]

Wow.

 

[rvassar]

For ZFS purposes, anything you place in-between the NVMe drives and the system is going to slow things down. This includes any PCIe switches, HBA's/controllers, etc... Direct attach is always going to win. This is one of the driving factors for the PCIe lane counts in the current & upcoming AMD & Intel server CPU offerings. More and more of the system is now housed under the CPU's heat spreader...

 

[Etorix]

I can get the P411W-32P for $375 (new open box). I know it retails for $2,500, so that's a great deal it would seem.

Great savings… but that's still $375 for 8*NVMe compared with $50 for a pair of passive x16 adapters, so an extra $325 for the privilege of keeping a free x16 slot, with the opportunity to go to 10*NVMe without losing the NIC.
Admittedly, I still wouldn't pass the opportunity if I were in your position. If you go for it, please report whether it works as intended with TrueNAS.

 

[Ericloewe]

I can get the P411W-32P for $375 (new open box).

I'll echo our Resident Grinch's words. Holy crap.

So yeah, a pair of passive x16 to SFF-8643 would be way cheaper at about 25 bucks a pop.

Worth a shot at those prices and for PCIe 3.0. For PCIe 4.0, I start getting nervous with the tighter signal integrity requirements.

They require Bifurcation as well from what I understand, but I think the X10SRW-F supports that with the latest BIOS. I know my X10SRL-F mobo's do.

Worst case, you could always just hack in the correct config yourself. I did that on my workstation to get four M.2 SSDs into a single x16 slot.