High count M.2 NVMe HBA cards?

[Asteroza]

So what are people buying for TrueNAS these days for high port count NVMe M.2 host carrier cards, preferably in the 8x and higher range?

Highpoint recently trotted out a NVMe RAID card that's dual slot/double wide 8xM.2 with proper dual fans that looks nice from a cooling a perspective, though it isn't obvious if the chip will do NVMe pass-thru mode and not RAID.
https://www.highpoint-tech.com/nvme1/ssd7749m

There's also that 21xM.2 card that's double wide apparently
https://www.apexstoragedesign.com/apexstoragex21


it also looks like the classic Amfeltec Squid is one of the few carrier boards with the option for x4 mechanical connections...

 

[Davvo]

It's the first time I see that much NVMe on a PCIe slot. Beware of port multipliers!

EDIT: After looking a bit into them they seem safe to use with ZFS.

I have seen threads of people that use the ASUS HYPER card.

Hyper M.2 x16 Gen 4 Card｜Motherboards｜ASUS Global

ASUS offers different kinds of motherboard accessories including Thunderbolt™ expansion cards, M.2 add-on cards, and fan extension cards give DIY PC users better choices when it comes to building their perfect workstation or gaming rig.

www.asus.com

 

[ChrisRJ]

An NVMe SSD is more or less a PCIe card, so you don't need an HBA. You "only" need a simple card that connects the SSDs to the PCIe slot and @Davvo linked a popular one (although I have no idea how well it works with TrueNAS). If that PCIe slot does not support bitfurcation you will need a card with a switch which will increase the cost.

There are also such cards from Supermicro, which I currently contemplate buying. Thanks to @Patrick M. Hausen for recommending them.

RAID is an absolute no-go for ZFS.

 

[ChrisRJ]

As a follow-up to my own post: The fact that NVMe SSDs are directly connected to PCIe lanes also means that you need a ton of those. It does not make sense to connect 24 NVMe SSDs to e.g. 16 PCIe lanes.

 

[Etorix]

Highpoint recently trotted out a NVMe RAID card that's dual slot/double wide 8xM.2 with proper dual fans that looks nice from a cooling a perspective, though it isn't obvious if the chip will do NVMe pass-thru mode and not RAID.
https://www.highpoint-tech.com/nvme1/ssd7749m

If the price is in line with the E1.S version, I would skip this one in favour of Chinese cards.
8 U.2 drives out a x16 slot, or 4 drives from a x8 slot should be reasonable.

There's also that 21xM.2 card that's double wide apparently
https://www.apexstoragedesign.com/apexstoragex21

4x oversubscription is too much. Not to mention the price and questionable cooling.

Then the question is how ZFS can cope with lots of NVMe storage, which is way faster than the spinning platters for which the code was designed.

 

[firesyde424]

So what are people buying for TrueNAS these days for high port count NVMe M.2 host carrier cards, preferably in the 8x and higher range?

Highpoint recently trotted out a NVMe RAID card that's dual slot/double wide 8xM.2 with proper dual fans that looks nice from a cooling a perspective, though it isn't obvious if the chip will do NVMe pass-thru mode and not RAID.
https://www.highpoint-tech.com/nvme1/ssd7749m

There's also that 21xM.2 card that's double wide apparently
https://www.apexstoragedesign.com/apexstoragex21


it also looks like the classic Amfeltec Squid is one of the few carrier boards with the option for x4 mechanical connections...

I wouldn't go with an NVME HBA\RAID card. As mentioned above, NVME drives use PCIE lanes directly so no storage controller is needed. If you've got any AMD Epyc CPU from the last three generations, you should be able to support 16+ drives without too much trouble. If you have anything but gen 4 Intel scalable, you'll need a PCIE switch card. I wouldn't run more than a couple of NVME drives on any AMD Ryzen or Intel Core iX. I'm not sure how many lanes you get from a Threadripper so those might be okay.

You can use PCIE switch cards to share a small number of PCIE lanes to a large number of drives, but the bigger that ratio of required lanes to available lanes is, the slower everything is going to run especially in a zpool. Think of it like trying to pull a boat and a trailer with a Honda Civic. You can do it, but there will be some problems along the way.

 

[Ericloewe]

If you have anything but gen 4 Intel scalable, you'll need a PCIE switch card.

That's not strictly true, as long as you're not looking for insane numbers of disks. As far back as Xeon E5 v1, all 40 CPU lanes can be bifurcated down to x4 (though this was not typically available to configure in system firmware until Xeon E5 v3/v4, and even then not all boards got it).

Even on a single-socket system, that's going to be some 4-5 SSDs, plus a NIC and/or HBA.

 

[Ericloewe]

So what are people buying for TrueNAS these days for high port count NVMe M.2 host carrier cards, preferably in the 8x and higher range?

8+ M.2 SSDs? That's a maintenance nightmare, you'd use U.2 or the new EDSFF formats for anything like that.

 

[Patrick M. Hausen]

@Ericloewe If you have two x16 slots you should be able to use 2 cards with 4 M.2 SSDs each, assuming 4x4x4x4 bifurcation is supported. What would be problematic about such a configuration in your opinion?

Does this hardware (bifurcation for two x16 slots) simply not exist?

 

[Ericloewe]

@Ericloewe If you have two x16 slots you should be able to use 2 cards with 4 M.2 SSDs each, assuming 4x4x4x4 bifurcation is supported. What would be problematic about such a configuration in your opinion?

No problem whatsoever, but you probably want other stuff on your system. You could easily do a dual-25GBe NIC on a x8 and four SSDs on each x16 for a total of eight SSDs, with the PCH handling USB, 1GbE and other bits and pieces.

Any Xeon E5 processor can do that, and I expect most boards can at least be hacked to configure the bifurcation mode appropriately. AMI firmware on Xeon E5 v3/v4 boards has the menu implemented even if it's hidden, so flipping the configuration bits in the UEFI NVRAM is totally doable and the firmware will handle things correctly. This stuff has long been implemented in the firmware to handle the ever-present "choose between one x16 or two x8 slots" thing most motherboards with PCIe slots offer in some way.
You can go the extra mile and hack the firmware image to change the default setting, making it persistent across UEFI settings wipes. Totally doable if your board happens to have dual flash ICs for the firmware, if you mess it up just flip the switch over.
Of course, you can just get a board that has the menu enabled. Dell, for instance, updated the Gen 13 systems to expose this option to the user for all slots halfway through the lifecycle of the systems. I expect the vast majority of Xeon Scalable systems to support this natively as well. Fun fact, for Xeon Scalable Gen 1, they basically took the Xeon E5 PCIe roots and swapped out the x8 for a third x16. Bam, 48 lanes up from 40. Same feature set, bifurcating down to x4/x4/x4/x4.

 

[StrangeOne]

It's the first time I see that much NVMe on a PCIe slot. Beware of port multipliers!

EDIT: After looking a bit into them they seem safe to use with ZFS.

I have seen threads of people that use the ASUS HYPER card.

Hyper M.2 x16 Gen 4 Card｜Motherboards｜ASUS Global

ASUS offers different kinds of motherboard accessories including Thunderbolt™ expansion cards, M.2 add-on cards, and fan extension cards give DIY PC users better choices when it comes to building their perfect workstation or gaming rig.

www.asus.com

Yea ASUS HYPER M.2 has worked flawlessly for me. Running 2 for 8 NVME drives.

Main reason why I went with NVME vs SATA SSD is just price/performance.

When you got tons of PCIE lanes. You might as well use them lol.

 

[Davvo]

Main reason why I went with NVME vs SATA SSD is just price/performance.

Also the latency.

 

[asap2go]

Yea ASUS HYPER M.2 has worked flawlessly for me. Running 2 for 8 NVME drives.

Main reason why I went with NVME vs SATA SSD is just price/performance.

When you got tons of PCIE lanes. You might as well use them lol.

Same for me. Wish I had an EPYC platform to use even more.

 

[daniel05227]

I came here looking for an answer I cannot find and I will also contribute to the original post.

The answer I am looking for is whether I can use a NVMe HBA to connect PCI-E devices to a computer (besides NVMe), such as a network card.
Reason 1: A nice recent find is PCI-E 3.0 x16 PLX board for £200, though I cannot find now and I think it was an import from Germany on eBay; A NVMe HBA PCIe 4.0 x8 costs £150 (Cheaper, same max bandwidth, also can connect NVMe/SAS/SATA).
Copilot gave me contradicting results, which isn't surprising considering I couldn't find a single bit of evidence of a test or claim.

Back to the original post.
If I'm not mistaken that ASUS hyper m.2 requires bifurcation (typically exclusive to expensive motherboards, but often is not available)
A NVMe HBA allows more NVMe devices than there are lanes, which suggests there is a switch and does not require bifurcation.
Why would you want more NVMe devices than there are lanes or even more than (max lanes)/(4x lane connection)?
It is unlikely you'll need to run all devices at max throughput concurrently.
I'm not sure how small operations would effect bandwidth, but presumably small operations have a much lower capacity and will fail to saturate anyway near the bandwidth of a large file transfer, much less all 4 NVMe drives trying to saturate 16 lanes concurrently. Furthermore, we are talking about ZFS, aren't we? So, the filesystem will further limit the capabilities to saturate the bandwidth.
Won't there be latency? Always
Will it make a difference? I doubt it.

To my recollection, there are enterprise PCI-E switches, presumably extremely expensive to capitalise on the enterprise market, but if they weren't useful they probably wouldn't exist because you'd simply buy more lanes if the switches are going to be expensive anyway.
As an example

PEX89104

The Broadcom PEX89144 switch allows data center and cloud providers to build hyperscale compute systems supporting ML/AI and server/storage applications.

www.broadcom.com

shows a switch capable of 144 PCI-E 5.0 lanes - though, it will be likely these will be setup very differently to plugging it into a PCI-E slot
Maybe they are useless and they just want to sell their product? Possibly.

So, as an overall summary for connecting multiple NVMe drives to a single slot you have
bifurcation, such as asus hyper m.2
switches, such as the PLX board £200 PCIe 3.0 x16 (sorry can't find it) or the raspberry pi hat £32 PCIe 2.0 x1 to 4 x1 lanes (super slow for this configuration)
HBAs with switches inside
If you have deep pockets, there are raid NVMe cards - considering people are discussing bandwidth - and no, I don't believe mixing raid and ZFS will be a bad idea in every scenario, but I know some people will complain it is always bad. It could be that the bad scenarios have bad configurations.

 

[Ericloewe]

The answer I am looking for is whether I can use a NVMe HBA to connect PCI-E devices to a computer

There is no such thing, because NVMe explicitly does not have HBAs, good riddance.

What you seem to be asking is "can I use a PCIe switch with stuff other than NVMe SSDs?" and the answer is, mostly, yes.

 

[Etorix]

A NVMe HBA PCIe 4.0 x8 costs £150 (Cheaper, same max bandwidth, also can connect NVMe/SAS/SATA).

If by "NVMe HBA" you mean "Tri-Mode HBA", the answer is no: Since these things run NVMe drives under the SCSI bus, they cannot handle generic PCIe devices.

tl;dr Use plain SAS HBAs and PCIe switches as appropriate. Do not buy into Tri-Mode HBAs.

 

[daniel05227]

If by "NVMe HBA" you mean "Tri-Mode HBA", the answer is no: Since these things run NVMe drives under the SCSI bus, they cannot handle generic PCIe devices.

tl;dr Use plain SAS HBAs and PCIe switches as appropriate. Do not buy into Tri-Mode HBAs.

Yes, by NVMe HBA I did mean the Tri-Mode HBA. I think from the documentation suggests it just 'passes' the NVMe through to the host. Or at least, when reading the documentation for a RAID card as it did as it could not offer hardware raid to the NVMe devices.
Possibly they act like a JBOD, but I could not find anything in the documentation to indicate whether it could or couldn't use other PCI-E devices - that would have been pretty cool.

 

[daniel05227]

There is no such thing, because NVMe explicitly does not have HBAs, good riddance.

What you seem to be asking is "can I use a PCIe switch with stuff other than NVMe SSDs?" and the answer is, mostly, yes.

Oh no, I am aware that motherboards have switches that connect to several PCI-E devices.
There are NVMe HBAs, both raid and non RAID. Some don't offer raid for NVM-E, though I believe some do, or at least some documentation seem to indicate, as does some videos showcasing hardware raid for NVM-E devices.

 

[Ericloewe]

It's not reasonable to call those things "NVMe HBAs". They're typically known as "tri-mode HBAs" and they're a pure scam. You do not get NVMe devices shown to the host, the HBA implements a PCIe host and then exposes everything to the OS as SCSI. Yes, this is bad for performance, no, it's not useful. And it definitely will not work with arbitrary PCIe devices.

 

[Patrick M. Hausen]

the HBA implements a PCIe host and then exposes everything to the OS as SCSI

Wasn't the point of NVMe to finally get rid of YAAL (yet another abstraction layer)? I agree this is really bad architecture.