No-HBA NVMe on PowerEdge 7415?

[lc10239]

Hello,
We have a low-criticality storage need in our datacenter, and the current solution has been letting us down so I wanted to give TrueNAS a spin.

The hardware is a PowerEdge 7415 running AMD Epyc 7402P with 256GB RAM. There is no storage controller other than the Dell BOSS boot storage, which is a 2-SSD boot RAID card. All of the other storage are Micron 9300 Pro NVMe hanging off of the PCIe bus.

I understand that usually TrueNAS discussions start with "use an LSI HBA", but that's not really an option when dealing with a dozen PCIe x4 SSDs. My understanding is that the entire point of the HBA is simply to provide an un-messed-around-with view of the disk to the underlying ZFS.

My hope is to set the BOSS card as the boot device, and configure the dozen NVMe as either a single 12-disk RAIDz2 or a pool of 6 mirror vdevs with no SLOG. Our IOP requirements are probably under 100k, which I'm hoping shouldn't be an issue given that each NVMe typically benchmarks ~150k write and ~900k read.

Is there any reason this would not work, or have poor performance?

Thanks!

 

[lc10239]

Seems to work and has good performance (~500k iops with no tuning).

 

[Ericloewe]

As you've found, NVMe eliminates the need for the controller as the middleman, as disks now speak PCIe.

On modern servers, you'll find a few different combinations:

1. Legacy SATA and/or SAS only
2. PCIe only
3. SATA/SAS/PCIe
4. SATA/PCIe, but no SAS
5. Any combination of the above in one chassis

There have also been interesting cost-cutting maneuvers: e.g. the Dell R6515 10-bay has eight bays that do SATA/SAS/PCIe and two bays that are PCIe only. That way, they get away with a standard 8-port HBA and no expander, while still having 10 bays with little impact on the customer. Gen 13 and earlier needed to include an expander to get the extra two bays, driving up cost substantially.
Some systems from the likes of Supermicro do SATA+PCIe, but no SAS, for even greater cost reduction with little impact.