TrueNAS on 1st-gen Epyc with NVMe SSDs?

[Brian Stretch]

I'm debating repurposing my Supermicro H11SSL and its Epyc 7351p as a TrueNAS host, using four U.2 Gen3 NVMe drives in RAIDZ1 mounted on two PCIe dual U.2 cards:

https://smile.amazon.com/dp/B097R2JK2V/

I'm not sure if I want to stick with TrueNAS Core for now or go straight to TrueNAS Scale Release Candidate and hope for the best. I'm assuming that running Core in an ESXi VM instead of bare metal is unwise. The machine is currently an ESXi host.
One of the four Micron 1100 SATA SSDs in my current Core host is starting to fail and its Xeon E3v5 board is inadequate for anything but SATA and SAS drives. I'll probably just replace the failing drive but using non-enterprise drives was a bit risky to begin with, so... tempting, despite such a beast being extreme overkill on my 10Gb LAN. Being able to run VMs for things like Lancache on Scale would be nice.
Anyhow, just looking for a sanity check before I do anything?

 

[sretalla]

Looks like that motherboard supports PCIe bifurcation, so those cards "should" work to do what you want.

If they don't you may find yourself needing 4 adapters instead of 2, but you will have enough PCIe slots as far as I can see.

Watch out for the heat... airflow in most chassis isn't particularly strong in that area of the case.

 

[Brian Stretch]

Thanks. I have 3 140mm fans in the front of the case and 1 in the back but they're running slow and quiet. Dual U.2 cards would leave room to add another vdev later but that's unlikely to be needed so if single U.2 cards are safer... or I can get cables for the two U.2 ports on the motherboard so I'd need fewer cards. It's probably a bad idea to spend this much money but it'd be really neat.

 

[TDA IT Dept]

I'm testing a somewhat similar arrangement. Passing thru NVMe disks (on a PE7525 backplane) to a SCALE VM.
Trying to see if I can use the SCALE RaidZ2 pool as a datastore for the ESXi host. Core definitely did NOT like the arrangement but so far SCALE has been readily working.

TrueNAS on ESXi7u2 PE R7525 with AMD EPYC 7352 and Intel/Dell Express Flash NVMe P4610 3.2TB SFF

No RAID card. Well there is a BOSS-S1 card for ESXi and TrueNAS OS storage. Idea is to use the ZFS RAIDz for storage for the ESXi datastore via iSCSI on 40Gb NICs + QSFP+ connectors. w/ 2 identical R7525 servers, double everything, NICs, switches all of it. Disclaimer1: I have read every one...

www.truenas.com

 

[sretalla]

RaidZ2 pool as a datastore for the ESXi host

Maybe you haven't seen this:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

For sure you could do better if you weren't reducing your IOPS to that of one disk (or even less).