45drives build

[jab416171]

So I recently purchased a 4U Storinator 45drives system, which comes with:

• LSI MegaRAID SAS 9260-8i (Connecting 40 Drives using 2 Expanders)
• LSI MegaRAID SAS 9211-4i (Connecting 4 Drives in JBOD mode)
• LSI MegaRAID SAS 9285CV-8e
• 2x Intel RES2SV240 24-port SAS2
• 44 4TB drives of unknown quality
• 128GB of DDR4 2133 MHz ECC RAM
• Dual Xeon 2673 V3 (total of 24 cores/48 threads)

I also purchased a HighPoint R750 Rocket 750 40-Channel SATA HBA Card, after reading up a bit about what HBAs 45drives uses.

I currently have an 8x12tb drive raidz2 freenas build with an Intel NVME SLOG in a PCIe slot, but it's getting full and I wanted to expand, and my current motherboard doesn't have any free PCIe slots. My use cases are:

• VM disk images over iscsi 10gbit
• Media (movies, tv shows, pictures, music)
• Backups for my various computers and VMs (use case for dedupe?)

Now for the questions

1. I've heard expanders are bad (although isn't that essentially what a backplane is?), hence why I bought the 40 channel HBA.
   1. Which of these SAS cards can be flashed into IT mode?
   2. If I have a mix of HDDs and SSDs, would it matter which card I use for which purpose?
2. I've read a little about Fusion Pools, where you can store your metadata on a separate vdev. Does anyone have experience with this? Is it recommended?
3. Is Scale ready for prime time or does it need a few more years to work out all the bugs? My current system runs Core 12, I see 13 is available but I have to switch trains, and 13 is now "Community Release Only" and "Not Enterprise Supported".
   1. This would let me move (some of) my VMs into truenas directly... should I?
4. And finally disk/vdev layout. I think I just want two data pools (ssd and hdd), would it be "best" to keep adding 8 drive raidz2's? is there another layout that's more optimal? And for the SSD pool I've heard that striped mirrors is the way to go for iops, so I'm learning towards this.
   1. What SSDs? I've been fond of the Samsung 860/870 EVO/PRO 2.5" drives, but I haven't really shopped around for a while now.
   2. I'm still undecided on how many (if any) of the 4TB drives I'll be using vs selling, but I plan on putting the VM disks on the SSD pool (and probably any Scale apps, assuming I go with Scale)
   3. Are there other things I should put here? Like should this be the system dataset (which iirc just holds RRD data)? What about as the metadata vdev? Or should that be separate?

 

[DigitalMinimalist]

Long post and I’m not a TrueNAS Pro, but a few thoughts:

OS: doesn‘t matter, but small SSD is fine
VMs: IOPS important-> SSD, best would be NVME
Data: I would probably add additional 12TB drives
To speed up the pool:
special vDev for metadata and even small files - IOPS crucial = mirrored NVME

If I understood correctly, iSCSI will not produce many sync writes, therefore SLOG doesn’t help a lot. Probably L2ARC not necessary with 128GB RAM

VDev Design:
Do you plan to further expand in the future?
If yes, smaller vDevs might allow you to expand easier.
If you will use a stripe set, e.g. 2x 6x12TB@RAIDZ2, you could add another stripe set when needed .

From speed perspective: Do you need IOPS and transfer speed for media and backups? Probably not and striping not necessary.
Though RAIDZ2, or Z3 pool and you are good.

 

[jab416171]

I left out boot on purpose, it just doesn't really matter that much. on my current system I use a sata dom, I think this one has an SSD (or two?) for boot. I'm not worried about boot, I'll figure something out.
I don't know if the motherboard (X10DRL-i) has any m.2 slots, should I get a pcie m.2 card if it doesn't? or just a pcie ssd? I'm currently using this https://amzn.to/3BEBeid (Intel 400GB 750 Series SSD PCiE 2.5") with this U.2 PCIe adapter https://amzn.to/3xf8qKh and I actually have a 2nd identical drive I'm not using for anything. The initial plan was to have a mirrored SLOG but I didn't have enough pcie slots for that.
I set sync=always on all of the vm zvols, although now I'm wondering how much of a difference an SLOG makes with an SSD pool.

 

[jab416171]

What if I set up l2arc but I only have it store metadata (with secondarycache=metadata)? would that give me similar performance to having a dedicated metadata vdev?

 

[DigitalMinimalist]

Should give you a similar boost.
Special vDev could also speed up access of small files, other than metadata.

 

[NugentS]

What if I set up l2arc but I only have it store metadata (with secondarycache=metadata)? would that give me similar performance to having a dedicated metadata vdev?

In theory - yes. It won't do small files - but that may be fine for you