Build review - VM storage with different workloads

[PiWa]

Hi all,

I've been using TrueNAS for a couple of years now, with ZFS. I've been reading these forums for a couple of months and decided to join and ask your opinions.
I have a storage server, with these specs:

Processor: 2x 8 Core Xeon E5-2667v4 / 3.2GHz / 25MB cache (16C/32T)
RAM: 512GB DDR4 PC4-17000P 2133MHz
Network: 2x 10Gbit uplinks

SSD: 4x Intel S3700 800GB
SSD's NVMe PCIe: 1x Intel Optane 900P 480GB
Harddisks: 4x 20TB Seagate Exos X20 7.2K SAS 12G

The usage of this storage server will be mainly using it as a storage for VM's over iSCSI. The VM's have different workloads: webserver, gameserver, fileserver, VPN, DNS...
I am planning on using ZFS mirroring for the 4x 20TB, splitting it in half but getting the performance for the VM storage. As for the SSD I will be using one drive as a boot drive. The other three as a RAID0 L2ARC cache (current storage at hit rate of 99-100% of ARC). The Optane drive I will use as a SLOG. Yes, I am using sync=always since I care about the data.

What are your thoughts? Do I downsize the Optane? To what size? Do I use the Optane for multiple functions? Do I use a Metadata vdev? Improvements? Things I forgot? Settings I need to keep in mind?

Thanks in advance.

 

[sretalla]

Do I downsize the Optane? To what size?

https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/over-provisioning-nand-based-ssds-better-endurance-whitepaper.pdf

The number floating around the forums is that you need around 30GB for a reasonable SLOG. You could give yourself a little more than that though.

SSD: 4x Intel S3700 800GB

three as a RAID0 L2ARC cache (current storage at hit rate of 99-100% of ARC).

What a terrible waste... make a pool and use that for hign-performance storage on its own.

Improvements?

You'll find that your pool disks are going to be somewhat of a bottleneck with only 2 VDEVs... maybe consider how you would add more.

Do I use a Metadata vdev?

Maybe that could take some of the load from your pool disks, so would also be a better consideration for the SSDs.

 

[PiWa]

Thank you for your reply!

I will use the S3700 disks for an extra high-performance pool of 2 disks in mirror. One I'm planning to use as a boot disk and the other one as a L2ARC for the HDD pool. Does this seem better?

For the bottleneck of the HDD pool, how do you think this will be? What could I do to improve this, besides adding more? There is no option of adding more disks into this solution.

 

[HoneyBadger]

Thank you for your reply!

I will use the S3700 disks for an extra high-performance pool of 2 disks in mirror. One I'm planning to use as a boot disk and the other one as a L2ARC for the HDD pool. Does this seem better?

For the bottleneck of the HDD pool, how do you think this will be? What could I do to improve this, besides adding more? There is no option of adding more disks into this solution.

An S3700, especially an 800GB one, is still quite overkill for a boot pool.

Provided that you wouldn't be limited by the capacity, using two of them for a dedicated 800GB "high performance pool" is certainly viable. Assuming your hypervisor allows it, you can always migrate your "hotter" workloads back and forth based on capacity and desired performance versus your HDD-based pool with L2ARC.

In regards to the HDD pool, it's the limited number of vdevs that will cause the bottleneck. Random reads that aren't served by one of the two (admittedly quite large!) ARC/L2ARC caches will be limited to the effective IOPS of two drives - but with 512GB of ARC, that's quite a lot of workload that can be served straight out of RAM.

One thing I haven't seen noted is the motherboard and chassis in use here - that could give some insight into the design decisions here with regards to connectivity for alternate boot device options or expansion.

 

[PiWa]

An S3700, especially an 800GB one, is still quite overkill for a boot pool.

True! I can always replace one with a Samsung 980 Pro that I have as spare.

Provided that you wouldn't be limited by the capacity, using two of them for a dedicated 800GB "high performance pool" is certainly viable. Assuming your hypervisor allows it, you can always migrate your "hotter" workloads back and forth based on capacity and desired performance versus your HDD-based pool with L2ARC.

That's a nice idea. I have a high-performance VM that uses 500GB and a couple smaller ones, so I could do that.
My hypervisor allows it.

In regards to the HDD pool, it's the limited number of vdevs that will cause the bottleneck. Random reads that aren't served by one of the two (admittedly quite large!) ARC/L2ARC caches will be limited to the effective IOPS of two drives - but with 512GB of ARC, that's quite a lot of workload that can be served straight out of RAM.

Either way, it will be an improvement over what I have now. My current storage has 64GB RAM and two WD Red 6TB drives in mirror, nothing else.
I planned on this new system to have a large ARC and a decent amount of storage to use for a couple of years.

One thing I haven't seen noted is the motherboard and chassis in use here - that could give some insight into the design decisions here with regards to connectivity for alternate boot device options or expansion.

I am using a HP Proliant DL360 G9 LFF with 4x SFF extender for the S3700 drives.

 

[HoneyBadger]

True! I can always replace one with a Samsung 980 Pro that I have as spare.

If desired, you could use all four SSDs in the "performance pool" and have it give you ~1.6T of space for your all-SSD pool. 512GB of RAM would hopefully be enough to frontload the HDD pool as well.

That's a nice idea. I have a high-performance VM that uses 500GB and a couple smaller ones, so I could do that.
My hypervisor allows it.

The challenge with having two separate pools will be the need to manually partition your Optane so it can be used as a log device for both. There's instructions for this, but it's not an officially supported method.

Either way, it will be an improvement over what I have now. My current storage has 64GB RAM and two WD Red 6TB drives in mirror, nothing else.
I planned on this new system to have a large ARC and a decent amount of storage to use for a couple of years.

You'll have a pretty solid experience here from a performance perspective, provided that your hardware is compatible.

I am using a HP Proliant DL360 G9 LFF with 4x SFF extender for the S3700 drives.

You'll want to replace or supplement the included P440ar RAID controller with a compatible HBA. RAID controllers aren't recommended for use with TrueNAS, either SCALE or CORE.

Can you describe the "SFF extender" and how it's attached?

 

[PiWa]

You'll want to replace or supplement the included P440ar RAID controller with a compatible HBA. RAID controllers aren't recommended for use with TrueNAS, either SCALE or CORE.

Thank you! I'll check into that. Currently the controller is in HBA\IT-mode and each disk is seen individually including SMART data.

Can you describe the "SFF extender" and how it's attached?

I can even show you. It's a case at the back internal inside the chassis. See the attached picture.

 

[jgreco]

Thank you! I'll check into that. Currently the controller is in HBA\IT-mode and each disk is seen individually including SMART data.

That's not sufficient.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[PiWa]

That's not sufficient.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Thank you!
I have added a LSI SAS 9207-8i to my build and will be using this.

Decided on using the 3x S3700 disks as a special vdev in 3-way mirror. I think this will show more improvement to the setup as a whole.

Searching through these forums I saw a recommendation to set the Metadata Small Block Size to 64KiB, is this still valid?
What about other settings I should be aware of? Record size? Re-Format drives block size?