30 Users: several VMs: Which SSD/HDDs combination?

[oguruma]

TL;DR: 30-users, 10TBHow to get good read/write and reliable performance for the VMs, and a good balance of cost/read-write performance for the spinning rust.

So far I am thinking:
2x Kioxia CD6 as a zfs-mirror for the VMs to live on
6x 6TB WD Red+ with 3 mirrored vdevs. Not super efficient, but it's more fault tolerant, more flexible expansion, and 6TB drives aren't super expensive.
1x Consumer NVMe to cache metadata, possibly.



I want to build a TrueNas SCALE box for a small news business. 30ish users. They have the VMs/Docker containers below as their main tools, in addition to about 10TB of data storage, which will probably grow to about 20TB 5 years from now. There's a secondard NAS to back up the data to.

A combination of in-office (a mxix of 1G/10G networking on the workstations) and remote/WFH employees.

I'm not limited to a specific chassis, but the In-Win "Hybrid" chassis are interesting, since they support a combination of NVMe (U.2) and SATA/SAS. I'd probably use a dual-socket Xeon 26XX V3/4 (PCIe Gen 3) with a retimer card for the NVMe drives.



VMs/Docker Containers:
Nextcloud - Probably in a VM in a DMZ; ncdata will be on the spinning rust
Superdesk - A docker-based CMS; something like Wordpress, except the general public doesn't access it, rather the articles are sent via API to Wordpress
Vaultwarden/Bitwarden - Docker, behind firewall and acessed via VPN or locally only
3CX PBX - To be installed in a VM, probably with it's own NIC, since there's a surplus of NICs on the router, and it might make QoS easier if they want that.
LAMP Server - Private Development/staging server in a container behind the firewall
LAMP Server - Public Staging server which is to be in the DMZ (for demoing sites for clients before they go live)

 

[NugentS]

Your Kioxia CD6 are read intensive SSD's. I would suggest write intensive or mixed workload would be better. As read Intensive they may not last that well..

No argument on the HDD's - I use mirrors as well - expansion is easier though these days I might not use WD on principle

 

[oguruma]

Your Kioxia CD6 are read intensive SSD's. I would suggest write intensive or mixed workload would be better. As read Intensive they may not last that well..

No argument on the HDD's - I use mirrors as well - expansion is easier though these days I might not use WD on principle

Thanks for the input.

Is there any idea what kind of performance I could expect from 3x2 mirrored vdevs?

 

[jgreco]

Thanks for the input.

Is there any idea what kind of performance I could expect from 3x2 mirrored vdevs?

Pessimistically, budget 700 IOPS. IOP defined as "a run of contiguous blocks of potentially large size, bounded by a disk seek".

ZFS is all about the compsci wizardry, though. So this is a trivialization of an intensely complicated question.

You will probably be able to get closer to 1400 read IOPS.

If you have sufficient RAM for ARC, and L2ARC, you can easily get much/most of your working set into ARC/L2ARC, at which point your read IOPS for frequently accessed materials becomes bounded by CPU/network/RAM/NVMe speeds, since the data is not read from the HDD pool. You can easily blow hundreds of thousands of IOPS that way.

If you maintain low pool occupancy, you can likely get very aggressive write IOPS as well, because ZFS tends to allocate contiguous space where possible when writing a transaction group. This means that if your pool is only 10-20% full, it MIGHT write turbo-SSD-like-fast in the tens of thousands of IOPS even though it is just a few crummy HDD's. This won't work if the HDD's are more full.

The problem people run into is that they EXPECT consistent performance out of systems, and they run these stupid things called "benchmarks" on their system, often when they are new, which is when ZFS will tend to do awesome. What you want to do is to design for the pessimistic case, or, at least, design to CONTROL how bad the pessimistic case becomes.

 

[oguruma]

Pessimistically, budget 700 IOPS. IOP defined as "a run of contiguous blocks of potentially large size, bounded by a disk seek".

ZFS is all about the compsci wizardry, though. So this is a trivialization of an intensely complicated question.

You will probably be able to get closer to 1400 read IOPS.

If you have sufficient RAM for ARC, and L2ARC, you can easily get much/most of your working set into ARC/L2ARC, at which point your read IOPS for frequently accessed materials becomes bounded by CPU/network/RAM/NVMe speeds, since the data is not read from the HDD pool. You can easily blow hundreds of thousands of IOPS that way.

If you maintain low pool occupancy, you can likely get very aggressive write IOPS as well, because ZFS tends to allocate contiguous space where possible when writing a transaction group. This means that if your pool is only 10-20% full, it MIGHT write turbo-SSD-like-fast in the tens of thousands of IOPS even though it is just a few crummy HDD's. This won't work if the HDD's are more full.

The problem people run into is that they EXPECT consistent performance out of systems, and they run these stupid things called "benchmarks" on their system, often when they are new, which is when ZFS will tend to do awesome. What you want to do is to design for the pessimistic case, or, at least, design to CONTROL how bad the pessimistic case becomes.

I appreciate the input.

In an ideal world, I'd be able to saturate the 10G network for large file transfer like video/photoshop/whatever, even though not every workstation has a 10G NIC, and some transfer will be over the internet.

 

[jgreco]

In the real world, you will not be saturating a 10G network with three mirror vdevs. The very fastest hard drives today can sustain at best 250MBytes/sec speeds, so aggregate 750MBytes/sec is the hard limit imposed by the hardware for write speeds, and that's only if everything else is perfect, there are no seeks, etc. That works out to 6 gigabits per second write speed, far short of 10G. You might luck out for 10G on reads, since both sides of a mirror can contribute to read speeds, but this doesn't seem to happen cleanly in practice.