CPU performance - Mirrors vs Raidz2 - all SSD

[Dice]

Hello Dear Forum,
I'm expanding my homelab.
I'm looking to increase my current RaidZ1, 3x500GB SSD worth of capacity for VM storage.
I've gathered (well, scavenged from other systems... ) two additional 500GB SSDs. I'm willing to pay for an additional up to 2 more units. Total: 7 units to play with.
During this revamp, I will also refit this pool with a PLP SSD as LOG device.

Options I'm considering:
- Add another vdev, 2 vdev (3xSSD, RaidZ1) + SLOG
- Remake pool to 1 vdev(7xSSD RaidZ2). + SLOG
- 3x mirrors. + SLOG

Questions;
1) Is there a significant difference in CPU load between the above setups?
2) Granted all SSDs provide enough IOPS, does it matter as much to choose mirrors over RaidZ(x) for VM storage as the traditional HDD advice?

Currently I see no negative performance impacts from a storage point of view, with the small exception of very high CPU use during installations of certain VMs using CentOS-stream dvd located on the TrueNAS box. The VM is 'living' on another physical box with separate CPU. It is the storage part that taxes the TrueNAS CPU. I suspect this is during the 'read all info from the DVD and check data part'. I might dig into that further somehow.
This got me thinking, that it may become a problem if my 'swarm' of VMs double or triple in numbers "soon" and I choose a known very poor pool design.


The hardware list; i3-6100, X11SSL, LSI 9201-16i. 16GB RAM. TrueNAS 12.0-U1

Cheers,

Edit:
Those 65% load is basically fulload. I give TN access to 3 vcpus (3 threads on 2 cores) and also set cputype to host, which potentially causes TN to show 65% in top, but 100% in GUI.

I made a effort in benchmarking to show the current load of the system:



Writing;

Code:

           capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank         517G   867G      0  1.49K      0  21.0M
tank         517G   867G      0  1.37K      0  18.0M
tank         517G   867G      0  1.38K      0  18.5M
tank         517G   867G      0  1.57K      0  20.5M
tank         517G   867G      0  1.33K      0  18.6M
tank         517G   867G      0  1.48K      0  21.0M 

Code:

last pid: 14678;  load averages:  3.54,  2.87,  1.68                                                 up 5+12:40:29  08:30:22
88 processes:  2 running, 86 sleeping
CPU:  0.3% user,  0.0% nice, 67.3% system,  0.3% interrupt, 32.2% idle
Mem: 324M Active, 1911M Inact, 12G Wired, 1001M Free
ARC: 9904M Total, 8601M MFU, 425M MRU, 588M Anon, 65M Header, 222M Other
     8665M Compressed, 11G Uncompressed, 1.35:1 Ratio 

Code:

 dd if=/dev/zero of=file bs=1048576
1376574767104 bytes transferred in 450.190785 secs (3 057 758 650 bytes/sec) 

Reading:

Code:

last pid: 14946;  load averages:  0.32,  0.89,  1.15                                                 up 5+12:51:05  08:40:58
87 processes:  1 running, 86 sleeping
CPU:  0.0% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.4% idle
Mem: 327M Active, 1909M Inact, 12G Wired, 978M Free
ARC: 9331M Total, 8620M MFU, 420M MRU, 2856K Anon, 65M Header, 221M Other
     8373M Compressed, 11G Uncompressed, 1.36:1 Ratio
Swap: 2048M Total, 2048M Free 

Code:

 157.496712 secs (8 740 346 395 bytes/sec) 

 

[jgreco]

Perhaps you could start out by explaining why you are using RAIDZ for VM storage.


https://www.truenas.com/community/r...and-why-we-use-mirrors-for-block-storage.112/

 

[Dice]

Perhaps you could start out by explaining why you are using RAIDZ for VM storage.
https://www.truenas.com/community/r...and-why-we-use-mirrors-for-block-storage.112/

When the pool was setup, it's main purpose was to figure as a silent All-in-one box, backup up a workstation and some syncthing files... At that time, it made the most sense to use the 3 available SSDs as a Raidz1.

I've read that block storage resource long time ago. Thanks for reminding.
Upon revisiting it, there are equally no particular references to either SSD nor HDD. That might give a clue. However, at the time of originally writing that resource (I recall reading prior to the resource section came around), the vast majority of discussions on the forums on VM-storage revolved around spinning rust configurations, where IOPS were the significant limiting factor of raidzx vs mirrors, and thus the fun to be had when hosting VMs. That shouldn't be relevant to my case as Im looking to use SSDs where IOPS are plentiful.
Additionally, the resource's main point is the raidzX limits performance to a single disk. I'd be super content with the performance of a single SSD for all of my VM needs as far as I can tell.

Am I missing something here?

 

[HoneyBadger]

Am I missing something here?

Not so much missing as "perhaps this part needs more emphasis in the resource section"

However, RAIDZ is not good at storing small blocks of data. To illustrate what I mean, consider the case where you wish to store an 8K block of data on RAIDZ3. In order to store that, you store the 8K data block, then three additional parity blocks... not efficient.

VM storage tends to create a lot of 4K and 8K writes, which RAIDZ doesn't do the best job with from a space-efficiency perspective. The wider and more redundant your vdev is, the worse it gets - so your 3-wide Z1 is probably okay, but someone with a 6-wide Z2 will wonder why they're getting the efficiency of a mirror3.

For the most part, SSDs do get to club the fragmentation-related performance with raw speed and clear the "fast enough" hurdle for general users. Mirrors will still be overall faster.

I need to set up a test environment to compare and put some numbers against this (eg: 4 SSDs, compare their performance and space-efficiency in both 2x2-mirror and 4-wide Z1, or 6 and compare in 3x2-mirror, 2x3-wide Z1, 1x6-wide Z2)

But to answer your questions:

1) Is there a significant difference in CPU load between the above setups?
2) Granted all SSDs provide enough IOPS, does it matter as much to choose mirrors over RaidZ(x) for VM storage as the traditional HDD advice?

1) No, the impact of mirror vs RAIDZ and RAIDZ width in terms of CPU utilization is minor. Things like your compression algorithm or encryption settings will have a far bigger impact.
2) SSD means it doesn't matter "as much" but it still matters. Mirrors will still be faster than RAIDZ. If the IOPS of a single SSD is sufficient for your use case, and space-inefficiency of RAIDZ on small blocks is tolerable, then you might get some greater usable space from RAIDZ.

 

[Herr_Merlin]

Raid z with SSDs do work.. even for VMs. But it will be freaking slow. If you are up for the space it might be fine with a few light VMs. I am doing the same at my private server.. 7x2tb SSDs at raid z1. Performance is about 2/3 of a single ssd. For my purpose that is enough as it's stores mostly data and a few VMs.
At work I would never do that.
I do see some heavy CPUs load with that configuration. Mostly while 2 or more VMs run updates at the same time. You have to run sync always for the VMs. Thus you performance is way slower than a single disk.