Opinions on best way to provision 12 SSDs

[dbesade]

Hello,

I'm in the process of deploying a new TrueNAS Scale system on metal. Specifications as follows:


Dell R730XD SFF (2.5")
128G/DDR4 ECC Memory
2 x E5-2643 v4 @ 3.40GHz
NVME Extender Card for the front NVME Slots
HBA330 (12G SAS, no RAID)
13 x Toshiba PX02SMF080 (800G / 12G SAS / 14.6PBW Endurance)
Workload is VMware via iSCSI
Servers are in proper Data Center with UPS and generator backup

Debating on the following VDEV layouts:

1. RAIDX1 (3 Drives) x 4 VDEVs with 1 Hot Spare
2. RAIDX2 (4 Drives) x 3 VDEVs with 1 Hot Spare
3. Mirror (2 Drives) x 6 VDEVs with 1 Hot Spare)

I Ran some FIO tests with a smaller pool of 9 drives and I've attached the output. I honestly did not see the performance improvement on mirrors over the RaidX1 that I was thinking I would see. The difference in available space is about 1.8TB (5.8TB Available with option 1) and (4.2TB or so with Option 3). Fault tolerance is about the same between the 2... can tolerate a 1 disk failure per VDEV X1 vs Mirrors.

Need some opinions here... attempting to maximize the storage available for vm workload while balancing performance and maxizing available IOPS. Everything is a trade off..

 

[MrGuvernment]

Stripped Mirrors for pure performance with I/O..

Some notes could be related

Planning a 2-disk-failure-safe, high speed, VM capable, NAS storage

Hi *, sorry for posting yet another "how to build my system" post; as a lot of people here I am rather unsure how to come up with the best solution for my requirements given that I'm rather new to the TrueNAS-World. I plan to build a new storage for a larger K-12-like school and have a couple...

www.truenas.com

 

[sretalla]

attempting to maximize the storage available for vm workload while balancing performance and maxizing available IOPS. Everything is a trade off..

You can see from fio that your IOPS are highest with mirrors (but with the test you did, you didn't give it a chance to really push the IOPS).

If you run fio with smaller block sizes (either 4K or perhaps 32K as I understand that's a standard of sorts for some hypervisors) you may well see the difference you're expecting and it will more closely reflect the real-world performance difference you will see with block storage.

 

[dbesade]

You can see from fio that your IOPS are highest with mirrors (but with the test you did, you didn't give it a chance to really push the IOPS).

If you run fio with smaller block sizes (either 4K or perhaps 32K as I understand that's a standard of sorts for some hypervisors) you may well see the difference you're expecting and it will more closely reflect the real-world performance difference you will see with block storage.

I appreciate the suggestion..

I ran the same tests with a smaller block size (32kb). My observations were a dramatic increase in IOPS for both the RAIDZ1 and the Mirror setup. Again however, the 4 RAIDZ1 VDEVs with a 3 drive width was ahead of the Mirror setup (See output below). Green text is writes, red is reads. My hypothesis is this has something to do with a 2 drive mirror being limited in the number of IOPS possible by the vDEV simply because there is only 2 drives. The IOPS are limited in the VDEV simply due to their only being 2 drives available.

The RAIDZ1 VDEV, with a single stripe has 3 devices in the VDEV. Thought process is that in a 2-Drive Mirror VDEV you only get the write performance of a single device and read performance of 2 drives

###RAIDZ1 (4 VDEV of 3 SSDs)
TEST: (g=0): rw=rw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=16
read: IOPS=230k, BW=7201MiB/s (7551MB/s)(844GiB/120001msec)
iops : min=163688, max=301550, avg=230629.93, stdev=2116.56, samples=3824
write: IOPS=230k, BW=7201MiB/s (7551MB/s)(844GiB/120001msec); 0 zone resets
iops : min=164836, max=302033, avg=230618.15, stdev=2114.51, samples=3824
!
###MIRRORS (6 VDEV of 2 SSDs)
TEST: (g=0): rw=rw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=16
read: IOPS=220k, BW=6882MiB/s (7216MB/s)(806GiB/120002msec)
iops : min=157172, max=263617, avg=220343.67, stdev=1497.52, samples=3824
write: IOPS=220k, BW=6881MiB/s (7215MB/s)(806GiB/120002msec); 0 zone resets
iops : min=156938, max=262200, avg=220326.00, stdev=1499.80, samples=3824

May be worth re-running these test with a 4k workload

 

[dbesade]

So I ran the same tests at 4K and had some more interesting observations. We did see an increase in IOPS for both the RZ1 and the Mirror above what we tested, but also noted a significant decrease in bandwidth for both sets of tests vs. 32k and 128k workloads. This still seems a bit off as I would expect to see the mirror exceed the performance of the RZ1, but I'm unable to see these results in the testing.

###RAIDX1 (4 VDEV of 3 SSDs)
TEST: (groupid=0, jobs=16): err= 0: pid=23832: Fri Jan 26 05:53:30 2024
read: IOPS=277k, BW=1083MiB/s (1135MB/s)(127GiB/120001msec)
iops : min=240387, max=298755, avg=277295.70, stdev=644.00, samples=3824
write: IOPS=277k, BW=1082MiB/s (1135MB/s)(127GiB/120001msec); 0 zone resets
iops : min=240877, max=297270, avg=277238.59, stdev=635.80, samples=3824

###MIRRORS (6 VDEV of 2 SSDs)
TEST: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
read: IOPS=262k, BW=1025MiB/s (1075MB/s)(120GiB/120002msec)
iops : min=204162, max=282124, avg=262526.79, stdev=604.91, samples=3824
write: IOPS=262k, BW=1025MiB/s (1075MB/s)(120GiB/120002msec); 0 zone resets
iops : min=204920, max=281108, avg=262468.21, stdev=595.42, samples=3824

 

[dbesade]

and realize- I am looking at the Average IOPS - clearly the Maximums show that the mirrors are faster. I may be getting into the weeds, but I find it interesting the Averages don't follow the Maximums

 

[sretalla]

You might consider changing the number of jobs to match your CPU thread count (if 16 isn't it).

Indeed RAIDZ1 can do better at reading and with 3 VDEVs you can get a reasonable amount of IOPS, but you should look at the block storage post again to understand what you lose.

 

[MrGuvernment]

and realize- I am looking at the Average IOPS - clearly the Maximums show that the mirrors are faster. I may be getting into the weeds, but I find it interesting the Averages don't follow the Maximums

From a post I did, if you need even more types of tests to do and try. also consider how many threads to run if you plan to have multiple devices accessing things, as that is where arrays tend to shine vs a single user / thread.

Tweaking my system - baselines - stumped on some numbers (PCIe Slot speeds vs actual)

Hello All, As I lurk these forums often and read far too much, I finally got some time to do some testing on my set up (in sig) Along with that, I added 2 more NVMe's to my VM pool. Using FIO (all tests run local on TrueNAS via SSH), I began doing some testing of each NVMe to get an idea of...

www.truenas.com

The source being this post, about half way down

How to check hard disk performance

How to check the performance of a hard drive (Either via terminal or GUI). The write speed. The read speed. Cache size and speed. Random speed.

askubuntu.com