8 SSD drives, RAIDZ3 or mirrors?

[urobe]

Hey there,

I'm about to setup my first virtual machine server using Truenas scale. The server will host 6 vms.

for the vms drives I was going to use 8 ssd drives. Now I'm wondering how to set them up.

So far I always used raidz3 for my 8/10 drive nas devices (office network storage) and was very happy with the performance.

But this time the 8 ssds will serve as systemdrives of the virtual machines and Im wondering what to do. I read that the RAID60 would give me four times the reading speed of a single drive, but now improvement in writing, plus less space as in the raidz3. is this assumption theoretically correct?
And what would the theoretical performance o f the raidz3?

The entire system replicates daily to an offsite location truenas, 6 drives, raidz3, so I am not that much worried about data safety.

any input is greatly appreciated.
-Tobi

 

[Patrick M. Hausen]

There is no RAID60 in ZFS. For good performance for VM block storage see @jgreco's resource:

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

So based on that information I would recommend 4 mirror vdevs of 2 SSDs each.

 

[urobe]

There is no RAID60 in ZFS. For good performance for VM block storage see @jgreco's resource:

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

So based on that information I would recommend 4 mirror vdevs of 2 SSDs each.

Thank you very much for the link. VEry interesting read.
That's what I'll do. But I might up my replication frequency...

 

[urobe]

There is no RAID60 in ZFS. For good performance for VM block storage see @jgreco's resource:

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

So based on that information I would recommend 4 mirror vdevs of 2 SSDs each.

I did follow the suggestion and created 3 Mirrors in one vdev. In this Vdev is now one dataset which holds the zvol for the virtual machines. Available space is 5 tb, of which are 128 gb in use for one systemdrive.
The entire system has 128 gb of ram, 8 are allocated to the vm. it is a dual Xeon Gold 6154 system, while the vm has now one cpu, two cores, two threads.
I did a disc test indside the vm, and am somewhat disapointed:

The 6 ssd used are Micron 5400 Pro drives.

Is this to be expected, or did I do something wrong?
any help or hint is greatly appreciated!
-Tobi

 

[ChrisRJ]

What OS is running on the VM? If it is Windows 10 or 11, the VM might certainly benefit from more resources.

 

[urobe]

What OS is running on the VM? If it is Windows 10 or 11, the VM might certainly benefit from more resources.

Yeah, its windows 10. How much do you think would be appropriate? The coming week the server will be upped to 384 GB total, I planed on leaving 128 for the system, the rest would be divided amongst 7 VMs, using between 8 and 80 GB of Ram. I was planing on giving them twice the Ram I would have, as if they were "real" comnputers.

 

[ChrisRJ]

Basically at least what you would have in physical machine that you use for performance testing. For me that would be at least 6 cores and 16 GB RAM. Plus nothing else should be happening on the physical box that could influence the results.

This may or may not change things for you. If it does, you know that you found a limiting factor (or two). If it does not, you know that the bottleneck is in another place. That is one of two things you can do for performance tuning: change things and see which ones actually make a difference. The other is system analysis to see where things are not performing as they should. Ideally the latter should be done before the former. But at the end of the day it is not exact science anyway. Rather a lot of experience and educated guesses.

 

[urobe]

Basically at least what you would have in physical machine that you use for performance testing. For me that would be at least 6 cores and 16 GB RAM. Plus nothing else should be happening on the physical box that could influence the results.

This may or may not change things for you. If it does, you know that you found a limiting factor (or two). If it does not, you know that the bottleneck is in another place. That is one of two things you can do for performance tuning: change things and see which ones actually make a difference. The other is system analysis to see where things are not performing as they should. Ideally the latter should be done before the former. But at the end of the day it is not exact science anyway. Rather a lot of experience and educated guesses.

Thanks for the input! I gave it 16 gb and 16 cores, it partially got better:

But I'll try and tweak the settings a bit - if I find the holy grail, I'll sure report back...

 

[urobe]

okay... after some more testing, I found that the ram and cpu had not much of an impact, here's my final setup and results:

Biggest plus came form the switch to the virtio driver. I'll probably rebuild the vdev with two more mirrors.

 

[urobe]

I created the pool new, from scratch, but this time with 5 mirrors of each two drives, instead of 3 mirrors.
The results are around the same. If you do the test again, you get different results, so for me, adding more mirrors didn't improve the results in this particular test. Soon I'll have the second vm running, then I#LL do some test at the same time, maybe there the additional mirrors will have an effekt.