ZFS Configuration for 8x8TB HDDs and 2x M.2s for Media and Dev/VM usage

[Nate W]

Hey all!

I am finally upgrading from my e3 1240v3 and was curious what people thought the best setup for the drives would be given the following specs and use case:

Use Case

• 60% Media server
• 40% VM/Docker/Dev loads

Specs

• Ryzen 3950X
• 128GB ECC Memory
• 8x 8TB Drives
• 2x M.2 slots available, but not sure if it really matters as a SLOG/L2ARC. I was thinking of using them for fast DB storage without any redundancy (pure Dev loads)

Currently I have a 6x8TB raidz2, but I wasn't sure if raidz2 @ 8x8TB would work alright for this newer CPU and use case. Perhaps more vdevs for the IOPS, but I wasn't sure on an optimal setup for 8 disks.

Thanks in advance.

 

[jgreco]

Worst possible use case.

RAIDZ2 is really a super-bad fit for block storage ("VM/docket/dev")

https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

While on the other hand, RAIDZ2 is the perfect thing for a media server.

 

[Hellione]

i would use at least two mirrored ssd´s for vm.
another two small mirrored ssd´s for boot would be great.

 

[ChrisRJ]

In a home setup you can also run the "VM" workload off consumer SSDs and have frequent backups to the RAIDZ2 pool.

Please also check the support for ECC with Ryzen. IIRC in many cases this means that the machine will boot with ECC UDIMMs installed but not really perform the amount of checks one expects from ECC.

 

[Nate W]

Thanks all.

I was thinking the same thing re: mirrored SSDs for the VMs and docker. In TrueNAS SCALE, is k3os run off of the system drive?

So the final setup would be:

2x SATA SSDs mirrored for Boot (Is this needed if the configs are regularly backed up? I have been YOLOing off a single SSD for a while :D USB before that)
6x8TB in RaidZ2 for the media storage
2xM.2 mirrored for VMs

I think this would definitely cover all my work loads in a fairly optimal way.

 

[jgreco]

The mirrored SSD's for datastore use are going to be much more pleasant than a HDD pool. Note that you do not necessarily need "M.2" (which people seem to think means NVMe, even though lots of M.2 SATA exists) SSD. Your 6Gbps SATA interface is probably a bit faster than your 1Gbps ethernet.

The likelihood that an SSD would fail as boot device is low, but if it blows, your system will go down.

Several scripts are available to back up your configuration and store it in your pool.

 

[Nate W]

The mirrored SSD's for datastore use are going to be much more pleasant than a HDD pool. Note that you do not necessarily need "M.2" (which people seem to think means NVMe, even though lots of M.2 SATA exists) SSD. Your 6Gbps SATA interface is probably a bit faster than your 1Gbps ethernet.

The likelihood that an SSD would fail as boot device is low, but if it blows, your system will go down.

Several scripts are available to back up your configuration and store it in your pool.

Makes sense. I know M.2 is the interface/form factor. I’ll likely use NVMes because they can be wicked fast. I want to use the M.2s as the motherboard I am looking at only has 8x SATA3 and I’d rather tap the fast SSDs directly into the PCIe4 lanes.

Anyone know if SCALE allows you to pick the data store for the k3os side of things or is it on the boot drive? I can cross this bridge when I get there I suppose.

I’ve used jails and VMs on my current raidz2 just fine, but not quite as blazingly fast as I’d like.

 

[Hellione]

now your setup sounds very fine to me :)
#2x SATA SSDs mirrored for Boot (Is this needed if the configs are regularly backed up? #
no but it is nice to have, it makes the system more rock stable, as truenas does use the boot device while running. thats why usb is not more recomm.
#Anyone know if SCALE ...#
dont know anything about it, sorry

 

[Nate W]

now your setup sounds very fine to me :)
#2x SATA SSDs mirrored for Boot (Is this needed if the configs are regularly backed up? #
no but it is nice to have, it makes the system more rock stable, as truenas does use the boot device while running. thats why usb is not more recomm.
#Anyone know if SCALE ...#
dont know anything about it, sorry

Its built and running SCALE happily. It is a BEAST. I need to put my CPU into Eco mode to save some energy.