Which of these configurations makes more sense for my hardware?

[Spock]

Hello everyone,

I am new to TrueNAS and ZFS but I have read a lot about them because I have been trying to build a NAS. Since the main purpose of this built was for storage and less for fast read/writes or VMs I decided to go for HDDs for my main vdev. However, since I managed to find some SATA SSDs at good prices I am looking at the opportunity to go more towards a mixed build since it would be nice to have some better virtualization performance and faster metadata caching.

My motherboard is a Fujitsu D3417-B21 with 64GB ECC UDIMM RAM which has 6 native SATA ports. I also managed to find a Broadcom 9207-8i SAS2308 which plugged in a PCIE 3 slot will extend my SATA ports by 8. That leaves me with 14 SATA ports in total. My other PCIE slot I will use to upgrade networking to a 10Gbit NIC.

I have 6x WD Red Plus WD120EFBX 12 TB 7200 rpm 256 MB 3,5 (+1 spare) and 6x Samsung 960GB SSD SM863 MLC (+1 spare)

The CPU will be Intel Xeon E3-1275v6 if its interesting for some reason.

This brings me to the first question. I have seen a lot of people report improved performance with metadata special vdevs. Essentially the metadata can be saved in this special vdev and thus the read and write of files is much faster than simply doing so only with the HDD vdev. The only catch is that it adds an extra failure point to the system as if this vdev dies you lose the whole pool. This is not a big deal for me since I would anyway deploy a 3-way mirror to add some redundancy and make sure that I dont lose my pool. However, in all of the examples of this working I saw people were always doing it with nvme SSDs (which makes sense because nvme is around 5 times faster than a SATA SSD). However, my motherboard only has one m2 slot so thats not an option for me.

My question is: Can I even build the 3-way mirror metadata using SATA SSDs instead of nvme? Does it help me read and write from my HDD vdev faster?

Assuming that the answer to that question is yes I was thinking going with the following configuration:

A)

250gb nvme slot for booting (backed up constantly)

Pool1
- 6 wide raidz2 HDD vdev (backups and media)
- 3-way metadata special vdev (using the SATA SSDs)

Pool2
- 3 wide raidz1 SSD vdev (this for VMs and raidz1 because I dont care about redundancy as I will be keeping constant backups)

This configuration leaves me with 1 SSD and 1 HDD as backups and 2 free SATA slots in mobo.


If the answer to the above question is no then:

B)

2TB nvme slot for L2ARC ( im not expecting to use the full 64GB of ram so this might be counterproductive?)

Pool1
- 6 wide raidz2 HDD vdev (backups and media)

Pool2
- 6 wide raidz2 SSD (VMs and video editting)

Pool3
- 1 SSD for booting (no redundancy because of backups)


This configuration leaves me with 1 HDD and no SSDs as backup although I would have to buy one just to be sure. This would also leave only 1 SATA slot free on the mobo.


I hope you guys are able to give me a bit of direction because at this stage I am very torn between these two options. I think it really depends on if the SATA SSDs are suitable for the metadata special vdev.

If you have any other ideas please let me know.

All tips appreciated

 

[Etorix]

My question is: Can I even build the 3-way mirror metadata using SATA SSDs instead of nvme? Does it help me read and write from my HDD vdev faster?

Yes and yes.

A)

250gb nvme slot for booting (backed up constantly)

Pool1
- 6 wide raidz2 HDD vdev (backups and media)
- 3-way metadata special vdev (using the SATA SSDs)

Pool2
- 3 wide raidz1 SSD vdev (this for VMs and raidz1 because I dont care about redundancy as I will be keeping constant backups)

Make that a 2-way mirror: Raidz# is not good for VMs.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

B)

2TB nvme slot for L2ARC ( im not expecting to use the full 64GB of ram so this might be counterproductive?)

Pool1
- 6 wide raidz2 HDD vdev (backups and media)

Pool2
- 6 wide raidz2 SSD (VMs and video editting)

The rule of thumb for L2ARC scaling is 5-10*RAM. You do not have enough RAM for a 2 TB L2ARC, which would be counterproductive.
There's also no evidence you would benefit from a L2ARC. Build the NAS, use it for some time, and then run arc_summary from a shell: This will tell whether you'd benefit from a 512 GB L2ARC (but not more). This advice may apply to A) above.

Those who do edit videos directly on the NAS typically have an EPYC CPU and an all NVME pool.
Overall, B) does not look like a happy plan.

By the way, I suppose you are NOT planning to install a legacy FreeNAS release, are you?

 

[Spock]

Thank you for the great answer @Etorix

Make that a 2-way mirror: Raidz# is not good for VMs.

This makes a lot of sense I should have thought about it. This 2-way mirror would use 2 drives and im assuming I cant use 3 drives as previously thought. Is there a mirror setup I could do instead for the VMs with 4 drives that would not be a 4 way mirror? Maybe x2 2-way mirrors? Is there any benefit to that or only the added storage?

When it comes to L2ARC I will do what you said and wait until I have used the NAS for some time before I decide on it. Im not certain but when I think about the special vdevs in place it sounds like L2ARC would be unnecessary. Are you saying that even with the special vdevs I might notice from arc_summary that I would benefit from it?

By the way, I suppose you are NOT planning to install a legacy FreeNAS release, are you?

I apologize if im in the wrong section of the forum. Im not familiar with the differences but is there any benefit to the legacy FreeNAS release? My mobo is a bit on the older side so Id be willing to do it if its better.

 

[Etorix]

This 2-way mirror would use 2 drives and im assuming I cant use 3 drives as previously thought. Is there a mirror setup I could do instead for the VMs with 4 drives that would not be a 4 way mirror? Maybe x2 2-way mirrors? Is there any benefit to that or only the added storage?

Striped mirrors, of course. 2*2-way has twice the storage and twice the IOPS of a single vdev.
You can do 3-way (and 2*3-way, 3*3-way, etc.), but that's probably over the top compared to backing up the VMs to a HDD pool.

When it comes to L2ARC I will do what you said and wait until I have used the NAS for some time before I decide on it. Im not certain but when I think about the special vdevs in place it sounds like L2ARC would be unnecessary. Are you saying that even with the special vdevs I might notice from arc_summary that I would benefit from it?

arc_summary reports on… lots of things. Look for ARC misses in data and metadata; basically, if the figure is low there's nothing to gain from a L2ARC.

I apologize if im in the wrong section of the forum. Im not familiar with the differences but is there any benefit to the legacy FreeNAS release? My mobo is a bit on the older side so Id be willing to do it if its better.

No benefit at all. FreeNAS is no longer maintained; go for TrueNAS, CORE or SCALE. Your Kaby Lake CPU and motherboard are not desperately old, by the way; many here are still happy of their Broadwell systems.
I suppose that the FreeNAS section should just be moved to the bottom of the list—if not downright archived and marked read-only.

 

[Saoshen]

no one really needs special vdevs or caching for media.

for VM/video editing or other high IOPs requirements, 2 or 3 mirror pairs (4 or 6 disk) would be a better choice than raidz.