SSD Cache or not ?

[ManuR]

Hello,
we nedd to use TrueNAS 12.x for target Backup repository (ISCSI and NFS). I need about 500 Tb of storage.
Data Written daily is about 1 Tb.

If I plan to use 60 x 12 Tb HDD SATA 7200Krpm (with 5x Raid Group RAIDZ2).
Server have dual socket Xeon Silver 4214 2.40GHz 12 core and 256 GB of ram.

In general for have optimal performance on READ and WRITE,
It's reccomended to have more RAM ?
It's reccomended to have SSD/Nvme for Read Cache (ZFS L2ARC) and/or SSD/Nvme for write cache (ZFS LOG) ?

Thanks
Manuel

 

[jenksdrummer]

ZFS LOG is not a write cache; it's a sanity check on data integrity for writing to the main pool. RAM flushes stripes to disk every 5 seconds to consolidate writes. SLOG will take the same write that goes to RAM. It never gets read from short of a system crash, then when the machine boots it sees there is data in SLOG and writes that out to disk.

Where that does help is if you enforce sync writes; and have no SLOG - that gets thrown to the main data pool initially, then gets written again from RAM. If you use Standard or Disabled, there really isn't a way faster.

Save for Metadata VDEV, where small writes/small IO can go to SSD instead of slower spindles. Just the default setting of just metadata writes, in my testing has shown improvement. But, note the redundancy needs as when you lose a metadata vdev, you lose the pool the same as if you lost a data vdev. Same with a dedup pool if you go that route.

 

[ManuR]

Regarding Pool Layout, 5x Raid Group RAIDZ2 it's optimal or remmcomend different layout with a total of 60 disks ?

Thanks

 

[jenksdrummer]

Depends on your workload and your goal;

As a rule of thumb, the most performance can be had from doing mirrored pairs of vdevs; essentially a RAID10 setup. Related to that is that if you decide to use SSD for metadata vdevs, you must match the topology in that you need mirrored pair(s) of disks allocated as metadata vdevs (though, you do NOT need more than a single mirrored pair of disks...just that if you do RaidZ2 vdev for your data, you're expected to do raidz2 for metadata vdev, which I am not sure is supported.

Also consider your bandwidth to your 60 disks. If you're running off 1 controller, that's pretty lean on the per-disk datarate; nothing mentioned how you're connecting those disks ;)

 

[blanchet]

The ZFS resilver performance collapses when you have large vdevs and a filled pool.

ZFS: resilver performance of various RAID schemas

Therefore, for 60 disks, 6x10 is a better choice than 5x12.

I bit myself with this issue : after 2 years of operations with 2x 12-disks-wide raidz3, I have switched the layout to 4x 6 disks-wide raidz2 to keep acceptable performances. If you have the money, the ideal layout would be rather:

• WD DC HC550 SATA 18 TB disks
• 10 x 6-disks-wide raidz2

If you are using Veeam Backup, you will be probably happier with more IOPS, for example when you use SureBackup jobs or Instant VM Recovery.

 

[Netfreak]

How much GB cache disk is enough for trueNAS? And I was wondering if a VDEV disk added as a cache disk only if that would make any difference in read/write processes and overall performance? Is a SSD better than a small size laptop HDD or 3.5 pc HDD?

 

[Johnny Fartpants]

How is this system connected to the network?

 

[Mustinmas]

Is a SSD preferable to a 3.5-inch or small-size laptop HDD?

 

[Netfreak]

Is a SSD preferable to a 3.5-inch or small-size laptop HDD?

There are PRO's n CON's.
SSD's are expensive, faster, durable, lightweight, compact, quiet, and use less power. Data recovery of SSDs are not yet possible.
HDD's are cheaper and getting cheaper by the day. Data recovery is an advantage in case of a crash or damage.
I have 2 and more TrueNAS servers running as test machines and in 2 of them I have a SSD as my system boot up disk and a laptop HDD as a cache drive.
Boot up is real fast compared with tests made earlier with a laptop HDD's as boot disk. The life span of any disk is vague thing. You never know when one can fail. I am ready with a second system disk, saved config, snapshots/rollback and offline and off site backups. I am currently only testing and not yet in real life.