Is it necessary to set the L2ARC and SLOG devices separately in the case of sufficient memory?

[molay]

Hi! I'm going to use TrueNAS Core to set up my home NAS server, and I have some questions about ZFS configuration.

The hardware configuration I'm currently preparing is:
DELL R730XD with H330 card (HBA/IT mode)
Intel E5-2680V4 x2
Samsung 64G DDR4 2400MHz REG ECC x4 or x8 (256GB or 512GB)
WD HC550 16TB SAS x12 (2 x 6 RAIDZ2)
Intel D3 S4510 240GB SATA x1 (TrueNAS Core)
APC SURT1000XLICH UPS

The main purpose is:
1. Sharing and backup of important data, such as accessing data through SMB/NFS, using macOS TimeMachine to remotely backup data, etc.;
2. Provide storage support for several virtual machines running on another Proxmox server based on iSCSI, such as MariaDB, DNS, Gitlab, Wiki, etc.

Questions about ZFS configuration:
0. Limited by the number of disk bays of the machine, there is still 1 disk bay available.
1. Is it necessary to set up an L2ARC device when using 256GB or 512GB of memory under the current disk capacity?
2. Do I need to set the SLOG device separately?

If you have good suggestions, please let me know, thank you!

 

[ChrisRJ]

Please check the content linked under "recommended readings" in my signature.

 

[jgreco]

RAIDZ is not recommended for block storage or database uses. Please see

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

It's not that it CANNOT work, it's more that it will not perform well.

Is it necessary to set up an L2ARC device when using 256GB or 512GB of memory under the current disk capacity?

Yes, no, maybe, and sure are all plausible (and correct) answers. What you need is a function of your workload and pool design. You have a crippled pool that will perform poorly for reads, so, you could increase performance by increasing ARC (RAM) or adding L2ARC to cache more of your pool's content. This does not have much to directly do with RAM size or disk capacity. It has to do with the working set size, the amount of data that you are frequently reading from the pool. If you have a single 50GB VM and nothing else on the pool, and that VM frequently reads 20GB of its contents, you have a working set size of at least 20GB but no more than 50GB. Having 20-30GB of cache will make this fly for reads. This can help offset the issues with RAIDZ performance, as well.

Do I need to set the SLOG device separately?

As opposed to ...? A SLOG device is a very specialized type of device, so one would expect it to be separate.

 

[molay]

RAIDZ is not recommended for block storage or database uses. Please see

Does this mean that even if I don't use the iSCSI block storage, but use SMB/NFS mounted storage in the virtual machine to read and write the database data will also not perform well?

If I need to use mac to connect the storage via SMB, and remotely edit small or large files (such as NodeJS project, Unreal Project, PSD file, MOV file, etc.), is RAIDZ2 also not a good choice?

If so, I'm a bit confused.

Because I need NAS to provide a remote storage to expand the space of my macs.

As opposed to ...? A SLOG device is a very specialized type of device, so one would expect it to be separate.

Do I have to use SLOG device? If yes, do I need to use two SSDs to mirror the SLOG data?

 

[jgreco]

Does this mean that even if I don't use the iSCSI block storage, but use SMB/NFS mounted storage in the virtual machine to read and write the database data will also not perform well?

That is correct.

ZFS has two basic types of storage.

RAIDZ is primarily (tempted to say exclusively) good at single-user access to large file sequential data. "Single-user" does not necessarily mean that it will die with two users, but it is not good at handling large amounts of concurrent I/O. If you have a backup system that writes large tar files one at a time, RAIDZ's your guy. If you have a large library of ISO or MPG video files, RAIDZ's your guy.

Mirrors are extremely good at any kind of data access. However, they suffer from inefficient use of space; a RAIDZ3 with 11 drives gives you the ability to tolerate 3 drives dying AND 8 drives worth of usable storage.

There isn't a "get out of jail free" pass if you use SMB/NFS. The performance loss is due to the fact that if you have an 11 drive RAIDZ3, and you ask for 32KB of data from a file, ZFS will need to seek all eight hard drives to retrieve the stripe. While it is doing that, those drives CANNOT be seeking to fetch other data for another user. By comparison, every component disk of a mirror pool can potentially be servicing different requests, even though this isn't always the way it ends up.

If I need to use mac to connect the storage via SMB, and remotely edit small or large files (such as NodeJS project, Unreal Project, PSD file, MOV file, etc.), is RAIDZ2 also not a good choice?

Large files will work well on RAIDZ2 or mirrors. Small files (and VM block storage and database files) perform better on mirrors due to the better IOPS capacity. Picking the wrong one is not fatal unless you have a bunch of workload. Some mitigations are possible with the addition of ARC and L2ARC.

Do I have to use SLOG device? If yes, do I need to use two SSDs to mirror the SLOG data?

I'll try to circle around to answer this later.

 

[molay]

I'll try to circle around to answer this later.

Hi, I read the relevant posts in the forum and the suggestions of the moderators. I plan to do the following configuration. Please help me to see if it is reasonable.

0. Use https://wintelguy.com/zfs-calc.pl to calculate available capacity.

1. Front 12 x 3.5" disk trays
-- 1 RAIDZ2 6-bays vdev (available capacity 61TB), used for data that is not frequently modified, such as macOS TimeMachine backup (a large number of files, many small files), multimedia files (basically large files, occasionally deleted), shared files (large and small, occasionally deleted or modified), etc.;
-- 2 Mirror 3-bays vdevs (available capacity 30TB), used for frequently changing data, such as iSCSI block storage, remote code projects containing a large number of small files, etc.;

2. Back 2 x 2.5" disk trays
-- 1 SSD as the system disk to install TrueNAS Core
-- 1 SSD is used for the SLOG device

3. Others
-- Enable LZ4 compression
-- Disable deduplication
-- Use 256GB of memory at first, if the hit ratio is found to be low in actual use, expand to 512GB of memory

The question that circle around :
Because the number of disk bays is limited, is it necessary to configure dual-disk RAID1 for both the system disk and the SLOG device when an online UPS is configured for this server?
I will reduce the number of data bays if this is necessary.

 

[jgreco]

is it necessary to configure dual-disk RAID1 for both the system disk and the SLOG device when an online UPS is configured for this server?
I will reduce the number of data bays if this is necessary.

Matter of opinion. Have you considered something like a Dell BOSS card?

 

[Etorix]

The main purpose is:
1. Sharing and backup of important data, such as accessing data through SMB/NFS, using macOS TimeMachine to remotely backup data, etc.;

Note that it is NOT recommended to access the same share through BOTH SMB and NFS. Pick one and only protocol per share.

1. Is it necessary to set up an L2ARC device when using 256GB or 512GB of memory under the current disk capacity?

A L2ARC is never "necessary". It may be useful when that ARC hit rate is too low despite an already large amount of memory.
Begin without L2ARC, use, measure and decide whether a L2ARC would be useful.

2. Do I need to set the SLOG device separately?

Again, a SLOG is never "necessary". It is useful where there are sync writes, which NFS, TimeMachine or iSCSI may require in your case.
A good SLOG drive has low write latency, high endurance and power-loss protection; little capacity and no requirements on read. These are quite different from the requirements for other devices. NVMe would be best if you have a slot: Optane, or a Radian RMS-200/300 card.
Mirroring is not required: With a single SLOG device, loss of data would only occur in the event of the SLOG failing at restart after unclean shutdown.

With that said, for home use, it could be acceptable to take a large enough Optane drive and use separate partitions(*) as SLOG and L2ARC.

(*) Partions, HERE BE DRAGONS!

1. Front 12 x 3.5" disk trays
-- 1 RAIDZ2 6-bays vdev (available capacity 61TB), used for data that is not frequently modified

About 45 TB usable before you have to consider expanding.

-- 2 Mirror 3-bays vdevs (available capacity 30TB)

About 15 TB usable, to keep below 50% use for use storage.
If the required space is much less than that, you may consider 2-way mirrors of large SSDs (not QLC though) rather than 3-way mirrors of large HDDs (if I understand correctly).

 

[molay]

Thank you all!

It's really a friendly community for newbies.

Matter of opinion. Have you considered something like a Dell BOSS card?

I'm not sure if the two 2.5" "Flex Bays" on the rear of R730 can support the NVMe protocol. Would be best if it could, because it's very convenient to install and replace the SSDs. (I really don't know enough about the forms of SSD.)

I'll look for some information, and if it doesn't support NVMe, I'll try to use the boss card to install two NVMe SSDs for the system disk and the SLOG device.

Although it will be more troublesome to install and replace the SSDs in this way.

NVMe would be best if you have a slot: Optane, or a Radian RMS-200/300 card.

I feel that the SSD in the form of PCIe slot is more inconvenient than the boss card.

If the boss card is also not possible, I will consider it.

About 45 TB usable before you have to consider expanding.

I can understand that the block storages will generate a lot of disk fragmentation and therefore the capacity will be limited by 50%.

But why do general application scenarios also require the capacity to be limited to 70% or 80%? (45TB/61TB, 6x16T RAIDZ2)

 

[HoneyBadger]

I'm not sure if the two 2.5" "Flex Bays" on the rear of R730 can support the NVMe protocol. Would be best if it could, because it's very convenient to install and replace the SSDs. (I really don't know enough about the forms of SSD.)

I'll look for some information, and if it doesn't support NVMe, I'll try to use the boss card to install two NVMe SSDs for the system disk and the SLOG device.

Although it will be more troublesome to install and replace the SSDs in this way.

I can answer a few of those:

1. The rear bays in the R730XD are SAS/SATA only.
2. The Dell BOSS cards (both S1 and S2) only support M.2 SATA devices. You could instead use a different NVMe compatible card like Supermicro's AOC-SLG3-2M2-O that can split a single PCIe x8 slot between two x4 M.2 sockets.
3. An SLOG device has a very different workload support that requires low-latency write performance. The Optane cards were recommended - there are M.2 Optane devices as well, but the Radian RMS cards are regular PCIe.

But why do general application scenarios also require the capacity to be limited to 70% or 80%? (45TB/61TB, 6x16T RAIDZ2)

Generally speaking, ZFS likes to have empty space to write into - keeping a good degree of free space allows the system to write in very efficient "large chunks" for HDDs. SSDs are of course less affected by this.

 

[molay]

1. The rear bays in the R730XD are SAS/SATA only.
2. The Dell BOSS cards (both S1 and S2) only support M.2 SATA devices. You could instead use a different NVMe compatible card like Supermicro's AOC-SLG3-2M2-O that can split a single PCIe x8 slot between two x4 M.2 sockets.
3. An SLOG device has a very different workload support that requires low-latency write performance. The Optane cards were recommended - there are M.2 Optane devices as well, but the Radian RMS cards are regular PCIe.

Thanks a lot!
I learned the relevant knowledge. SATA and NVMe are two completely different protocols, although they can both be M.2 interfaces.
I'd ditch the rear flex bays (convenient, but slow transfer rates).

SSDs are of course less affected by this.

Does this mean that if SSDs is used as data disks, both Mirror and RAIDZ can be used for block storage?
Because SSDs don't suffer as badly from disk fragmentation as HDDs?

 

[jgreco]

Does this mean that if SSDs is used as data disks, both Mirror and RAIDZ can be used for block storage?
Because SSDs don't suffer as badly from disk fragmentation as HDDs?

No, no one said this, and it isn't true. SSD's don't incur the same seek penalties as HDD, so in some senses, fragmentation isn't as significant a performance issue for reads, but fragmentation still impacts writes because the system has to work hard to search through the pool metadata to find a suitable free space to write new data. That's expensive in terms of IOPS and memory and CPU.

Additionally, in a RAIDZ vdev, when reading a single block of data, that will be striped across multiple drives, which is inefficient when compared to what happens on a mirror vdev, where that block is accessible with a single IOP to a single drive. Remember what I said above about a 32KB block on a RAIDZ3. RAIDZ typically looks like random I/O to an SSD controller, and you should carefully look at the benchmarks for SSD's where the I/O size is small. Most SSD's are MUCH slower at small random I/O.

Further, RAIDZ uses a variable amount of space for parity, and if you have not chosen your parameters carefully, you can find your block storage using a horrifying amount of excess space for padding and parity.

 

[molay]

No, no one said this, and it isn't true.

OK, please excuse my stupid question :P

Is Intel® Optane™ SSD 900P 280G enough as a SLOG device?

I have roughly determined that RAIDZ is used for files that are not frequently modified, and Mirror is used for any files (especially frequently modified files).
Let me consider the type of Mirror again.
The available space of 3x 2x16 TB Mirror vdevs is 46T (48.26% of 96T), but each group can only lose 1 disk. How safe is this?
Of course, I will set up regular cold backup, just wondering if it will be much worse than 6x16TB RAIDZ2 in terms of safety.

Based on the comprehensive consideration of the probability of total destruction of Mirror vdevs themselves, as well as UPS and regular cold backup, if the risk of data loss is not high, I intend to follow your suggestions:
Workhorse with 3x 2x16TB Mirror vdevs for daily use, then use 6x16TB RAIDZ2 for static share and backup.

 

[jgreco]

Is Intel® Optane™ SSD 900P 280G enough as a SLOG device?

The Optanes are typically conside
red to be a good SLOG choice due to their low latency and endurance. There is a thread around here discussing good SLOG choices somewhere. I suggest checking there.

The available space of 3x 2x16 TB Mirror vdevs is 46T (48.26% of 96T), but each group can only lose 1 disk. How safe is this?

Relatively safe. You can increase safety by moving to a three-way mirror, and/or making sure that you have a dedicated hotspare available so if one of a mirror pair starts to fail, the system immediately begins an automatic replacement operation.

 

[molay]

Thank you all again for your generosity and patience in answering!
There are some key points that I will keep in mind to give sound advice to my friends when they use TrueNAS.