Is it posible for multiple zpools to use the same cache disk ?

[vn_mnm]

Hello everyone,

I plan to have a single 500GB NVMe SSD serving as a cache disk for my TrueNAS SCALE system. Is it possible for multiple zpools to use the same cache disk, be it L2ARC or SLOG ? If not, is there a workaround that does not involve adding another NVMe SSD ?

Thank you in advance.

 

[sretalla]

Is it possible for multiple zpools to use the same cache disk, be it L2ARC or SLOG ? If not, is there a workaround that does not involve adding another NVMe SSD ?

Yes, but not in a supported way/via the GUI... it's CLI only.

It has already been well documented in the forums, so I won't re-post that here. (seach for gpart to create the partitions and for zpool add for adding the device to the pool)

Then comes the question ... Why would you want to do that?

Firstly, cache is sort-of equivalent to what L2ARC does, but SLOG is absolutely not a write cache. (so wave goodbye to any hopes you had about adding a super-fast NVME drive and having all writes simply go to that until it's full, then eventually going to the disk slowly later).

L2ARC will also put heavy wear on your SSD... everything written to or read from the pool will cause writes to it and if you don't have a good one with very high write endurance, you'll burn it out quickly (like a year or 2)... all while maybe not even giving you a measurable performance boost... sometimes even potentially slowing you down (since there's always writing going on, even when you're reading).

If you're thinking about SLOG, make sure you understand what it is and why you would want it... at best, what it does is reduce the amount of slowness that comes with doing sync writes, it will never make things go faster than async writes (set sync=disabled on the dataset/zvol).

Have a look at something like this: https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

ALso make sure that before you do either of those, you have already addressed the obvious thing (at least 64GB of RAM)... without that, you'll be unlikely to get any benefit from either L2ARC or SLOG.

 

[vn_mnm]

Yes, but not in a supported way/via the GUI... it's CLI only. It has already been well documented in the forums, so I won't re-post that here. (seach for gpart to create the partitions and for zpool add for adding the device to the pool) Then comes the question ... Why would you want to do that? Firstly, cache is sort-of equivalent to what L2ARC does, but SLOG is absolutely not a write cache. (so wave goodbye to any hopes you had about adding a super-fast NVME drive and having all writes simply go to that until it's full, then eventually going to the disk slowly later). L2ARC will also put heavy wear on your SSD... everything written to or read from the pool will cause writes to it and if you don't have a good one with very high write endurance, you'll burn it out quickly (like a year or 2)... all while maybe not even giving you a measurable performance boost... sometimes even potentially slowing you down (since there's always writing going on, even when you're reading). If you're thinking about SLOG, make sure you understand what it is and why you would want it... at best, what it does is reduce the amount of slowness that comes with doing sync writes, it will never make things go faster than async writes (set sync=disabled on the dataset/zvol). Have a look at something like this: https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/ ALso make sure that before you do either of those, you have already addressed the obvious thing (at least 64GB of RAM)... without that, you'll be unlikely to get any benefit from either L2ARC or SLOG.

Thank you for your time and caring in this comment buddy. I am afraid that diving the drive into multiple partitions each caching a different zpool for different purpose might cause each partition and hence the disk to wear out un-evenly. Will partitioning the disk drive do any changes to the effectiveness of the wear-leveling algorithm embedded in the disk's firmware ?

My super-fast NVMe SSD is this one (the 500 GB version) : https://www.tomshardware.com/reviews/sk-hynix-gold-p31-m2-nvme-ssd-review

Thank you in advance again.

 

[Etorix]

Thank you for your time and caring in this comment buddy. I am afraid that diving the drive into multiple partitions each caching a different zpool for different purpose might cause each partition and hence the disk to wear out un-evenly. Will partitioning the disk drive do any changes to the effectiveness of the wear-leveling algorithm embedded in the disk's firmware ?

Only the manufacturer could tell, but partitioning is likely to have adverse effects.

My super-fast NVMe SSD is this one (the 500 GB version) : https://www.tomshardware.com/reviews/sk-hynix-gold-p31-m2-nvme-ssd-review

Just a (good) consumer drive.
No PLP: Not suitable for a SLOG (which you probably do not need) under any circumstance.
And not the data centre-grade endurance one would like for a L2ARC (assuming you have enough RAM already).

Use the P31 as system drive in a desktop, or to host VM/containers in SCALE.

If you do have a use for a SLOG, get an Optane or Radian RMS.
If you do have a use for a L2ARC, get a data centre SSD with a decent DWPD rating.

 

[vn_mnm]

Only the manufacturer could tell, but partitioning is likely to have adverse effects.


Just a (good) consumer drive.
No PLP: Not suitable for a SLOG (which you probably do not need) under any circumstance.
And not the data centre-grade endurance one would like for a L2ARC (assuming you have enough RAM already).

Use the P31 as system drive in a desktop, or to host VM/containers in SCALE.

If you do have a use for a SLOG, get an Optane or Radian RMS.
If you do have a use for a L2ARC, get a data centre SSD with a decent DWPD rating.

Thanks for your comment buddy. I have some questions regarding partitioning and replacing the cache disk. If @sretalla is still interested in helping me out, I look forward to your opinion too.

1. Do you know how to use the gpart command to automatically divide the cache disk into multiple partitions of equal capacity ?
2. In case of the P31 SSD serving as the cache disk (possibly L2ARC) and dying unexpectedly but I have not yet replaced it with another unit, during that period of time, where will the cached data be written and stored on and will there be any harm on RAM and (or) OS drives ?
3. If, in another scenario, the P31 serves as the OS drive and another slower but 3K P/E ratio serves as the cache drive ( https://www.transcend-info.com/Embedded/Products/No-1164 ), will the system be bottlenecked by it ?

Thank you in advance again.

 

[sretalla]

In case of the P31 SSD serving as the cache disk (possibly L2ARC) and dying unexpectedly but I have not yet replaced it with another unit, during that period of time, where will the cached data be written and stored on and will there be any harm on RAM and (or) OS drives ?

If the L2ARC drive dies, your pool can work without L2ARC (using ARC only, meaning RAM... which is already used in front of the L2ARC when it's in use)... no guarantees of how gracefully it will fail, but it's easy enough to remove a failed L2ARC from the pool and continue as normal right after.

Do you know how to use the gpart command to automatically divide the cache disk into multiple partitions of equal capacity ?

Just eyeball it for the first partition (half the nominal capacity like 256/512) , then don't specify a size for the second and it'll take up the rest.

If you do decide to go down this road, read this: https://openzfs.readthedocs.io/en/latest/performance-tuning.html#whole-disks-vs-partitions

And also this, since you have an NVME device... maybe the stated use case matches for you: https://openzfs.github.io/openzfs-d....html?highlight=overprovision#synchronous-i-o

If, in another scenario, the P31 serves as the OS drive and another slower but 3K P/E ratio serves as the cache drive ( https://www.transcend-info.com/Embedded/Products/No-1164 ), will the system be bottlenecked by it ?

The OS doesn't need to be a fast drive, just don't use a USB stick (nor an SD card), basically anything else is fine.

I would again counsel a strong review of your actual need for any kind of L2ARC or SLOG... generally speaking, most folks who think they need it actually don't and then come back complaining that nothing got faster (because it didn't, since they didn't need the feature in the first place).

 

[vn_mnm]

If the L2ARC drive dies, your pool can work without L2ARC (using ARC only, meaning RAM... which is already used in front of the L2ARC when it's in use)... no guarantees of how gracefully it will fail, but it's easy enough to remove a failed L2ARC from the pool and continue as normal right after.


Just eyeball it for the first partition (half the nominal capacity like 256/512) , then don't specify a size for the second and it'll take up the rest.

If you do decide to go down this road, read this: https://openzfs.readthedocs.io/en/latest/performance-tuning.html#whole-disks-vs-partitions


The OS doesn't need to be a fast drive, just don't use a USB stick (nor an SD card), basically anything else is fine.

I would again counsel a strong review of your actual need for any kind of L2ARC or SLOG... generally speaking, most folks who think they need it actually don't and then come back complaining that nothing got faster (because it didn't, since they didn't need the feature in the first place).

Thank you very much for your advice again. Earlier, I forgot to also include a question about post-partitioning: After having partitioned the cache disk using the gpart command, can I use the webgui to assign these partitions as cache drives to various zpools instead of the cli ?

Well, regarding my usage & hardware, I will be using this rig as both a NAS & VM server hosting a router firewall and a CCTV host.

My motherboard is the Gigabyte X570S Aorus Elite ( https://www.gigabyte.com/Motherboard/X570S-AORUS-ELITE-rev-10) which has 3 m2 nvme slots.

I plan to put 2x Transcend MTE452T2 ssds in the lower slots as OS drives and the SK Hynix P31 in the upper slot as cache drive. I will also be having 2x 3.5” SATA HDDs serving as mirroring NAS data zpool & 1x Samsung 850 Evo 2.5” SATA SSD serving as VM zpool. I may later buy a Samsung 870 QVO 2.5” SATA serving as the CCTV zpool.

 

[sretalla]

After having partitioned the cache disk using the gpart command, can I use the webgui to assign these partitions as cache drives to various zpools instead of the cli ?

No.

zpool add <poolname> cache <device identifier> (I recommend using the gptid even if you can get away with the disk identifier)

If your pools are already SSD, I really have to question why you need any of this. Why not make a pool out of 2 of the NVME drives and leave it there for your VMs, then use a couple of the disk slots for spinning disks instead to back everything up to?

Samsung 870 QVO 2.5” SATA serving as the CCTV zpool.

Those are terrible for write endurance and you'll be replacing them in a year or so.

 

[Etorix]

Ninja'ed by @srellata, but I'll add my two cents.

Do you know how to use the gpart command to automatically divide the cache disk into multiple partitions of equal capacity ?

man gpart
basically you're expected to be a seasoned UNIX admin and do everything by hand.
If you do insist that you want multiple L2ARC and/or SLOG devices and do intend to serve everything from a single drive, get an Optane to have enough endurance and IOPS.
But please do not use "cache", use the proper terminology and say "L2ARC" where you mean "read cache", or "SLOG" where you wrongly mean "write cache" because a SLOG is NOT a write cache (ZFS write cache is the RAM).

In case of the P31 SSD serving as the cache disk (possibly L2ARC) and dying unexpectedly but I have not yet replaced it with another unit, during that period of time, where will the cached data be written and stored on and will there be any harm on RAM and (or) OS drives ?

L2ARC only ever holds a copy of (meta)data and can be removed at any time without prejudice to the data.
For this reason, it may speed reads but never has any effect on writes.

If, in another scenario, the P31 serves as the OS drive and another slower but 3K P/E ratio serves as the cache drive ( https://www.transcend-info.com/Embedded/Products/No-1164 ), will the system be bottlenecked by it ?

If cache ever becomes a bottleneck, the general design is flawed. This can very well happen if one introduces a big L2ARC without having lots of RAM in the first place!
But, in line, with the request above, I no longer pretend to know what you mean by "cache".

I would again counsel a strong review of your actual need for any kind of L2ARC or SLOG... generally speaking, most folks who think they need it actually don't and then come back complaining that nothing got faster (because it didn't, since they didn't need the feature in the first place).

Best advice ever given in this thread!
SLOG is only useful for sync writes. Common workloads have no sync writes. If sync writes happen, performance is only maximised by removing sync writes.
L2ARC should not even be considered with less than 64 GB RAM.

 

[Etorix]

I plan to put 2x Transcend MTE452T2 ssds in the lower slots as OS drives and the SK Hynix P31 in the upper slot as cache drive. I will also be having 2x 3.5” SATA HDDs serving as mirroring NAS data zpool & 1x Samsung 850 Evo 2.5” SATA SSD serving as VM zpool. I may later buy a Samsung 870 QVO 2.5” SATA serving as the CCTV zpool.

There's little need to mirror the boot drive if you keep a copy of the configuration file to restore after replacing a boot drive.
Get a second P31 and mirror your VM pool (sized taking into acount that block storage prefers <50% occupancy). And "spinning rust" for data and CCTV.
If you don't use all SATA ports for HDDs, an old, re-used, 850 EVO should do well as boot drive. Don't bother buying "embedded" 2242 drives for boot, go for any cheap small SSD.

No obvious use for SLOG or L2ARC in your system description.

 

[vn_mnm]

Thank you very much @sretalla @Etorix for all of your kind advice. I have finger fatigue pressing the like button for every single comment of yours.

If I do not need SLOG or L2ARC, should I not need to partition for SWAP upon installation as well ?

 

[Patrick M. Hausen]

TrueNAS will create swap partitions on your HDD pool when you create that.

 

[vn_mnm]

TrueNAS will create swap partitions on your HDD pool when you create that.

It does give an option not to SWAP.

 

[vn_mnm]

man gpart

Can I use Hiren’s Boot to partition the L2ARC drive prior to install TrueNAS !? Post installation, I would already have these partitions on the drive for serving the zpools.

 

[Patrick M. Hausen]

TrueNAS will always overwrite the entire boot drive at installation.

What exactly is your use case for L2ARC vdevs? Do you plan to use at lease 64 GB of RAM or even more? Why would you want an L2ARC for a pool build from SSDs?

 

[Patrick M. Hausen]

It does give an option not to SWAP.

Yes. So? You asked if you should partition for swap in advance. I answered, TrueNAS will do that for you when you create the pool. You definitely want some swap.

 

[vn_mnm]

Why would you want an L2ARC for a pool build from SSDs?

My NAS zpool consists of 3.5” HDDs.

 

[vn_mnm]

Yes. So?

So the YouTuber opted out for SWAP and dug in for L2ARC. Have you watched it ?

 

[vn_mnm]

There's little need to mirror the boot drive if you keep a copy of the configuration file to restore after replacing a boot drive.

After having replaced the boot drive, freshly installed the OS & restored the configuration, will my system recognise and operate all the zpools, zvold as well as VMs properly ?

 

[Patrick M. Hausen]

After having replaced the boot drive, freshly installed the OS & restored the configuration, will my system recognise and operate all the zpools, zvold as well as VMs properly ?

Yes.