Pushing SMB large multi-gigabyte files TO the NAS causes a large transfer to slow and then fail.

[SpiritWolf]

Everyone's jumped all over this one and linked to the excellent paperwork from @Stilez on this one - let's pull the raw numbers out with the following and find out what the "cost" and "return on investment" has been for dedup.

zpool list

zpool status -D poolname

Output in [code][/code] tags please?

As requested...
root@truenas[~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT WolfTruePool 81.8T 18.4T 63.5T - - 18% 22% 2.03x ONLINE /mnt boot-pool 912G 3.87G 908G - - 0% 0% 1.00x ONLINE - root@truenas[~]# zpool status -D WolfTruePool pool: WolfTruePool state: ONLINE scan: scrub repaired 0B in 20:54:22 with 0 errors on Sun Dec 4 20:54:23 2022 config: NAME STATE READ WRITE CKSUM WolfTruePool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/72fc28af-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/7305b845-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72e94646-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72dd14f9-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72f2da11-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 errors: No known data errors dedup: DDT entries 68935787, size 1.20K on disk, 224B in core bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 32.7M 4.07T 3.89T 3.90T 32.7M 4.07T 3.89T 3.90T 2 26.2M 3.26T 3.17T 3.17T 56.8M 7.08T 6.88T 6.88T 4 5.67M 718G 673G 675G 30.7M 3.81T 3.57T 3.58T 8 1.06M 136G 133G 133G 12.1M 1.51T 1.48T 1.48T 16 78.6K 9.76G 8.99G 9.02G 1.35M 171G 155G 156G 32 1.88K 222M 97.4M 104M 85.3K 9.78G 4.04G 4.36G 64 354 17.7M 10.4M 12.0M 28.9K 1.57G 975M 1.08G 128 52 3.84M 1.61M 1.82M 9.03K 660M 264M 301M 256 25 1.98M 700K 795K 8.16K 634M 225M 256M 512 17 1.14M 240K 320K 10.4K 656M 134M 183M 1K 4 512K 16K 28.4K 4.89K 626M 19.6M 34.8M 2K 2 128K 4.50K 14.2K 4.53K 314M 10.8M 32.2M 1M 1 128K 4K 7.10K 1.14M 146G 4.55G 8.09G Total 65.7M 8.18T 7.86T 7.87T 135M 16.8T 16.0T 16.0T

 

[SpiritWolf]

One is enough, and 960 GB is WAY oversized since it only needs to hold 5 seconds of input.


Bump that to 128 GB for dedup.
Else this is a prime case for a (metadata-only) persistent L2ARC, to hold the dedup table. With a bit of command line sorcery, you may actually use two partitions on a single 905p as SLOG and L2ARC devices. (But your use case rather warrants "sync=never" and no SLOG, and 960 GB is too big a L2ARC with only 64 GB RAM.)
Best overall would be to disable dedup and move the data to a new dataset, to remove dedup entirely.

Umm, they were on sale at 399 and I was feeling ornery. I wasn't going to let this problem beat me even at the cost of a pair of those beasts.

Honest answer is, I had NO ******g idea what kind of rabbit hole I had slid into.

 

[jgreco]

I had NO ******g idea what kind of rabbit hole I had slid into.

Around here, we have a hearty respect for anyone who can say that.

 

[Etorix]

Few jump into a rabbit hole knowing where they're going…

Umm, they were on sale at 399 and I was feeling ornery. I wasn't going to let this problem beat me even at the cost of a pair of those beasts.

That was a good opportunity. At worst, you can always sell them at no loss.
The question is whether you have, or can make up, an actual use case for these drives.

 

[SpiritWolf]

Etorix​

"That was a good opportunity. At worst, you can always sell them at no loss.
The question is whether you have, or can make up, an actual use case for these drives."

So...

I have the pair of drives. Would they be something I can work with or should I just return them and buy another 64Gib RAM? I'm trying to wrap my brain around what I can do with what I have.

If RAM is the answer, I can go up to 128GiB but my chances are limited with the equipment I have and I think I've spent *more* than enough. 128 more would be less than the pair of 905ps I've purchased so I'll be guided by you guys with this.

I thought the Optane would help with this as possibly moving the metadata and tables off the primary drives if it can be done; voodoo isn't my strong suit.

 

[HoneyBadger]

As requested...
root@truenas[~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT WolfTruePool 81.8T 18.4T 63.5T - - 18% 22% 2.03x ONLINE /mnt boot-pool 912G 3.87G 908G - - 0% 0% 1.00x ONLINE - root@truenas[~]# zpool status -D WolfTruePool pool: WolfTruePool state: ONLINE scan: scrub repaired 0B in 20:54:22 with 0 errors on Sun Dec 4 20:54:23 2022 config: NAME STATE READ WRITE CKSUM WolfTruePool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/72fc28af-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/7305b845-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72e94646-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72dd14f9-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 gptid/72f2da11-0216-11ed-9816-04421af0b180 ONLINE 0 0 0 errors: No known data errors dedup: DDT entries 68935787, size 1.20K on disk, 224B in core bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 32.7M 4.07T 3.89T 3.90T 32.7M 4.07T 3.89T 3.90T 2 26.2M 3.26T 3.17T 3.17T 56.8M 7.08T 6.88T 6.88T 4 5.67M 718G 673G 675G 30.7M 3.81T 3.57T 3.58T 8 1.06M 136G 133G 133G 12.1M 1.51T 1.48T 1.48T 16 78.6K 9.76G 8.99G 9.02G 1.35M 171G 155G 156G 32 1.88K 222M 97.4M 104M 85.3K 9.78G 4.04G 4.36G 64 354 17.7M 10.4M 12.0M 28.9K 1.57G 975M 1.08G 128 52 3.84M 1.61M 1.82M 9.03K 660M 264M 301M 256 25 1.98M 700K 795K 8.16K 634M 225M 256M 512 17 1.14M 240K 320K 10.4K 656M 134M 183M 1K 4 512K 16K 28.4K 4.89K 626M 19.6M 34.8M 2K 2 128K 4.50K 14.2K 4.53K 314M 10.8M 32.2M 1M 1 128K 4K 7.10K 1.14M 146G 4.55G 8.09G Total 65.7M 8.18T 7.86T 7.87T 135M 16.8T 16.0T 16.0T

So you're getting that 2.03x space savings from deduplication, but you're paying 68935787 * 224B = 15441616288 bytes or ~14.4GiB of RAM to index it. If that was all staying pinned in RAM, it would be working acceptably, but the gstat output from earlier in thread indicates it's not, as your disks are getting hammered with 4K reads during this write operation.

Would be interested to see output from /usr/bin/www/arc_summary.py but unless something's changed recently I was pretty sure the current defaults would allow that entire DDT to live in RAM - unless you have rebooted recently.

Adding a persistent L2ARC for metadata-only (which would catch deduplication) would help on the reads, and is reversible if it doesn't help significantly enough. Because your data vdev is RAIDZ2 though, adding a special/dedup vdev would be permanent.

 

[SpiritWolf]

So you're getting that 2.03x space savings from deduplication, but you're paying 68935787 * 224B = 15441616288 bytes or ~14.4GiB of RAM to index it. If that was all staying pinned in RAM, it would be working acceptably, but the gstat output from earlier in thread indicates it's not, as your disks are getting hammered with 4K reads during this write operation.

Would be interested to see output from /usr/bin/www/arc_summary.py but unless something's changed recently I was pretty sure the current defaults would allow that entire DDT to live in RAM - unless you have rebooted recently.

Adding a persistent L2ARC for metadata-only (which would catch deduplication) would help on the reads, and is reversible if it doesn't help significantly enough. Because your data vdev is RAIDZ2 though, adding a special/dedup vdev would be permanent.

How would it benefit and how would I implement it should be be useful?

 

[HoneyBadger]

How would it benefit and how would I implement it should be be useful?

The deduplication tables are considered ZFS metadata, and as such would be cached onto the L2ARC. This would help alleviate the flurry of reads you're seeing in your gstat output in post #4 ( https://www.truenas.com/community/t...fer-to-slow-and-then-fail.105166/#post-729794 ) which is a result of the deduplication tables being kicked out of RAM for some reason - although it likely shouldn't be getting evicted, which is why I'm looking for the arc_summary information above.

Steps are as follows:

1. Add the device(s) as a cache vdev to your pool - make certain you don't add them as regular vdevs!
2. Enter the line zfs set secondarycache=metadata Poolname/Datasetname from a shell, which will tell ZFS to only use the L2ARC SSD for metadata.
3. Add a new tunable of type sysctl with name vfs.zfs.l2arc.rebuild_enabled and a value of 1

Note that this can increase the time to import a pool on reboots as well, and potentially slow down the middleware and webUI response.

L2ARC

Provides information on L2ARC, caches drives, and persistent L2ARC implementations in TrueNAS CORE and SCALE.

www.truenas.com

 

[SpiritWolf]

The deduplication tables are considered ZFS metadata, and as such would be cached onto the L2ARC. This would help alleviate the flurry of reads you're seeing in your gstat output in post #4 ( https://www.truenas.com/community/t...fer-to-slow-and-then-fail.105166/#post-729794 ) which is a result of the deduplication tables being kicked out of RAM for some reason - although it likely shouldn't be getting evicted, which is why I'm looking for the arc_summary information above.

Steps are as follows:

1. Add the device(s) as a cache vdev to your pool - make certain you don't add them as regular vdevs!
2. Enter the line zfs set secondarycache=metadata Poolname/Datasetname from a shell, which will tell ZFS to only use the L2ARC SSD for metadata.
3. Add a new tunable of type sysctl with name vfs.zfs.l2arc.rebuild_enabled and a value of 1

Note that this can increase the time to import a pool on reboots as well, and potentially slow down the middleware and webUI response.

L2ARC

Provides information on L2ARC, caches drives, and persistent L2ARC implementations in TrueNAS CORE and SCALE.

www.truenas.com

So, while I was looking to see if my hardware would show up after installing it, I saw this:

And saw that the hardware *did* show up (using a PCI-E card with the NVME adaptor to the OPTANE device) and saw the above.

A few things stood out for me:
1) It's there!,
2) I have choices!,
3) It says if I use a single/multi device as a stripe, Hell could reign on earth in a data-rific horror display and I'll be doomed for eternity if I ignore it's decree. Seems like it knows something I was aware of through my readings, but need to make certain of.

Ok; I do understand that things I don't understand can byte me. So i'm curious...

I do have a second device but I don't have a second PCIE adaptor; I guess I'd have to get one should I use the wee beasties as an L2ARC device. While I'm ordering and waiting for the PCI , and in no particular order I have a few questions in my head...

1) I see that it understands dedup tables as a last choice in that drop-down list. What does that do that won't match your previous steps (zfs set secondarycache=metadata Poolname/Datasetname)?
2) Does the "zfs set secondarycache=metadata Poolname/Datasetname" as implemented cash the entire pool or just certain datasets? And only the DeDupe parts?
3) Are there any other things I should show you before I go forth and Newegg/Amazon my way with another PCI card? Any other outputs from the CLI?
4) As I have two OPTANE devices, I can certainly make the cache pool mirror. Noting that last DeDup choice, is there something to the notion of partitioning the beasties to make two mirror-pools and placing the first VDEV cache in one part, do...something else in the second partition pool? I read that elsewhere and thought it was interesting as a larger-than-nessessary cache pool could sometimes be deleterious in varied cases. Might be good to know when those cases are useful and when they're not as these things seem to have wicked I/O in comparison to the spinning rust I have now and have long life-spans.
5) I read through the L2ARC section you had a link to, of course. I it says: "Cached drives are always striped, not mirrored."
Now that's odd as the TrueNAS CORE cache Vdev is VERY specific about that Hell-Reigning-On-Earth scenario and the warning is exactly opposite. It's even in red. Prophetic or something I've missed?

I should reiterate, as the thread is getting a bit long, that read speeds are usually ok, but writing to the box is good then slopes off to about zero.

There's other stuff I was thinking of but I didn't want to hurt your brain more than I have to.

Thanx in advance - again.

 

[Etorix]

3) It says if I use a single/multi device as a stripe, Hell could reign on earth in a data-rific horror display and I'll be doomed for eternity if I ignore it's decree.

If you got that after selecting "Cache" the GUI is actually wrong because the loss of a L2ARC device does not result in data loss: It only hold a copy (cache) of what is safely stored in the pool.
The warning does apply to data vdevs and special vdevs.

4) As I have two OPTANE devices, I can certainly make the cache pool mirror. Noting that last DeDup choice, is there something to the notion of partitioning the beasties to make two mirror-pools and placing the first VDEV cache in one part, do...something else in the second partition pool?

It is actually possible to partition an Optane drive and use the partitions for separate purposes, but this is not officially supported (mandatory warning: HERE BE DRAGONS!) and you'll have to work it out from the CLI, and use the CLI to maintain if necessary. I do have an Optane 900p partitioned for use as both a SLOG and L2ARC.
But do you have the use case for a partitioned drive?

 

[SpiritWolf]

If you got that after selecting "Cache" the GUI is actually wrong because the loss of a L2ARC device does not result in data loss: It only hold a copy (cache) of what is safely stored in the pool.
The warning does apply to data vdevs and special vdevs.

Rodger Rodger...so I can use a single device for it for the moment and not fear if I read you correctly.

It is actually possible to partition an Optane drive and use the partitions for separate purposes, but this is not officially supported (mandatory warning: HERE BE DRAGONS!) and you'll have to work it out from the CLI, and use the CLI to maintain if necessary. I do have an Optane 900p partitioned for use as both a SLOG and L2ARC.
But do you have the use case for a partitioned drive?

That's the point, do I? You're using it your drive partitioned for both the L2 and the SLOG; I have two and I can mirror them if need be. I have LOTS of room. These Intel Optane 960 905P units are kinda pricy and they were going for a decent price on Newegg.

As to your point, do I have the use case? I can't tell honestly. I've made Linux, Mac and Windows servers but this is the rabbit hole of rabbit holes. SO many interdependencies and things to consider.

I don't mind using the CLI; not at all. Been doing them since Mac in the 2010s to Linux in the 2020s (now 2030s). Would partitioning both drives as dual-mirrored-pairs be a possible fix to my ills? Im more than happy to. Although I'm not a deft hand at gpart, usually working with Linux and Mac, both *NIX OSs, I can usually find my way around if the depth of my understanding is broad enough.

As you can see by my screenshot, I have 800+gigs of space on each of the drives. If I partitioned them both by, say half, for example, would the TrueNAS OS say to itself, "Gee, there are 4 volumes here and I can use them as mirrors (on different devices, of course) as special-case vdevs"?

That way i can devote 400 GiB of data to two mirrors and one be an L2ARC and the other as something useful? At that point, I'd be in a mirrored state with two devices for redundancy and still live if a device failed.

It may be a case of "just try the L2ARK and see if that helps" and thats why I'm knowledge-transferring your Guru stuff to my more limited experience.

I have HoneyBadgers' info and I have a single 905P in there now, I can just try it and see if it solves "a thing".
Would I have to do something special to remove it?

 

[Etorix]

Rodger Rodger...so I can use a single device for it for the moment and not fear if I read you correctly.

L2ARC does not need to be mirrorred. That's the benefit of hosting a DDT in persistent L2ARC. It is also reversible, if the result is not good enough.
Special vdev should be mirrorred, quite possibly 3-way.

As to your point, do I have the use case? I can't tell honestly. I've made Linux, Mac and Windows servers but this is the rabbit hole of rabbit holes. SO many interdependencies and things to consider.

That's relatively simple. SLOG is for sync writes only; it does nothing for async.
Sync writes are an order of magnitude slower than async. Sync writes with a SLOG are better than without, but still a lot worse than async.
If you do not absolutely need sync writes, go for async and forget about SLOG.

As you can see by my screenshot, I have 800+gigs of space on each of the drives. If I partitioned them both by, say half, for example, would the TrueNAS OS say to itself, "Gee, there are 4 volumes here and I can use them as mirrors (on different devices, of course) as special-case vdevs"?

No. TrueNAS always use whole drives, not partitions. That's why you have to use the CLI, and take all responsibility for the unsupported setup.
Special vdevs are pool-critical, and thus would be especially problematic in an unsupported setting.
In my case, SLOG and L2ARC are devices which can be removed from a pool, so I can always go back to a supported configuration if I run into issues.
A special vdev would remove any need for a metadata L2ARC (and probably for any kind of L2ARC). I essentially submitted above that you do not need a SLOG. So if you go for a special vdev, mirror two or three (whole!) Optane drives to host metadata (including DDT) and small files (or even not so small, since the drives are way larger than necessary). But do not go this route lightly: With RAIDZ vdevs in the mix, the special vdev cannot be removed without destroying the pool, so the choice is irreversible.

I have HoneyBadgers' info and I have a single 905P in there now, I can just try it and see if it solves "a thing".
Would I have to do something special to remove it?

Storage > Pool > (gear) > Status > (cache, 3-dot) > Remove

 

[SpiritWolf]

L2ARC does not need to be mirrorred. That's the benefit of hosting a DDT in persistent L2ARC. It is also reversible, if the result is not good enough.
Special vdev should be mirrorred, quite possibly 3-way.


That's relatively simple. SLOG is for sync writes only; it does nothing for async.
Sync writes are an order of magnitude slower than async. Sync writes with a SLOG are better than without, but still a lot worse than async.
If you do not absolutely need sync writes, go for async and forget about SLOG.


No. TrueNAS always use whole drives, not partitions. That's why you have to use the CLI, and take all responsibility for the unsupported setup.
Special vdevs are pool-critical, and thus would be especially problematic in an unsupported setting.
In my case, SLOG and L2ARC are devices which can be removed from a pool, so I can always go back to a supported configuration if I run into issues.
A special vdev would remove any need for a metadata L2ARC (and probably for any kind of L2ARC). I essentially submitted above that you do not need a SLOG. So if you go for a special vdev, mirror two or three (whole!) Optane drives to host metadata (including DDT) and small files (or even not so small, since the drives are way larger than necessary). But do not go this route lightly: With RAIDZ vdevs in the mix, the special vdev cannot be removed without destroying the pool, so the choice is irreversible.


Storage > Pool > (gear) > Status > (cache, 3-dot) > Remove

ok! I'm going to run they add for the L2ARC and see what happens... I'm kinda excited (sue me, I'm enjoying all this).

 

[SpiritWolf]

The deduplication tables are considered ZFS metadata, and as such would be cached onto the L2ARC. This would help alleviate the flurry of reads you're seeing in your gstat output in post #4 ( https://www.truenas.com/community/t...fer-to-slow-and-then-fail.105166/#post-729794 ) which is a result of the deduplication tables being kicked out of RAM for some reason - although it likely shouldn't be getting evicted, which is why I'm looking for the arc_summary information above.

Steps are as follows:

1. Add the device(s) as a cache vdev to your pool - make certain you don't add them as regular vdevs!
2. Enter the line zfs set secondarycache=metadata Poolname/Datasetname from a shell, which will tell ZFS to only use the L2ARC SSD for metadata.
3. Add a new tunable of type sysctl with name vfs.zfs.l2arc.rebuild_enabled and a value of 1

Note that this can increase the time to import a pool on reboots as well, and potentially slow down the middleware and webUI response.

L2ARC

Provides information on L2ARC, caches drives, and persistent L2ARC implementations in TrueNAS CORE and SCALE.

www.truenas.com

Done!

Awaiting reboot and see if I orked it up or it's running...

 

[Etorix]

It may take a while until the L2ARC fills up.
I would launch a scrub to make sure all metadata is loaded, but this may be overkill.

 

[SpiritWolf]

It may take a while until the L2ARC fills up.
I would launch a scrub to make sure all metadata is loaded, but this may be overkill.

This is going to sound like vudu...

I'm running the unit now and its working. Except, I can't see any activity in the cache.

Perhaps the UI is not exactly what I need to be looking at. Or I configured it improperly.

Could you suggest a few CLI metrics I can pass look up and show you guys and see if what I *thought* did was actually what I did?

 

[Davvo]

Could you suggest a few CLI metrics I can pass look up and show you guys and see if what I *thought* did was actually what I did?

Try arcstat and arc_summary.

 

[SpiritWolf]

yessir!

Code:

arc_summary
------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Jan 02 23:36:18 2023
FreeBSD 13.1-RELEASE-p2                                    zpl version 5
Machine: truenas.local (amd64)                          spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    92.3 %   53.1 GiB
        Target size (adaptive):                        92.2 %   53.0 GiB
        Min size (hard limit):                          3.5 %    2.0 GiB
        Max size (high water):                           28:1   57.5 GiB
        Most Frequently Used (MFU) cache size:         39.8 %   20.3 GiB
        Most Recently Used (MRU) cache size:           60.2 %   30.7 GiB
        Metadata cache size (hard limit):              75.0 %   43.1 GiB
        Metadata cache size (current):                 10.7 %    4.6 GiB
        Dnode cache size (hard limit):                 10.0 %    4.3 GiB
        Dnode cache size (current):                    21.0 %  928.6 MiB

ARC hash breakdown:
        Elements max:                                               1.5M
        Elements current:                             100.0 %       1.5M
        Collisions:                                               875.5k
        Chain max:                                                     4
        Chains:                                                   118.4k

ARC misc:
        Deleted:                                                   10.1M
        Mutex misses:                                                114
        Eviction skips:                                             7.7k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                   4.1 GiB
        L2 eligible MFU evictions:                     46.6 %    1.9 GiB
        L2 eligible MRU evictions:                     53.4 %    2.2 GiB
        L2 ineligible evictions:                                 1.2 TiB

ARC total accesses (hits + misses):                                 8.3G
        Cache hit ratio:                              100.0 %       8.3G
        Cache miss ratio:                             < 0.1 %       1.6M
        Actual hit ratio (MFU + MRU hits):            100.0 %       8.3G
        Data demand efficiency:                        69.4 %       2.9M
        Data prefetch efficiency:                       6.3 %     165.8k

Cache hits by cache type:
        Most frequently used (MFU):                    99.8 %       8.3G
        Most recently used (MRU):                       0.2 %      14.8M
        Most frequently used (MFU) ghost:             < 0.1 %     104.5k
        Most recently used (MRU) ghost:               < 0.1 %      29.0k
        Anonymously used:                             < 0.1 %     297.0k

Cache hits by data type:
        Demand data:                                  < 0.1 %       2.0M
        Demand prefetch data:                         < 0.1 %      10.5k
        Demand metadata:                               99.9 %       8.3G
        Demand prefetch metadata:                       0.1 %       5.2M

Cache misses by data type:
        Demand data:                                   54.4 %     886.9k
        Demand prefetch data:                           9.5 %     155.3k
        Demand metadata:                               30.5 %     496.8k
        Demand prefetch metadata:                       5.6 %      91.5k

DMU prefetch efficiency:                                          909.1k
        Hit ratio:                                     26.0 %     236.7k
        Miss ratio:                                    74.0 %     672.3k

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                 969216
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                              61713000000
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           347464
        arc_max                                              61713000000
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        btree_verify_intensity                                         0
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3 avx2
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.exclude_special                                          0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               0
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          1
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                       40000000
        l2arc.write_max                                         10000000
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               0
        l2arc_norw                                                     0
        l2arc_write_boost                                       40000000
        l2arc_write_max                                         10000000
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               14
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       18436625920
        mfu_ghost_data_esize                                 15392326144
        mfu_ghost_metadata_esize                              1142402048
        mfu_ghost_size                                       16535033856
        mfu_metadata_esize                                      18134528
        mfu_size                                             21690452480
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       30884521472
        mru_ghost_data_esize                                 10056774656
        mru_ghost_metadata_esize                              2258934784
        mru_ghost_size                                       12315709440
        mru_metadata_esize                                    1523879424
        mru_size                                             32973550592
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        9000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          14
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                               10
        vdev.nia_delay                                                 2
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          8
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022101100-zfs_2a0dae1b7
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        wrlog_data_max                                        8589934592
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1


VDEV cache disabled, skipping section

ZIL committed transactions:                                          909
        Commit requests:                                             144
        Flushes to stable storage:                                   144
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:        1.8 MiB        136

 

[SpiritWolf]

Could be my lack of ZFS-fu, but it looks like the cache I thought I started, didn't.

I suspect it was the zfs set secondarycache=metadata Poolname/Datasetname command that went poof. It wouldn't go as written even when I substituted pname/data name with my pool as it was (WolfTruePool).
I didn't see an indication of action which is what I expected if the CLI command went through (zfs set secondarycache=metadata WolfTruePool). I saw no error.

Tuneable went in, no apparent error...

Yet, I'm being ignored. I guess that's preferable to "all your data is gone", but it seems to lack flavor even after I rebooted.

 

[Davvo]

I didn't follow the thread, but why did you create another pool? Shouldn't you add the cache device as a special vdev in the pool you want to use it?
That could be the reason the secondarycache command didn't work.