Reality Check on Storage Upgrade Plan

[JamesNisly]

I’d love it if I could just get a quick reality check on my planned method to upgrade the storage in our main file server. We’re a photography studio mostly dealing in large quantities of 20-50MB raw and jpg image files and 5k-ish xmp files.

We are currently running a 7 wide Z2 vdev with 8TB drives in a 12 bay SuperMicro chassis. We have about 22TiB stored on there bringing us to about 66% of our space used. We have another big year coming up with a lot of new data coming in, so it’s time to expand.

I was able to snag 9 16TB Western Digital Reds on Black Friday deals for a pretty decent price so those are on the way. The goal is to use 8 of them for storage and one as a spare.

Since I only have five empty bays in the chassis, my plan was to:

1. Use four of those bays to create a four wide Z1 vdev pool with the 16TB drives
2. Replicate our 22TiB of data from the old pool to the new pool
3. Remove old drives, and add four more 16TB drives in a second Z1 vdev to stripe with the first

This should leave me with a pool with two striped four wide Z1 vdevs. Based on what I read here this should get is slightly better IOPS for all those image and XMP files but slightly less fault tolerance than an 8 wide single Z2 pool. Correct?

The alternative would be to install all 8 16TB drives at once in a single Z2 vdev and then replicate from the backup server, but I‘d expect that would take a LOT longer our our 1Gbe network than it would do to it all within the same system.

Once I’m confident that the new pool is up and running, I’m also considering adding a few mirrored SSDs as a metadata special vdev to the pool to help when browsing those directories of thousands and thousands of files.

Are there any major blind spots here that I should be aware of? Am I missing anything major?

Thanks so much for any insight you might have.

 

[Etorix]

This should leave me with a pool with two striped four wide Z1 vdevs. Based on what I read here this should get is slightly better IOPS for all those image and XMP files but slightly less fault tolerance than an 8 wide single Z2 pool. Correct?

Correct but for that "slightly". With very large drives (16 TB here), losing any single drive in a raidz1 vdev loses any redundancy and raises the risk of losing some files to an URE while resilvering. That would be limited damage, which may be acceptable if there's a backup to restore from, but still a hassle.
With 12 bays, two 6-wide raidz2 vdevs would be safer.

Possible ways to that could be:

• remove one 8 TB drive, create a 6*16 TB raidz2 pool, replicate from the degraded old pool to the new one, destroy the old pool and reuse 6*8 TB as a second vdev for the new pool;
• create a degraded 6-wide raidz2 pool with 5 drives only, replicate from old 7-wide pool to the new (degraded) pool, remove old pool, add the 6th drive and let the new pool resilver, reuse 6*8 TB as second vdev.

Add four more 16 TB drives (three for the pool and one spare) and you can upgrade to 2*(6*16 TB) for 8*16=128 TB of raw storage space in reasonably safe raidz2. That should be good for a while. (Some more RAM may come handy though.)

Once I’m confident that the new pool is up and running, I’m also considering adding a few mirrored SSDs as a metadata special vdev to the pool to help when browsing those directories of thousands and thousands of files.

A special vdev is critical for the pool: Lose it, lose the whole pool. So you'd want at least a 3-way mirror here.
If this mostly about browsing, i.e. reading, most of the benefits could be achieved with a single SSD (preferably NVMe) as persistent metadata-only L2ARC. This does not speed up writes, which have to go through the HDD pool but it speeds up reads. The metadata L2ARC is not critical, since it only holds a copy of what's on the pool; if you lose the L2ARC, you lose performance but the data is still safe.

 

[JamesNisly]

Thank you so much, Etorix. I hadn't even considered replicating from a degraded pool. That definitely opens up some options with regards to available drive bays. I do have a backup of everything on both a TrueNas Mini and in Backblaze but obviously restoring would be a time consuming hassle.

I'll look into persistent metadata-only L2ARC. I have not heard of that. There's a step in our workflow where one bit of software has to write out an XMP file for every image in the folder (sometimes as many as 10k files) and then anther bit of software has to read them in. This is usually the point where performance just takes a nose dive on all that spinning rust. You might as well go grab lunch while it churns.

Increased write AND read speeds of all these little files would be great but even just an increase on the write side would improve things.

I have three 1TB Red SSDs coming that I was considering for the metadata only vdev but I had not yet decided whether to mirror two and have a spare or just mirror all three.

 

[HoneyBadger]

One additional thing that I've noticed is that your signature lists your clients as "OSX via SMB" - have you looked at the option of setting the SMB advanced parameter strict sync = no? This might significantly improve the throughput, especially on the small (5K) XMP file writes.

Because you're not likely ever doing an "overwrite in place" on any files, a sudden loss of connectivity would still allow you to re-save your work locally or to another location.

See the attached screenshot:

 

[JamesNisly]

One additional thing that I've noticed is that your signature lists your clients as "OSX via SMB" - have you looked at the option of setting the SMB advanced parameter strict sync = no? This might significantly improve the throughput, especially on the small (5K) XMP file writes.

Because you're not likely ever doing an "overwrite in place" on any files, a sudden loss of connectivity would still allow you to re-save your work locally or to another location.

See the attached screenshot:

View attachment 60421

Funny you should mention that. I JUST found out about that in this thread not more than 20 minutes ago and I'm going to give that a try. Thanks!

 

[HoneyBadger]

Funny you should mention that. I JUST found out about that in this thread not more than 20 minutes ago and I'm going to give that a try. Thanks!

Let us know the results. Forcing sync=disabled on the entire dataset itself is a bit of a blunt-instrument way to do it, so the SMB setting is possibly a better way to start.

If this change doesn't resolve it, then we'd want to see if you're binding up on metadata read or write speed. Read speed you could correct with either the pL2ARC for meta-only, or forcing a higher "minimum metadata floor" value in RAM if it's being ejected too aggressively. Metadata write speeds would need the meta-only special vdev to accelerate, but I suspect that if it's an IOPS level limit, taking the sync-write pressure of your data off of the disks would free up a lot of spindle time - hopefully enough so that the special vdev isn't needed.

And beyond that, with 16x spinning disks, you're likely well into the area where 10Gbps starts to make a good deal of sense.

 

[JamesNisly]

I tried adding it to the share and restarting the SMB service but it borked my connection pretty bad so I reverted back for the time being. Is that something that likely needs a full reboot? A couple of other people are using the server right now so it's not the best time to try that, but I will later today.

The 10Gbe upgrade is already underway. The server is upgraded and running at 10Gbe to the rack switch, but I'm just waiting for another switch and a couple more thunderbolt to 10gbe adapters to arrive for the office. Unfortunately, I don't know if any method to upgrade my TrueNAS mini to 10Gbe, so any restore from that is going to be 1Gbe. If there's something that I don't know about, I'm all ears.

 

[HoneyBadger]

I tried adding it to the share and restarting the SMB service but it borked my connection pretty bad so I reverted back for the time being. Is that something that likely needs a full reboot? A couple of other people are using the server right now so it's not the best time to try that, but I will later today.

strict sync = no in the auxiliary parameter field shouldn't haven't borked anything. Was it clients refusing to connect, or did something else happen?

You can always use the "blunt instrument" approach of setting sync=disabled which will let SMB remain unchanged but just have ZFS lie to it.

The 10Gbe upgrade is already underway. The server is upgraded and running at 10Gbe to the rack switch, but I'm just waiting for another switch and a couple more thunderbolt to 10gbe adapters to arrive for the office. Unfortunately, I don't know if any method to upgrade my TrueNAS mini to 10Gbe, so any restore from that is going to be 1Gbe. If there's something that I don't know about, I'm all ears.

The TN Mini datasheet seems to suggest that there's no PCIe slot available in the Mini E or Mini X chassis, so unfortunately you're limited to the 4x1Gb interfaces there.

 

[JamesNisly]

After a few minutes of adding that aux param to that particular share, all of the shares from that server disappeared from finder on my computer. I could still see the server under Network, but if I clicked on it, I just got "Connecting as..." and none of the shares would appear.

At the same time, I also got this alert:

Once I deleted it and restarted SMB services, it all came back right away. It could have been a fluke and I want to try again when I can give everything a fresh reboot.

Just to be clear, putting strict sync = no in the auxiliary parameter field of an SMB share is NOT the same as having sync disabled on the dataset, right? I already have that disabled on all of my datasets.

It looks like I could also add it as an auxiliary parameter of the SMB service, but I don't think I want that because I do have a dataset that we use for TimeMachine Backups.

 

[Etorix]

Thank you so much, Etorix. I hadn't even considered replicating from a degraded pool.

That's riskier than replicating to a degraded pool (second option) but it saves one resilver step. If it can be done while no-one is using the pool, the risk is probably quite low.

I'll look into persistent metadata-only L2ARC. I have not heard of that. There's a step in our workflow where one bit of software has to write out an XMP file for every image in the folder (sometimes as many as 10k files) and then anther bit of software has to read them in. This is usually the point where performance just takes a nose dive on all that spinning rust. You might as well go grab lunch while it churns.

Enough spinners in raidz2 should be able to write as fast as one SATA SSD. The bottleneck in your use case may be IOPS for the subsequent read.

I have three 1TB Red SSDs coming that I was considering for the metadata only vdev but I had not yet decided whether to mirror two and have a spare or just mirror all three.

Definitely at least three, else the special vdev would be the weak point (can only sustain a single drive loss).
A special vdev cannot be removed from a pool with raidz2 vdevs, and should preferably be introduced at pool creation. A pL2ARC can be introduced and removed at any time, so it may be worth testing that first with a 256-512 MB NVMe SSD.

 

[JamesNisly]

To follow up, strict sync = no is working fine now after a reboot. Not sure what was going on the other day.

That being said, it didn't make a dramatic difference with the XMP writing and reading. Browsing folders of images in Bridge might be a touch snappier but it's hard to say.

After HBAs and the 10Gbe NIC, all I have left for PCI slots are PCI-X 133MHz on my aging X8DTN+. Is it even worth trying to get an NVMe on one of those?

As for the pool upgrade plans, I think I may make temporary use of the motherboard's SATA ports in order to connect up enough drives to replicate from my current 7 wide pool to a new 8 wide Z2 pool.

 

[HoneyBadger]

That being said, it didn't make a dramatic difference with the XMP writing and reading. Browsing folders of images in Bridge might be a touch snappier but it's hard to say.

If this hasn't helped, it's probably just the nature of "small file on wide vdev" that happens with RAIDZ2. If your XMP file is compressing down to 4K or smaller, the Z2 nature means it'll write one sector worth of data and two for parity - a bit of an amplification effect.

The challenge is that while the XMP files are "metadata" from the application level, ZFS sees them as a regular file - so while metadata-specific tuning might help the overhead of those small files, the data reading might still need help.

The full output of arc_summary.py inside of [CODE][/CODE] tags would help as well to break down your hits/misses by data vs metadata. If we see a bunch of meta misses, it might be worthwhile tossing in a sysctl tunable with the name vfs.zfs.arc.meta_min and a value of 4294967296 (or 4GiB) - this will tell ZFS never to evict your metadata unless it consumes more than that. (You can see the current amount of metadata living in ARC with a simple one-liner: sysctl kstat.zfs.misc.arcstats.metadata_size)

If it's truly bottlenecking on metadata writes, or the data-level XMP file read/writes, this might actually be a situation where I'd consider a special vdev (and optionally "special_small_files" if it's the XMP data itself) as a solution. As mentioned by @Etorix above there are some significant considerations here, including the need for strong redundancy (a 3-way mirror would be recommended) as well as the inclusion of RAIDZ in your pool making it a "one-way trip" once added. I'd suggest doing this in a test-case environment with a smaller pool of duplicated data first, to see if it scales out properly. Advantage is that your workload is pretty neatly split along the "RAW/JPG" vs "XMP" line, so you would be fairly confident of only the XMP files being stored on SSD.

After HBAs and the 10Gbe NIC, all I have left for PCI slots are PCI-X 133MHz on my aging X8DTN+. Is it even worth trying to get an NVMe on one of those?

Technological limitations will preclude this - PCI-X has no concept of NVMe, it's an extension of the legacy PCI model rather than the newer PCIe, so there's no way to wedge an NVMe SSD in there.

Questions of my own follow:

1. I'm not too familiar with Adobe Bridge; is it possible to tell it to store the XMP metadata files in a separate folder/location apart from the file? This could be set as a separate ZFS dataset on SSD if so.

2. What's your current recordsize for your dataset (zfs get recordsize from a shell)? With your files being multiple MB in size, a larger size like 1MB would benefit here, and as this value is a maximum, it wouldn't change how the smaller XMP files are handled.

 

[JamesNisly]

Thank you so much for taking the time to help me through this.

This is probably an embarrassingly simple question but is arc_summary.py a command, script or a file? Or something else? I tried running it in shell and just got "command not found". If it's a file, any idea where to find it?


Here's my current metadata size in ARC.

Code:

root@Leviathan:/ # sysctl kstat.zfs.misc.arcstats.metadata_size
kstat.zfs.misc.arcstats.metadata_size: 1236632064

And record size:

Code:

root@Leviathan:/ # zfs get recordsize Rasputin


NAME      PROPERTY    VALUE    SOURCE


Rasputin  recordsize  128K     default

I see a menu option right there in the GUI to change that. Any issues with changing it to 1M on existing datasets with data on them? Does it need a reboot to take affect? And this wouldn't apply retroactively; only to new data, correct?

1. I'm not too familiar with Adobe Bridge; is it possible to tell it to store the XMP metadata files in a separate folder/location apart from the file? This could be set as a separate ZFS dataset on SSD if so.

You are right about the XMP files containing metadata for the images, but as far as ZFS is concerned, they are just files to be stored. Unfortunately, the XMP files generated by our culling software need to live alongside the RAW files in order for Lightroom to be able to read them.


I'm fulling prepared to go down the triple mirrored special vdev route if that's what it takes. I have three WD Red 1TB SSDs in hand and trays on the way. Testing all of this out on a smaller duplicated pool sounds like very wise advice.

 

[HoneyBadger]

Thank you so much for taking the time to help me through this.

This is probably an embarrassingly simple question but is arc_summary.py a command, script or a file? Or something else? I tried running it in shell and just got "command not found". If it's a file, any idea where to find it?

Should be runnable from the shell - I'd recommend opening up an SSH session directly to TrueNAS though for easier copy-and-paste, or redirect the output to a file you can copy.

Full path is /usr/bin/www/arc_summary

Here's my current metadata size in ARC.

Code:

root@Leviathan:/ # sysctl kstat.zfs.misc.arcstats.metadata_size
kstat.zfs.misc.arcstats.metadata_size: 1236632064

So your metadata is only using about 1.2G of your RAM. You should have basically zero metadata misses here. If we have to force it to keep more, we'll do that.

And record size:

Code:

root@Leviathan:/ # zfs get recordsize Rasputin


NAME      PROPERTY    VALUE    SOURCE


Rasputin  recordsize  128K     default

I see a menu option right there in the GUI to change that. Any issues with changing it to 1M on existing datasets with data on them? Does it need a reboot to take affect? And this wouldn't apply retroactively; only to new data, correct?

No issue with changing this to 1M live - no reboot needed, but as you suspect it will only apply to new data.

You are right about the XMP files containing metadata for the images, but as far as ZFS is concerned, they are just files to be stored. Unfortunately, the XMP files generated by our culling software need to live alongside the RAW files in order for Lightroom to be able to read them.

Darn it. There goes the "easy approach" of bumping those files to a separate dataset on NAND.

I'm fulling prepared to go down the triple mirrored special vdev route if that's what it takes. I have three WD Red 1TB SSDs in hand and trays on the way. Testing all of this out on a smaller duplicated pool sounds like very wise advice.

Definitely test it out, since adding the special vdev to a RAIDZ2 is a non-reversible change. You could likely use that additional HDDs you're planning to use for expansion to get a large enough data set to validate that performance gets a significant improvement out of it.

 

[JamesNisly]

ETA: I have not done the XMP read/write operation since rebooting the server this morning. I'm not sure how far back this summary goes but I can run it again after a cycle of XMP file read/writes.

Alright, got it. Here's the output of arc_summary.

Code:

root@Leviathan:/usr/bin # arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 30 12:21:37 2022
FreeBSD 13.1-RELEASE-p1                                    zpl version 5
Machine: Leviathan.local (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    34.5 %   49.3 GiB
        Target size (adaptive):                        34.6 %   49.4 GiB
        Min size (hard limit):                          3.1 %    4.5 GiB
        Max size (high water):                           31:1  142.9 GiB
        Most Frequently Used (MFU) cache size:         42.8 %   19.6 GiB
        Most Recently Used (MRU) cache size:           57.2 %   26.1 GiB
        Metadata cache size (hard limit):              75.0 %  107.2 GiB
        Metadata cache size (current):                 10.0 %   10.7 GiB
        Dnode cache size (hard limit):                 10.0 %   10.7 GiB
        Dnode cache size (current):                    17.7 %    1.9 GiB

ARC hash breakdown:
        Elements max:                                             866.2k
        Elements current:                             100.0 %     866.2k
        Collisions:                                                14.3k
        Chain max:                                                     2
        Chains:                                                    10.9k

ARC misc:
        Deleted:                                                      46
        Mutex misses:                                                  0
        Eviction skips:                                             4.7k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 745.5 KiB
        L2 eligible MFU evictions:                     30.0 %  224.0 KiB
        L2 eligible MRU evictions:                     70.0 %  521.5 KiB
        L2 ineligible evictions:                                 6.0 KiB

ARC total accesses (hits + misses):                               540.5M
        Cache hit ratio:                               99.8 %     539.6M
        Cache miss ratio:                               0.2 %     857.8k
        Actual hit ratio (MFU + MRU hits):             98.7 %     533.7M
        Data demand efficiency:                        94.9 %       3.7M
        Data prefetch efficiency:                       2.0 %     146.7k

Cache hits by cache type:
        Most frequently used (MFU):                    98.5 %     531.3M
        Most recently used (MRU):                       0.4 %       2.4M
        Most frequently used (MFU) ghost:               0.0 %          0
        Most recently used (MRU) ghost:                 0.0 %          0
        Anonymously used:                               1.1 %       5.9M

Cache hits by data type:
        Demand data:                                    0.6 %       3.5M
        Demand prefetch data:                         < 0.1 %       3.0k
        Demand metadata:                               97.8 %     528.0M
        Demand prefetch metadata:                       1.5 %       8.1M

Cache misses by data type:
        Demand data:                                   21.9 %     187.7k
        Demand prefetch data:                          16.8 %     143.8k
        Demand metadata:                               45.0 %     386.0k
        Demand prefetch metadata:                      16.4 %     140.3k

DMU prefetch efficiency:                                          504.9k
        Hit ratio:                                     26.5 %     134.0k
        Miss ratio:                                    73.5 %     370.9k

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                5319680
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           783276
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       12898283520
        mfu_ghost_data_esize                                           0
        mfu_ghost_metadata_esize                                       0
        mfu_ghost_size                                                 0
        mfu_metadata_esize                                    1384842752
        mfu_size                                             21011129856
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       25745052672
        mru_ghost_data_esize                                           0
        mru_ghost_metadata_esize                                       0
        mru_ghost_size                                                 0
        mru_metadata_esize                                     489703424
        mru_size                                             28043150848
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022081800-zfs_27f9f911a
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         1.4k
        Commit requests:                                             212
        Flushes to stable storage:                                   212
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:        2.6 MiB        175

 

[JamesNisly]

And here is arc_summary immediately after writing and then reading about 500 xmp files into Lightroom. As well as some browsing of finished jobs in bridge, which is mostly folders of 1000-ish 5-15MB jpeg files.

Code:

root@Leviathan:~ # arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 30 13:18:46 2022
FreeBSD 13.1-RELEASE-p1                                    zpl version 5
Machine: Leviathan.local (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    37.2 %   53.1 GiB
        Target size (adaptive):                        37.2 %   53.2 GiB
        Min size (hard limit):                          3.1 %    4.5 GiB
        Max size (high water):                           31:1  142.9 GiB
        Most Frequently Used (MFU) cache size:         44.0 %   21.8 GiB
        Most Recently Used (MRU) cache size:           56.0 %   27.7 GiB
        Metadata cache size (hard limit):              75.0 %  107.2 GiB
        Metadata cache size (current):                 10.0 %   10.7 GiB
        Dnode cache size (hard limit):                 10.0 %   10.7 GiB
        Dnode cache size (current):                    17.7 %    1.9 GiB

ARC hash breakdown:
        Elements max:                                             901.1k
        Elements current:                             100.0 %     901.1k
        Collisions:                                                17.7k
        Chain max:                                                     3
        Chains:                                                    11.9k

ARC misc:
        Deleted:                                                      46
        Mutex misses:                                                  0
        Eviction skips:                                             4.7k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 745.5 KiB
        L2 eligible MFU evictions:                     30.0 %  224.0 KiB
        L2 eligible MRU evictions:                     70.0 %  521.5 KiB
        L2 ineligible evictions:                                 6.0 KiB

ARC total accesses (hits + misses):                                 1.1G
        Cache hit ratio:                               99.9 %       1.1G
        Cache miss ratio:                               0.1 %     890.2k
        Actual hit ratio (MFU + MRU hits):             99.4 %       1.1G
        Data demand efficiency:                        97.2 %       7.1M
        Data prefetch efficiency:                       2.4 %     164.7k

Cache hits by cache type:
        Most frequently used (MFU):                    99.1 %       1.1G
        Most recently used (MRU):                       0.4 %       4.6M
        Most frequently used (MFU) ghost:               0.0 %          0
        Most recently used (MRU) ghost:                 0.0 %          0
        Anonymously used:                               0.5 %       5.9M

Cache hits by data type:
        Demand data:                                    0.6 %       6.9M
        Demand prefetch data:                         < 0.1 %       4.0k
        Demand metadata:                               98.6 %       1.1G
        Demand prefetch metadata:                       0.8 %       8.7M

Cache misses by data type:
        Demand data:                                   22.5 %     200.5k
        Demand prefetch data:                          18.1 %     160.7k
        Demand metadata:                               43.6 %     388.3k
        Demand prefetch metadata:                      15.8 %     140.7k

DMU prefetch efficiency:                                          665.7k
        Hit ratio:                                     23.7 %     157.7k
        Miss ratio:                                    76.3 %     508.0k

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                7562240
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           783276
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       15064508928
        mfu_ghost_data_esize                                           0
        mfu_ghost_metadata_esize                                       0
        mfu_ghost_size                                                 0
        mfu_metadata_esize                                    1394535424
        mfu_size                                             23371582464
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       27517395968
        mru_ghost_data_esize                                           0
        mru_ghost_metadata_esize                                       0
        mru_ghost_size                                                 0
        mru_metadata_esize                                     490209280
        mru_size                                             29772718592
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022081800-zfs_27f9f911a
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         1.8k
        Commit requests:                                             312
        Flushes to stable storage:                                   312
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:        3.1 MiB        252

 

[HoneyBadger]

ETA: I have not done the XMP read/write operation since rebooting the server this morning. I'm not sure how far back this summary goes but I can run it again after a cycle of XMP file read/writes.

The stats clear on every reboot along with ARC itself, so it might be challenging to draw conclusions until it's been filled and is having to make actual decisions about what to hold on to vs. discard.

Looking at the second one here:

And here is arc_summary immediately after writing and then reading about 500 xmp files into Lightroom. As well as some browsing of finished jobs in bridge, which is mostly folders of 1000-ish 5-15MB jpeg files.

Code:

root@Leviathan:~ # arc_summary

ARC size (current):                                    37.2 %   53.1 GiB
        Metadata cache size (hard limit):              75.0 %  107.2 GiB
        Metadata cache size (current):                 10.0 %   10.7 GiB

ARC total accesses (hits + misses):                                 1.1G
        Cache hit ratio:                               99.9 %       1.1G
        Cache miss ratio:                               0.1 %     890.2k
        Actual hit ratio (MFU + MRU hits):             99.4 %       1.1G
        Data demand efficiency:                        97.2 %       7.1M
        Data prefetch efficiency:                       2.4 %     164.7k

Cache hits by cache type:
        Most frequently used (MFU):                    99.1 %       1.1G
        Most recently used (MRU):                       0.4 %       4.6M
        Most frequently used (MFU) ghost:               0.0 %          0
        Most recently used (MRU) ghost:                 0.0 %          0
        Anonymously used:                               0.5 %       5.9M

Cache hits by data type:
        Demand data:                                    0.6 %       6.9M
        Demand prefetch data:                         < 0.1 %       4.0k
        Demand metadata:                               98.6 %       1.1G
        Demand prefetch metadata:                       0.8 %       8.7M

Cache misses by data type:
        Demand data:                                   22.5 %     200.5k
        Demand prefetch data:                          18.1 %     160.7k
        Demand metadata:                               43.6 %     388.3k
        Demand prefetch metadata:                      15.8 %     140.7k

DMU prefetch efficiency:                                          665.7k
        Hit ratio:                                     23.7 %     157.7k
        Miss ratio:                                    76.3 %     508.0k

Looking at the combination of hit ratio (99.4%) the majority of that being MFU (99.1%) and metadata dominating the hits by type (98.6%) seems to suggest that you do have a metadata-heavy workload - the misses are likely from the fact that you rebooted and cleared the ARC recently, so it doesn't know what it needs to get until it's been requested. The prefetcher is at least putting in decent work at 23.7% effectiveness.

Once ARC gets full and has to decide what to prioritize, we want to see it deciding to keep the metadata around since that should have a very high weight from the MFU (most frequently used) metric. If you see a bunch of misses in the "demand metadata" values then we might need to bump the minimum amount up.

For writes, the next time you're about to hit the big XMP workload, run an shell and look at the gstat values. Interactively, you can see them with:

gstat -dps

This will show you the drive activity in a table similar to below:

Code:

 L(q)  ops/s    r/s     kB   kBps   ms/r    w/s     kB   kBps   ms/w    d/s     kB   kBps   ms/d   %busy Name
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| nvd0
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da0
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da1
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da2
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da3
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da4
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da5
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da6
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da7
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| da8
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada0
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada1
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada2
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada3
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada4
    0      0      0      0      0    0.0      0      0      0    0.0      0      0      0    0.0    0.0| ada5

If you see big numbers in the %busy value, this could indicate that your disks are busy with queued operations (potentially due to lots of small I/O from the XMP files needing all of that seeking)

You can also dump it to a file with the -C option and redirection, as in

gstat -dpsC > /mnt/poolname/some_file.csv

Log for the entire period of the XMP operation and attach it here, but that file can grow quickly.

 

[JamesNisly]

Alright, I have some data. Linked here is a csv of the gstat data during an XMP read/write cycle of a sample of about 300 images, a Lightroom export of 100 images, all while browsing client folders in Bridge. All very, very common day to day activities around here.

I had the TrueNAS dashboard up during this and I did happen to notice that during the XMP read/writes, there was a significant portion of time with a single core of the server CPU at our near 100%. Much more than during the export. So whatever is going on doesn't seem to multithread very well. I don't know if that means anything, just something I happened to notice.

For completeness sake, here's arc_summary taken here towards the end of two days of heavy work.

Code:

ZFS Subsystem Report                            Thu Dec 01 15:52:09 2022
FreeBSD 13.1-RELEASE-p1                                    zpl version 5
Machine: Leviathan.local (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    60.7 %   86.7 GiB
        Target size (adaptive):                        60.9 %   87.0 GiB
        Min size (hard limit):                          3.1 %    4.5 GiB
        Max size (high water):                           31:1  142.9 GiB
        Most Frequently Used (MFU) cache size:         38.8 %   32.3 GiB
        Most Recently Used (MRU) cache size:           61.2 %   50.9 GiB
        Metadata cache size (hard limit):              75.0 %  107.2 GiB
        Metadata cache size (current):                  8.8 %    9.4 GiB
        Dnode cache size (hard limit):                 10.0 %   10.7 GiB
        Dnode cache size (current):                    17.3 %    1.9 GiB

ARC hash breakdown:
        Elements max:                                               3.7M
        Elements current:                              31.3 %       1.2M
        Collisions:                                               471.9k
        Chain max:                                                     4
        Chains:                                                    19.5k

ARC misc:
        Deleted:                                                    5.3M
        Mutex misses:                                              10.5k
        Eviction skips:                                             2.0M
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 123.0 GiB
        L2 eligible MFU evictions:                     25.7 %   31.6 GiB
        L2 eligible MRU evictions:                     74.3 %   91.4 GiB
        L2 ineligible evictions:                               669.3 GiB

ARC total accesses (hits + misses):                                 5.2G
        Cache hit ratio:                               99.9 %       5.2G
        Cache miss ratio:                               0.1 %       7.0M
        Actual hit ratio (MFU + MRU hits):             99.7 %       5.2G
        Data demand efficiency:                        97.6 %      44.2M
        Data prefetch efficiency:                       6.6 %       2.7M

Cache hits by cache type:
        Most frequently used (MFU):                    99.5 %       5.2G
        Most recently used (MRU):                       0.4 %      20.2M
        Most frequently used (MFU) ghost:             < 0.1 %     263.3k
        Most recently used (MRU) ghost:               < 0.1 %     410.8k
        Anonymously used:                               0.1 %       6.7M

Cache hits by data type:
        Demand data:                                    0.8 %      43.1M
        Demand prefetch data:                         < 0.1 %     179.2k
        Demand metadata:                               98.7 %       5.2G
        Demand prefetch metadata:                       0.5 %      24.7M

Cache misses by data type:
        Demand data:                                   15.2 %       1.1M
        Demand prefetch data:                          36.3 %       2.5M
        Demand metadata:                                7.7 %     534.1k
        Demand prefetch metadata:                      40.8 %       2.8M

DMU prefetch efficiency:                                            7.7M
        Hit ratio:                                     33.5 %       2.6M
        Miss ratio:                                    66.5 %       5.1M

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                6212608
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           783276
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       26550305280
        mfu_ghost_data_esize                                           0
        mfu_ghost_metadata_esize                             13093597184
        mfu_ghost_size                                       13093597184
        mfu_metadata_esize                                    1302961152
        mfu_size                                             34640840192
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       51368858112
        mru_ghost_data_esize                                  2449567744
        mru_ghost_metadata_esize                              1349032960
        mru_ghost_size                                        3798600704
        mru_metadata_esize                                     110140416
        mru_size                                             54649847808
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022081800-zfs_27f9f911a
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

 

[Etorix]

The single threaded part is SMB being SMB, so here you're throttled by the old Xeons.

 

[JamesNisly]

The single threaded part is SMB being SMB, so here you're throttled by the old Xeons.

I've been seriously considering putting this old X8DTN+ and these Xeons to rest but haven't gone through the effort to figure out what to replace them with. Maybe it's time to bump that up the priority list.