How helpful is a special metadata device in practice?

[Johnny Fartpants]

If a system has a cache hit ratio of 99% and 97% of that is demand metadata then how helpful (if at all) would a special metadata device be in practice?

Should people explore their ARC hit ratio before using a special metadata device or am I missing something?

 

[Etorix]

Not much helpful for metadata, but a spacial vdev can also take small files (up to a user-defined limit), which could still be useful if the bulk storage is on a wide raidz# vdev with large recordsize (i.e. terribly inefficient with small files).

 

[NugentS]

I use one for small files. I have a dataset full (90GB) of small 1-4MB files that I need quick access to. I use a special vdev to put almost all of them on the SSD's which is appreciably quicker

 

[sretalla]

If a system has a cache hit ratio of 99% and 97% of that is demand metadata then how helpful (if at all) would a special metadata device be in practice?

Since metadata must be stored somewhere in the pool and with writing of large numbers of small files, this means huge loads of IOPS during writes in addition to high IOPS needed to write the file contents.

If you can have metadata (always small blocks) and smaller data blocks written to a device that handles many more IOPS than your data VDEVs, then that can certainly help.

Should people explore their ARC hit ratio before using a special metadata device or am I missing something?

For sure looking into caching of metadata and making sure that's optimal (including thinking about a secondarycache=metadata = a metadata-only L2ARC if you have the system for it) can mean that reads are much better for things like large rsync jobs or large file shares with millions of files.

Edit: Look here for some tips on that:

ZFS "ARC" doesn't seem that smart...

Without yet resorting to adding an L2ARC, is there a "tuneable" that I can test which instructs the ARC to prioritize metadata? I upgraded from 16GB to 32GB ECC RAM. Yet there is zero change in this behavior. This keeps happening: I run regular rsync tasks from a few local clients, which is...

www.truenas.com

ZFS "ARC" is getting smarter with version 2.2+ :smiley_face_emoji:

This is a continuation and update of this thread: https://www.truenas.com/community/threads/zfs-arc-doesnt-seem-that-smart.100423/ Looks like OpenZFS 2.2+ introduces some cleaned up and rewritten code¹ to more intelligently and gracefully handle ARC data / metadata eviction from RAM. It's such...

www.truenas.com