Are reads Single-core but writes Multi-core?

[HoneyBadger]

- Metadata for the entire pool will be cashed in L2ARC (except the dataset called DataSetThatShouldBeCached)
- PoolName/DataSetThatShouldBeCached would use the L2ARC disk for data caching and not metadata(i.e. this dataset's metadata would only live on spinners)?

Metadata for all datasets in the pool would be cached in L2ARC.

PoolName/DataSetThatShouldBeCached would leverage L2ARC for both data and metadata, as implied by the "all" value for secondarycache.

Basically, metadata is small enough that it's almost always worthwhile allowing it to live in L2ARC. You would just selectively enable data caching where you know it would benefit, and avoid it elsewhere - eg, don't cache your Plex video directory, as it's unlikely to be useful putting your most-recently-viewed 4K videos onto an SSD after a single viewing, but do cache your shared office files as they're both small and frequently hit and will benefit from the quick random-access speed of an SSD.

In all cases, the permanent copy of metadata lives on the pool vdevs; L2ARC only holds a copy of data that exists elsewhere in a permanent state, be that the actual records or metadata referring to them.

 

[souporman]

Metadata for all datasets in the pool would be cached in L2ARC.

PoolName/DataSetThatShouldBeCached would leverage L2ARC for both data and metadata, as implied by the "all" value for secondarycache.

Basically, metadata is small enough that it's almost always worthwhile allowing it to live in L2ARC. You would just selectively enable data caching where you know it would benefit, and avoid it elsewhere - eg, don't cache your Plex video directory, as it's unlikely to be useful putting your most-recently-viewed 4K videos onto an SSD after a single viewing, but do cache your shared office files as they're both small and frequently hit and will benefit from the quick random-access speed of an SSD.

In all cases, the permanent copy of metadata lives on the pool vdevs; L2ARC only holds a copy of data that exists elsewhere in a permanent state, be that the actual records or metadata referring to them.

Thanks, that makes sense. Assuming one weren't taking advantage of small block files on a special VDEV, and it was only used for metadata, do you feel like the added risk of adding that VDEV to a pool outweighs the usefulness of it vs a metadata L2ARC? Or, is a special VDEV for metadata significantly better at handling metadata than the L2ARC, assuming the disks used would be the same?

 

[HoneyBadger]

Thanks, that makes sense. Assuming one weren't taking advantage of small block files on a special VDEV, and it was only used for metadata, do you feel like the added risk of adding that VDEV to a pool outweighs the usefulness of it vs a metadata L2ARC? Or, is a special VDEV for metadata significantly better at handling metadata than the L2ARC, assuming the disks used would be the same?

For most users, the risk of the special vdev doesn't yield enough benefits - especially with the restrictions on vdev removal making it permanent if you add it to a pool build on RAIDZ vdevs. Using the tunables to retain metadata is primary ARC is an option if you have sufficient RAM, but if not then the meta-only L2ARC is the next best option.

Checking the amount of ARC used by meta/data and the hitrates in the arc_summary script will give you guidance if it's worth weighting metadata more heavily into RAM as well.

Where a special vdev beats out an L2ARC set for "metadata only" would be in a heavy metadata write workload, like touching lots of small files which causes the "last accessed" or "last updated" time to be changed.

 

[souporman]

For most users, the risk of the special vdev doesn't yield enough benefits - especially with the restrictions on vdev removal making it permanent if you add it to a pool build on RAIDZ vdevs. Using the tunables to retain metadata is primary ARC is an option if you have sufficient RAM, but if not then the meta-only L2ARC is the next best option.

Checking the amount of ARC used by meta/data and the hitrates in the arc_summary script will give you guidance if it's worth weighting metadata more heavily into RAM as well.

Where a special vdev beats out an L2ARC set for "metadata only" would be in a heavy metadata write workload, like touching lots of small files which causes the "last accessed" or "last updated" time to be changed.

Thanks. Very clear and concise as always.