zvol on special vdev?

[jasonsansone]

I have been attempting to push certain zvol data on to a metadata / special vdev without any success. The root dataset has record size set at 1MB. "Special Small Blocks" is set at 128kb and below for the pool. Small block data in datasets works as expected. I have created new zvols with 16kb zvolblocksize but no matter what I do, the data doesn't populate on the special vdev. Has anyone been able to get data inside a zvol, or an entire zvol, in to a special vdev?

 

[Ericloewe]

I don't think that's possible. The basic point of special vdevs is to handle small amounts of performance-sensitive data - i.e. metadata.

I'm sure it would be a fairly easy addition, but what would you gain at that point? Sounds like a separate pool would be better in 90% of scenarios.

 

[jasonsansone]

Understand. I am taking a somewhat "round peg square hole" approach to the problem. The motivation is that I have a mirror of 1.6TB Intel P3605's that metadata won't even come close to filing. With a 1MB record size, 80TB of data doesn't consume a tenth of the capacity on the metadata vdev. Moving my VM's to the same pool, but forcing them onto the NVMe vdev, allows me to consolidate. At present I am doing what you suggest - I have two mirrors (4x) of Samsung 860 Pro's for my VM pool. Also... because playing with and breaking things is the purpose of a home lab...

 

[Ericloewe]

If you're playing fast and loose and can rebuild without too much trouble, I'd suggest partitioning the SSDs to serve the special vdev plus a separate pool.

 

[ornias]

a zvol is a seperate "entity", if you need dataset settings specifically to a zvol those need to be set on the zvol itself, not the parent dataset.

 

[jasonsansone]

a zvol is a seperate "entity", if you need dataset settings specifically to a zvol those need to be set on the zvol itself, not the parent dataset.

@ornias can you please elaborate? A zvol doesn't have a recordsize, but the volblocksize is below the cut off for special_small_blocks for the dataset containing the zvol. If I attempt to set special_small_blocks directly on the zvol I am told "special_small_blocks' does not apply to datasets of this type".

I assume that the answer is what @Ericloewe provided - zvols are excluded from special_small_blocks?

 

[Ericloewe]

My understanding is that the special allocation class is exclusive to metadata (which may include embedded data in block pointers, to be clear, but not normal data blocks).

 

[jasonsansone]

My understanding is that the special allocation class is exclusive to metadata (which may include embedded data in block pointers, to be clear, but not normal data blocks).

Metadata and optionally small blocks.

Fusion Pools - Allows creating pools with all-flash vdevs that store metadata and small-block IO. This provides flash-like performance on all metadata operations, but with the cost effectiveness of spinning disk for storage blocks.

TrueNAS 12.0-RC1 Release Notes

special_small_blocks = size This value represents the threshold block size for including small file blocks into the special allocation class. Blocks smaller than or equal to this value will be assigned to the special allocation class while greater blocks will be assigned to the regular class. Valid values are zero or a power of two from 512B up to 128K. The default size is 0 which means no small file blocks will be allocated in the special class.

Manpage of ZFS

 

[jenksdrummer]

You can use file based extents for iSCSI, which then dataset rules apply. Bonus, I've seen much better compression out of file based extents. Un-bonus, I have seen between different workloads, where file based extents are slower, but I have also seen where they are faster; during the same course of tests and attempting to give all things equal.

Basically, that might be an avenue to test, if iSCSI is the reason for ZVOL use.

 

[jenksdrummer]

Metadata and optionally small blocks.

TrueNAS 12.0-RC1 Release Notes

Manpage of ZFS

This is what I take of it too. Setting the special_small_blocks value starts writing data to the special vdev outside of metadata (as does turning on deduplication; the tables go to the special VDEV), and once that fills up, then those writes happen back at the data vdev...IE...poor performance takes over.