Tuning recordsize for hosting VM over NFS + backup storage over NFS

[Forza]

Hi!

I have been searching around the forum and the internet for examples of tuning recordsize for my use cases.

1) Exporting a pair of SSDs over NFS as VM storage for XCP-ng (Xenserver fork).
2) Exporting mirrored HDD storage as backups for mixed set of data over SMB where Windows clients dump data that should be stored for a long time (many years). I also intend to keep snapshops throughout all the years.

My initial thought is that for 1) I could reduce the recordsize somewhat to improve random access as all VMs use 4K reads/writes. The VMs are a mix of Linux and Windows VMs but none are extreamly high I/O.

For the backup storage I am a little worried about fragmentation so a higher record size might be beneficial there. I _think_ that files smaller than recordsize will use a smaller record, is that right?

 

[QonoS]

That is an interesting question and to my own surprise the answers seems to be:
[1] "All files are stored either as a single block of varying sizes (up to the recordsize) or using multiple recordsize blocks."
[2] "By default, the recordsize on ZFS is 128KiB, which means it will dynamically allocate blocks of any size from 512B to 128KiB depending on the size of file being written. "

Regarding 1) VMs usually do only a small percentage of synchronous writes. That means most of writes (which are async) will be cached in RAM first and then written to pool. So a small record size is probably not important fo writes. How about read? For random read I/O 4K-records should be faster but in reality it could also be not noticeable.
You could do a test to find out easily, as recordsize can be modified. Just rewrite test-data to the pool after setting a new resordsize. ;-)

zfs set recordsize=4K pool/dataset

 

[HoneyBadger]

Recordsize is a maximum, the minimum is derived from the ashift value, which should give you 4K.

For the live VMs, try a recordsize of 32K. It's large enough to benefit from compression (which should be on) and greater transfer speeds, but not so large that you pay too heavy of a read-modify-write penalty for partial record updates.

For the backup/general file storage, the 128K default is usually fine. If you know that a large percentage of files are several MB or larger (movies, photos) then you could manually set it to an even larger value to further reduce metadata overhead.

 

[Forza]

Valuable suggestions from both of you. I'll try a few different scenarios and see what gives. Perhaps it isn't worth it in the end but I'm too curious. At least none of you said oh that's going to kill your drives and eat the dog :)