Does Windows Server 2019 push UNMAPs to storage on a continuous basis?

[richardm1]

So here's an odd one for you guys. I'm playing around with a Server 2019 VM running on ESXi 6.7. The storage is provided via iSCSI in the usual AIO manner. The zvol is sparse/thin and the VM itself is a thin vmdk. This is a new/pristine VM on a new/pristine VMFS 6 datastore residing on a new/pristine zvol in a new/pristine zpool. No other processes or VMs are present.

When I run CrystalDiskMark with a zpool iostat looping the background, I see capacity alloc suddenly increase when the benchmark run begins. No surprise there. What amazes me is watching capacity alloc slowly drift back down to its original number after the benchmark is finished and CrystalDiskMark cleans up after itself. The only possible explanation I can think of is that Server 2019 is pushing UNMAP/TRIM commands down to storage in the background after a file has been deleted.

Thoughts?

 

[dlavigne]

Did you find the answer to your question?

 

[HoneyBadger]

In your case, it's not Windows Server, but rather ESXi that's sending the UNMAPs. The guest OS notifies the disk that a file has been deleted, and VMFS6 can pass those deletes down to the array.

In VMFS6, there is also a default level of background space reclamation that is done. You can change the space reclamation policy through either the vSphere client or esxcli.

See the VMware KBs below for details:

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc/GUID-BC1A172C-E649-4812-B8B2-A9E45AC97051.html

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc/GUID-5E1396BE-6EA8-4A6B-A458-FC9718E2C55B.html

 

[richardm1]

The guest OS notifies the disk that a file has been deleted

If ESXi is pushing UNMAPs to storage then the guest OS must be telling ESXi which blocks have been freed. How would the guest OS "notify the disk" by any mechanism other than TRIM or UNMAP?

 

[HoneyBadger]

If ESXi is pushing UNMAPs to storage then the guest OS must be telling ESXi which blocks have been freed. How would the guest OS "notify the disk" by any mechanism other than TRIM or UNMAP?

I was speaking specifically to your situation where you're running a Windows VM on top of a VMFS datastore; the commands are going Windows->VMFS->ZFS.

A modern OS will send TRIM or UNMAP commands, but unless you're using a VMware version that will automatically pass them on to the storage (VMFS6) or manually run the commands to reclaim it, they won't make it through.

If you present an iSCSI LUN as a raw device, then the guest OS can do the UNMAPs directly and you'll see the space allocation reflect that.