Out of space ZFS volume housing empty esxi datastore

[yanky83]

Hello,

I have a somewhat odd issue:

- esxi 5.5, VMFS5
- FreeBSD 10.3
- ctl iscsi
- Full SSD RAIDz2 volume exposed as datastore to esxi

When I copy VMs to the datastore, and subsequently move them off, the space on the ZFS volume does not get freed. I am aware, that ZFS doesnt know, whats free and whats not - that is not the issue.

However: even though esxi reports the datastore to have free space of more than 1TB, it says there is no more free space, and just halts any VM on it. The ZFS volume will report to be completely full.

Currently trying to experiment with unmap off in ctl settings.

Has anyone ever encountered similar behavior?

Any help would be much appreciated - pretty stuck.

Thanks,
Dennis.

 

[darkwarrior]

Hi there,

if you want the diskspace to get freed, when you delete data on your zVol you need to use UNMAP. Without it the blocks will never get freed ...

 

[yanky83]

I fully agree, technically.

However, esxi should know, which space is free, even though ZFS thinks that space is referenced. This is however not the case, even though esxi reports the datastore to have more than available space (in datastore properties), it still throws errors the datastore to be out of space in relation to the VMs running on it - which is confusing.

I assume somewhere ZFS, SSD volumes, and esxi dont play together as they should.

Ideas?

 

[darkwarrior]

They work very well together. Many people on the forums are running ESXi datastores on FreeNAS zVOLS.
The thing is that FreeNAS is COW filesystem. So you need to have freespace on your zVOL to be able accept any incoming write IOPs via iSCSI.
In your case they are getting refused because all your space is referenced...

Please check with the command below, if all the VAAI primitives are enabled:
esxcli storage core device vaai status get -d

And check the links below for the command line in ESXi to manuallt initiate a UNMAP of the diskspace.
Direct Guest OS UNMAP in vSphere 6.0
Using esxcli in vSphere 5.5 and 6.0 to reclaim VMFS deleted blocks on thin-provisioned LUNs (2057513)
How to Reclaim Free Block Space from a Lun with VMware vSphere 5.5 and Netapp Cluster Mode

 

[yanky83]

Still the odd thing remains, why would esxi report the datastore as almost empty, the zvol be full, and esxi produce error thinking the datastore is full - its just bizarre.

Seems supported:

Code:

~ # esxcli storage core device vaai status get
t10.FreeBSD_san3__tank1__vol_vmware01
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

t10.FreeBSD_san3__tank1__vol_vmware02
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

t10.FreeBSD_san3__tank1__vol_vmware03
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

t10.FreeBSD_san3__tank1__vol_vmware04
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

naa.600508b1001cf21e54eb7809a3ddc8bd
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

t10.FreeBSD_san5__tank1__vol_vmware01
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

naa.600508b1001cf8267d003ae3aa752b25
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

t10.FreeBSD_san5__tank2__vol_vmware01
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

t10.FreeBSD_san4__tank1__vol_vmware01
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

t10.FreeBSD_san4__tank1__vol_vmware02
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

 

[darkwarrior]

No it's not bizarre ... This is what happens when you fill a COW filesystem ... :p

Good thing is that UNMAP is supported.
You really need to run the UNMAP command if you want anything to happen ;)

 

[yanky83]

I still dont think, you are realizing the problem:

say I have a datastore, I move a VM to it, I move the VM off - I can repeat this until the underlying zvol is full, and VMWare will fail to allocate space on a datastore, it assumes to be empty

 

[darkwarrior]

Right now it seems that the UNMAP (that should be initiated by the ESXi host) is not working as intended...
This is why I was suggesting you to run it manually ...
Are you running a sparse zVOL ?

Did you do all the RTFM on the subject ? For now it seems not ...

 

[yanky83]

I sure read, what I could

This particular behavior seems to only affect the SSD volume, any HDD volume is fine, and UNMAPs as expected (by default I set unmap on in ctl config)

 

[yanky83]

I tested a bit further:

- I have another server, also running FreeBSD 10.3 with ctld (slight lower patch level), here everything works as expected (i.e. I delete a file from the datastore, and the space gets free in the zvol shortly after)

What am I missing?

One difference, there is, is vfs.zfs.vol.mode being 1 on the working system, and 2 on the system with issues? Any insights on the setting?

 

[bigphil]

First off...if you want unmap support, don't turn it off in ctl or zfs options. Make sure you have unmap turned back on for the system that is not working. A reboot may be needed as some options are only updated at pool import. The default vfs.zfs.vol.mode=2 in FreeNAS and should be used for ESXi zvol datastores. https://www.freebsd.org/cgi/man.cgi?query=zfs&sektion=&n=1
Something in your setup sounds strange though. Unmap support in ESXi 5.5 should be triggered manually via https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2057513 so I'm not sure how its automatically unmapping as you say. Guest OS unmap support was only added in ESXi 6.0 I believe and requires vm compatibility level 11 and EnableBlockDelete option set on the host and a supported guest OS and actually supports unmap. Also...like darkwarrior asked...are you setup as a sparse volume on both boxes? It almost sounds like one is and one isn't.

 

[yanky83]

Thanks for the reply.

The setups are very similar, both 10.3, one is mirror dev, one is raidz2, esxi 5.5 and hyperv 2012 r2 as clients

Very good question, how it is automatically unmapping, but it is - if I move a VM on and off the volume, the zvol occupancy goes up and down again, without manually doing unmap. The only thing, that is not behaving is the pure SSD raidz2 volume.

Could it be different disk type related?

Happy to send all kinds of configuration details

 

[bigphil]

When you remove files from the volume, which client are you using? Hyper-v or ESXi? Or does it unmap properly with either client? Post the output of "zfs get all volPath" where volPath is the path of your volume. Post the output for both...the one that works and the one that doesn't.

 

[yanky83]

Some testing seems to indicate it only automatically unmaps using Hyper-V (could have sworn I had seen it unmap using esxi before)

Would this indicate a problem with esxi? The weird error esxi halting VMs cause it thinks the datastore is full, even though it itself reports there to be plenty space.

Below zfs options, anything looking funny to you?

Code:

[root@san4 ~]# zfs get all tank1/vol_vmware02
NAME				PROPERTY			  VALUE				  SOURCE
tank1/vol_vmware02  type				  volume				 -
tank1/vol_vmware02  creation			  Mon Dec  5 22:01 2016  -
tank1/vol_vmware02  used				  5.67T				  -
tank1/vol_vmware02  available			 5.47T				  -
tank1/vol_vmware02  referenced			238G				   -
tank1/vol_vmware02  compressratio		 1.21x				  -
tank1/vol_vmware02  reservation		   none				   default
tank1/vol_vmware02  volsize			   5.50T				  local
tank1/vol_vmware02  volblocksize		  8K					 -
tank1/vol_vmware02  checksum			  on					 default
tank1/vol_vmware02  compression		   lz4					inherited from										  tank1
tank1/vol_vmware02  readonly			  off					default
tank1/vol_vmware02  copies				1					  default
tank1/vol_vmware02  refreservation		5.67T				  local
tank1/vol_vmware02  primarycache		  all					default
tank1/vol_vmware02  secondarycache		all					default
tank1/vol_vmware02  usedbysnapshots	   0					  -
tank1/vol_vmware02  usedbydataset		 238G				   -
tank1/vol_vmware02  usedbychildren		0					  -
tank1/vol_vmware02  usedbyrefreservation  5.44T				  -
tank1/vol_vmware02  logbias			   latency				default
tank1/vol_vmware02  dedup				 off					default
tank1/vol_vmware02  mlslabel									 -
tank1/vol_vmware02  sync				  always				 local
tank1/vol_vmware02  refcompressratio	  1.21x				  -
tank1/vol_vmware02  written			   238G				   -
tank1/vol_vmware02  logicalused		   265G				   -
tank1/vol_vmware02  logicalreferenced	 265G				   -
tank1/vol_vmware02  volmode			   default				default
tank1/vol_vmware02  snapshot_limit		none				   default
tank1/vol_vmware02  snapshot_count		none				   default
tank1/vol_vmware02  redundant_metadata	all					default

Code:

[root@san3 ~]# zfs get all tank1/vol_vmware04
NAME				PROPERTY			  VALUE				  SOURCE
tank1/vol_vmware04  type				  volume				 -
tank1/vol_vmware04  creation			  Sat Oct  1 20:49 2016  -
tank1/vol_vmware04  used				  1.03T				  -
tank1/vol_vmware04  available			 1.38T				  -
tank1/vol_vmware04  referenced			189G				   -
tank1/vol_vmware04  compressratio		 1.36x				  -
tank1/vol_vmware04  reservation		   none				   default
tank1/vol_vmware04  volsize			   1T					 local
tank1/vol_vmware04  volblocksize		  8K					 -
tank1/vol_vmware04  checksum			  on					 default
tank1/vol_vmware04  compression		   lz4					inherited from tank1
tank1/vol_vmware04  readonly			  off					default
tank1/vol_vmware04  copies				1					  default
tank1/vol_vmware04  refreservation		1.03T				  local
tank1/vol_vmware04  primarycache		  all					default
tank1/vol_vmware04  secondarycache		all					default
tank1/vol_vmware04  usedbysnapshots	   0					  -
tank1/vol_vmware04  usedbydataset		 189G				   -
tank1/vol_vmware04  usedbychildren		0					  -
tank1/vol_vmware04  usedbyrefreservation  868G				   -
tank1/vol_vmware04  logbias			   latency				default
tank1/vol_vmware04  dedup				 off					default
tank1/vol_vmware04  mlslabel									 -
tank1/vol_vmware04  sync				  always				 local
tank1/vol_vmware04  refcompressratio	  1.36x				  -
tank1/vol_vmware04  written			   189G				   -
tank1/vol_vmware04  logicalused		   255G				   -
tank1/vol_vmware04  logicalreferenced	 255G				   -
tank1/vol_vmware04  volmode			   default				default
tank1/vol_vmware04  snapshot_limit		none				   default
tank1/vol_vmware04  snapshot_count		none				   default
tank1/vol_vmware04  redundant_metadata	all					default

 

[darkwarrior]

And what about the volume type ?
Is it a sparse volume or not ?

 

[yanky83]

I actually am not exactly sure, what configuration details you are referring to?

Could you clarify?

 

[darkwarrior]

And what about the volume type ?
Is it a sparse volume or not ?

Right now it's probably the single most important setting:
When you created the zVOL did you tick the checkbox (like in the screenshot) ?

http://www.virten.net/wp-content/uploads/2015/01/freenas-sparse-zvol.png

UNMAP will never work if you don't have a sparse volume...

Edit: Could not show the link to the official documentation. doc.freenas.org seems down

 

[yanky83]

As said, its plain FreeBSD, no FreeNAS.

Which actual setting would this get translated into?

I assume, the difference being either a ZFS dataset or a volume?

All are volumes, and esxi reports "Delete Status" to be supported

 

[darkwarrior]

So that's explains a lot ... :p
No, the difference I'm referring to here is bascially thinprovisioning.
You need your zvol to be thinprovisioned for UNMAP to work properly on BSD.

When you created your zVOL, which command did you use ?
1. zfs create -s -V 5t tank/vol
or
2. zfs create -V 5t tank/vol

-s is the option for sparse volume

 

[yanky83]

Without -s.

But I doubt, that this would be a deal breaker to spare reclamation:

"
Though not recommended, a "sparse volume" (also known as "thin pro-
visioning") can be created by specifying the -s option to the "zfs
create -V" command, or by changing the reservation after the volume
has been created. A "sparse volume" is a volume where the reserva-
tion is less then the volume size. Consequently, writes to a sparse
volume can fail with ENOSPC when the pool is low on space. For a
sparse volume, changes to volsize are not reflected in the reserva-
tion.
"

If I understand correctly, all it (without the -s) does, pre-reserve the full volume size, but certainly not reference/use the entire space.

Code:

[root@san5 ~]# zfs list tank1/vol_vmware01
NAME				 USED  AVAIL  REFER  MOUNTPOINT
tank1/vol_vmware01  2.06T  1.80T   325G  -

Thoughts?

Still interesting, didnt know this setting before.