Incorrect free space freenas-vmware

[remonv76]

Hi there,

I have a question. I have a small iscsi zvol using compression. The zvol is exported as a datastore to my VMWare infra. The question i have is about the free space.

The compression ratio freenas accomplishes is 2.11x, which is great. But the free space in VMWare is not the same as in freenas.

For instance: I have a zpool of 4TB and a zvol which is 80% of 4TB = around 3TB.
My data is 2.4TB and with a compression ratio of, say 2.00x, you get 1.2TB of used space.
But VMWare only sees 2.4TB and sais i have 600GB of free space. But actually it should be 1.8TB. (3TB-1.2TB)
So what will happen if my data exceeds the 3TB, which is actually 1.5TB with compression?

I know VMWare will alert me every day that my datastore is full, which isn't actually right.

 

[Artion]

Hi,
I think that is right what you see. VMWare does see how much data you have stored in the volume not how much space does it use in FreeNAS zvol. VMWare is not aware of the compression as it is done by FreeNAS on the block device. You can expand the zvol so VMWare sees more space if you want.

 

[remonv76]

Yep that's what I figured out also. So it's better to use NFS over iSCSI if you're using compression/deduplication. Only then you will see the actual freespace. But because I'm on iSCSI and have no option to convert to NFS (backup/restore takes too long) I'm wondering what will happen if I overcommit the 3TB, because of the 2x compression i get. I know I will get alerts every night that my datastore is full, which isn't correct. But what I don't hope is that VMWare will pause every VM, because it thinks it has no space left.
My worries are that this is gonna happen.

So I was talking to my collegue about this and he made the remark to create a second zvol of 3T. But Freenas does not allow you to do this. Also Freenas does not allow you to set the zvol to 6T on a 4TB zpool. So how on earth can I use the extra free space, created by the compression?

 

[mav@]

For the first, since it is not recommended to use more then 50% of pool space when doing VM storage to not grow space fragmentation to much, you may be fine setting zvol size close to pool size and make compression hold you within 50% of pool usage. If you are sure that your data are compressible, you may go even further -- make zvol sparse, in which case you may set almost any size for it, independently of your pool size. In such case FreeNAS won't complain until real pool usage get dangerous. But in both cases, and especially in second, all responsibility for not overflowing pool if something unexpected happen during the night is on your side. FreeNAS will send warnings to VMWare servers about pool usage reaching configured thresholds and errors when there is no more space, but it is your responsibility to handle that in time.

 

[mav@]

But before going into overcommit, make sure your version of VMware supports UNMAP and you know how to use it.

 

[remonv76]

Thanks Mav for your response.
UNMAP is also a feature of VAAI. Is this not implemented in the VAAI module for Freenas? I could run it on the esxi host, but I think that does not work. It's not an issue of reclaiming blocks, but how the LUN information is presented to VMWare. It's fixed and unlike NFS can not be dynamically presented. (free/used space)
So VMware calculates what is used within the datastore.

The only solution to use the extra "free" space is by creating a second zvol of 3T or creating a zvol bigger than the zpool. I get 2.12x compression, so forcing the size to 3.5T should be no problem and i should be ok. I will think this over, because I also use snapshots and probably have to restart the iscsi service when resizing the zvol. I will have to double check if it's safe enough.

Thank you all for the replies. It was a big help.

 

[mav@]

FreeNAS supports UNMAP in all possible ways. It even reports real pool usage to VMware via respective SCSI log page. VMware knows about that log page and reads values from it to log when FreeNAS reports crossing free space threshold warning, but for some reason VMware does not report it in any UI in easy way.

I mentioned UNMAP support, since VMware 5.x was not running UNMAP automatically, requiring explicit administrative attention. In 6.x and especially 6.5 it should work by itself, though with lower priority.

Having snapshots on FreeNAS does not allow to free space immediately. That is another cause to be careful with overcommit, even if you sure your data are compressible. Every time data change, they will take another new part of space, while old space will be still held for the snapshot hold time.