ZFS Fragmentation question

[zeroluck]

This may be a simple answer, but I am genuinely curious and would like to know the ramifications of something I am planning to do. Maybe it is an easy answer as well!

Say I create a volume, let's say for the pointless sake of argument it is mirrored striped vdevs of 10x2x3TB (20 devices, 10 mirrored sets). I then share this out with iSCSI to a hypervisor and fill up 50% of my capacity (15TB) with vMotion (vastly sequential writes). Soon after this I decide to remove (either delete or vmotion to another storage device) one of the virtual volumes (vmware vmdk) that I am storing on there, let's say it is 3TB.

Would this cause less fragmentation for my ZFS filesystem than normal copy on write changes to the data? Will ZFS see this as free space and write sequential stripes into this space after it is vacated?

Part of the reason I ask is I have noticed that when I evacuate VMFS volumes which live on FreeNAS, VMware reports the new free space but the RRD graphs in FreeNAS reporting section under partitions don't seem to ever go back down.

Thanks for any insight anyone has!

 

[depasseg]

Are you running freenas 9.3.something?

It sounds like you might have thin provisioned vmdk's, which could explain why freenas sees it as used.

Or it could be that VAAI isn't being used and freeing up the iscsi zvol.
https://forums.freenas.org/index.php?threads/iscsi-nfs-win-for-me.40719/#post-260866

 

[diehard]

You must be using a file extent as i don't believe a device extent will show up in that graph, it simply "carves" it out of the free space.

Before you get too worried about it.. use device extents :)

 

[jgreco]

You must be using a file extent as i don't believe a device extent will show up in that graph, it simply "carves" it out of the free space.

Before you get too worried about it.. use device extents :)

That's my impression too.

 

[zeroluck]

Are you running freenas 9.3.something?

It sounds like you might have thin provisioned vmdk's, which could explain why freenas sees it as used.

Or it could be that VAAI isn't being used and freeing up the iscsi zvol.
https://forums.freenas.org/index.php?threads/iscsi-nfs-win-for-me.40719/#post-260866

Yes, I am running FreeNAS 9.3.x, the latest stable version. I am completely removing the VMDK from the array. Although the vmdk is thin provisioned, completely removing the vmdk from the datastore makes the fact that it is thin provisioned irrelevant. Did you maybe mean I am using a thin provisioned VMFS volume to store my vmdk files? That would be accurate. I will check into seeing if VAAI is working.

You must be using a file extent as i don't believe a device extent will show up in that graph, it simply "carves" it out of the free space.

Before you get too worried about it.. use device extents :)

Yes, I am using a file extent. The reason I am using file extents is the manual says:

In theory, a zvol and a file extent should have identical performance. In practice, a file extent outperforms in reads/writes but this is only noticeable at 10 GB Ethernet speeds or higher. For high performance, file extents are recommended at this time. Future changes to FreeBSD's zvol code will increase its performance.

Is there a functional difference between a device extent and a file extent on FreeNAS? Are file extents incapable of reclaiming free space or something? The arrays I am using are full of a ton of data (30+ TB live data) so evacuating them to reconfigure my pools and extents is a pain. Plus I would genuinely like to understand the difference.

To add to the confusion, this http://blogbt.net/index.php/2014/07/vcap-dca-setting-external-storage-iscsi-on-freenas/ guide says

simply put, you should always choose file based extents since they are more flexible and offer identical performance to the device extents

The next place I can find references the FreeNAS docs saying exactly the opposite! http://www.vladan.fr/how-to-configure-freenas-8-for-iscsi-and-connect-to-esxi/

A device extent allows a raw partition (volume) to be exported via iSCSI. The advantage of device extent is that they are faster than file extents. The disadvantage is that the entire volume is exported. If you only want to share a portion of a volume using iSCSI, you will need to create a file extent instead.

A file extent allows you to export a portion of a volume. When creating a file extent, you specify a file name that iSCSI clients will have access to (similar in concept to a mount point) and the maximum size of that file (storage area). The advantage of file extents is that you can create multiple exports per volume. The disadvantage is that they are slower than device extents.

I just find it funny that the documentation itself has been flip-flopping polar opposite answers. My main concern here is that deleted space is getting reused.

 

[diehard]

Ensure VAAI is working, and use UNMAP http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2057513

 

[jgreco]

Yes, I am running FreeNAS 9.3.x, the latest stable version. [...snip...]
Yes, I am using a file extent. The reason I am using file extents is the manual says:

You can't assume advice from the 9.2.1 manual is still correct for 9.3; in fact, kernel iSCSI was introduced into 9.3 and totally changed the ballgame. The 9.2.1 material you quoted is no longer considered good advice under 9.3.

 

[zeroluck]

You can't assume advice from the 9.2.1 manual is still correct for 9.3; in fact, kernel iSCSI was introduced into 9.3 and totally changed the ballgame. The 9.2.1 material you quoted is no longer considered good advice under 9.3.

Fair enough. Looking at the 9.3 documentation, reference to that one way or another is gone and the documentation says to create either a device OR file extent and states it rather indifferently. Still, according to this post https://forums.freenas.org/index.ph...vice-extent-vs-file-extents.17668/#post-95096 by a core FreeNAS team member who conversed with the FreeBSD developer responsible for this, since we're still on FreeBSD 9 we might assume that file extents are still faster as of this writing? Either way, I think the functional difference in the context of this problem is that with file extents I get to see a partition graph.

Ensure VAAI is working, and use UNMAP http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2057513

I think this points me in the right direction. I'm going to try it and see what the graph and vmware's usage statistics report back.

Edit: According to this article http://www.virten.net/2015/01/working-with-freenas-9-3-block-vaai-primitives/ dead space reclamation (UNMAP) only works if your iSCSI extent is a ZVOL created as sparse volume.

 

[jgreco]

You can assume whatever you want. I already said that there was a massive change in 9.3 and you are welcome to ignore that as hard as you like in favor of a two year old discussion about how things used to work.

 

[zeroluck]

You can assume whatever you want. I already said that there was a massive change in 9.3 and you are welcome to ignore that as hard as you like in favor of a two year old discussion about how things used to work.

Well you know what they say about assuming... How do the changes in 9.3 affect this conversation? Are device extents now better than file extents in 9.3 for any other reason than thin provisioned volume unmap support on vmware?

 

[diehard]

Edit: According to this article http://www.virten.net/2015/01/working-with-freenas-9-3-block-vaai-primitives/ dead space reclamation (UNMAP) only works if your iSCSI extent is a ZVOL created as sparse volume.

Yeah this is wrong, I'm 100% sure it works on a non-sparse volume.

 

[Mr_N]

Another fragmentation question although not related to this one:

When adding my 2nd vdev to my pool which is a bit over 50% full, I have about 600GB of data i need to replace in my pool and more in the future to redistribute some of it across the new vdev...

Which will cause less fragmentation if I delete all 600GB off pool first and then copy the replacement data back to it, or if I delete and replace chunks of it one by one?

 

[Robert Trevellyan]

Which will cause less fragmentation if I delete all 600GB off pool first and then copy the replacement data back to it, or if I delete and replace chunks of it one by one?

I would expect to see less fragmentation with the first option.

 

[Mr_N]

good thats what i thought :)

 

[zeroluck]

Yeah this is wrong, I'm 100% sure it works on a non-sparse volume.

How 100% sure are you of that? I am using file-based non-sparse extents and the command

Code:

esxcli storage core device vaii status get

for this volume returns:

Code:

naa.6589cfc0000006788931aadb43621c32
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: unsupported

on all of my VMware ESXi hosts for all of my FreeNAS volumes. UNMAP support is enabled globally on all my hosts. Does Delete Status need to be listed as supported there for UNMAP operations to work? I am rebuilding one of my arrays after I overcome a vMotion issue so I will test and see whether it works in the 3 scenarios when I do that.

1. iSCSI file extent
2. iSCSI device extent
3. iSCSI device extent with thin volume checkbox enabled

 

[diehard]

Delete Status is support under my LUN's, all device extents non-sparse. UNMAP commands work, it frees space and lowers fragmentation.

 

[mav@]

Delete/UNMAP should work for both forms of iSCSI device extents, independently from thin volume checkbox. Thin volume allows you to overcommit your storage, but it is not required for UNMAP operation. Opposite is almost mandatory same time -- if are doing overcommit, you must use UNMAP.

 

[zeroluck]

Any ideas why my hosts are reporting this then?

Code:

naa.6589cfc0000006788931aadb43621c32
VAAI Plugin Name:
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: unsupported

 

[mav@]

Those are you words: "I am using file-based non-sparse extents". That is the answer. You must use device/zvol extent to have UNMAP. Here is what I see on my system with zvol-based iSCSI extent:

Code:

~ # esxcli storage core device vaai status get
naa.6589cfc00000000064c13924d5403612
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

With "both" I meant thin and think provisioned extents.

 

[zeroluck]

Those are you words: "I am using file-based non-sparse extents". That is the answer. You must use device/zvol extent to have UNMAP. Here is what I see on my system with zvol-based iSCSI extent:

Code:

~ # esxcli storage core device vaai status get
naa.6589cfc00000000064c13924d5403612
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

With "both" I meant thin and think provisioned extents.

Excellent! I wish you had popped in with that comment a lot earlier in the conversation as I was looking for reasons to use device/zvol volumes over file-based ones. I couldn't find that information anywhere. For clarity of people later reading this, I don't think you can do sparse file-based extents at all. Thanks everyone!

Edit: I see now that depasseg's link had that information in it. My bad.