Letni
Explorer
- Joined
- Jan 22, 2012
- Messages
- 63
HOW-TO: Managing ESX Thin-Provisioned devices at the VMWare Layer as well as the ZFS Layer
In ESX(i) Virtual environments where it is necessary to worry about pooled type storage and all that accompanys them (Thin Provisioning, oversubscribing, etc) I have learned quite a bit when it comes to managing and maintenance for ESXi utilizing storage (iSCSI) on FreeNAS (FreeBSD) ZFS storage and I'm hoping that this is the proper place to document such experiences.
Audience:
This guide is intended for I.T. professionals who are versed in VMWare ESX(i) technology enough to be able to create / manage Virtual Machines in a small / medium scale. A lot of this guide references utilizing FreeNAS with ZFS pooled storage for the back end of said virtualization infrastructure, however portions are specific to ESXi and may generally help non FreeNAS users as well.
**If you don't know what server virtulaization is or even considerations around storage level management, then don't waste your time reading beyond this.***
What this guide does not do (it assumes you already know how to):
1. Explain how to set up FreeNAS on a physical machine and configure storage in ZFS
2. Explain how to set up iSCSI on FreeNAS
3. Explian how to install ESXi 4.x / 5.x on a separate physical machine
4. Explain how to configure ESXi to talk to iSCSI on FreeNAS and create VMFS datastores
5. Explain how to set up ESX to run or host virtual machines
6. Explain how to set up vCenter Server for ESX(i) environment
7. Explain day-to-day management of ESX environment through vCenter
8. Act as any sort of guide configuring FreeNAS to run WITHIN a virtual environment
Background:
I have build myself a nice sandbox utilizing FreeNAS 8.0.3-p1 where I have several larger drives set up in a RAID-Z1 pool that has both ZFS Datasets defined for NAS type storage, as well as ZFS Volumes to be used with the iSCSI Service for running the ESXi (currently 5.0) environment that I have a separate server built to run. Having somewhat a storage background, I prefered to be as "thin-active" as the technology allows (both ESXi 5 and ZFS). There is quite some capability here after researching out both technologies but they come with some caveats required to understand to successfully maintain thin storage with the combinations of these two infrastructre softwares.
Technology Required:
In order to be able to follow the methodoligies and best practices I'm including in this guide, the following infrastructure will need to be configured:
1. FreeNAS 7.X, 8.X with some type of ZFS pool that iSCSI volumes can be carved up for use by #2 below as VMFS datastores.
2. ESXi Server (4.1 or 5.0) - Supports of Thin Provisioned Virtual Machine datastores
3. Virtual Machines configured for use in the virtual environment (storage remaining perferably on FreeNAS/ZFS backend)
a. Windows 7/2003/2008 64 Bit to host ESX vSphere Server (required for VMotion)
b. Other ESXi Virtual Machines that are created with their storage (VMDK) as thin type.
3. Management machine (place to connect to FreeNAS via SSH, as well as connect to ESXi via vSphere client)
Terminology
-Thin Maintenance / Shrinking:
The concept of taking consumed space away from an object in the pool when the object has previously "consumed" a pools resources (space), but no longer needs that resources.
-Dual Layer Thin environment:
Both ZFS and VMFS provide thin provisioning type environments, one layered below the other. Maintenance done to one layer may not effect the other.
Technology Properties
-ZFS Provisioning - ZFS pool and ZFS Volumes (type of resource) configured in that pool
1. ZFS observed behavior around ZFS block volumes
a. ZFS will "reserve" a ZFS volume's space upfront and and subtract the volume's reserved space from all other objects in the pool (datasets, other volumes)
b. ZFS does not actually allocate that space from the pool until data is written to the volume (Thin-type property)
c. Consumed blocks that are zero'd out will be release from the volume and be given back to the pool (VERY good attribute of this technology)
-ESX / VMFS Filesystem
1. VMFS Filesystem can be considered to be a storage pool containing other filesystems . When it is created on a device presented from FreeNAS sitting on ZFS, it technically ends up being a "pool within a pool". The VMWare Guest OS filesystem then sits within this, creating many layers of storage allocation..
ZFS Pool
->ZFS Block Volume
-->VMFS Pool (via iSCSI)
--->Guest OS filesystem (VMDK) - NTFS/ext3/etc
2. ESXi VM provisioning (in a VMFS filesystem / datastore) can be configured 3 ways.
Eager Zero: (all blocks are zeroed out ahead of time and all VMFS space consumed up-front: (need 10 GB for a VM-regardless of how much data, it will take 10 GB out of VMFS)
Lazy Zero: Similar to Eager Zero, however data is zero'd out on first write to block
Thin provisioned (sparse file): Space is taken out of VMFS only as data is loaded into the VM (10 GB provisioned VM, but only 5 GB of data results in only 5 GB of space out of VMFS)..
Observed basic behaviors:
a. Eager Zero and Lazy Zero thick devices are NOT thin-devices, they are to consume the expected space, which means quickly can consume all useable space configured to VMFS
b. Non-Thin devices linearlly eat up all of VMFS capacity. 10 x 10GB non-ESX-thin VMDK devices will consume 100% of a 100GB VMFS datastore, even if all 10 VMDKs are EMPTY.
c. For Eager and Lazy Zero VMs (VMDK), ZFS in FreeNAS will still recognize blank space and adjust accordingly at the ZFS Thin Pool Layer. So a 10 GB thick device with 5 GB data will at the VMFS layer starts out consuming 10 GB of the VMFS datastore, however ZFS will only consume 5 GB worth of space out of the ZFS pool (even less if ZFS compression is enabled). So in essence, Eager and Lazy zero devices are THICK to VMFS, but are thin to ZFS.
d. VMFS thin devices start out at consuming less space at the VMFS and ZFS layers, as data is added VMFS will consume more space in the VMFS datastore (pool), as well as the ZFS Layer. These devices are THIN to VMFS and thin to ZFS.
e. Deleting files within a Virtual Machine (Windows recycle bin) do not FULLY delete data from the disk (allowing pool resources to be cleaned up). A full free-space wipe must be performed to Zero out deleted data. This can be done with multiple tools:
- ccleaner has a free space wipe tool built in
- sdelete.exe from systernals (Microsoft) can zero out free space
- precompact.exe from Microsoft / Virtual PC can be used to zero out free space
i. ZFS will instantly react to wiping free space (can confirm by running '#zfs list' within FreeNAS while the tool is running in the VM)
ii. ****VMFS (version 3 or 5) will NOT instantly react to wiping the data within the VM (within the VMDK) on ESX-thin-allocated VMs****
f. Performing free space wipe will inflate the VMDK to almost full capacity at the ESXi Level, at the same time it deflates the ZFS pool use.
g. VMFS can only reclaim (maintenance) zero'd out space within a VMDK by use of Storage vMotion or through CLI tools (if Storage vMotion is not available - ie no vSphere server)
h. VMFS will only reclaim space on ESX-thin-allocated VMs when source VMFS (Datastore) and target VMFS (datastore) use a different formatted block size (IE 1MB Block -> 2MB Block)
i. ESXi 5.0 with VMFS version 5.0 only supports one block size (1MB) so creating the second VMFS datstore with VMFS version 3.0 will be necessary to select a different block size)
j. Storage VMotion a VM off of a datastore (VMFS) will NOT Zero out the blocks on that data store - not cleaning up ZFS pool underneath.
Best Practice for Maintenance on ESX with VMFS/Thin and ZFS for reclaiming space given the plethora of information above:
A. If using ZFS for thin in the backend disk layer, but allocating full (non-thin) VMFS/VMDK based virtual machines
1. Simply run the noted free space tools above to wipe data from the ZFS pool
B. If using ZFS for thin at the backend disk layer, but are using ESX 4.X or 5.X thin type devices:
1. Make sure all VMs are created as type THIN and are not fully expanded (or planning on converting from thick to thin)
2. Use one of the formentioned tools above (unfortunately I haven't found tools for other than Windows that can do this) to zero out all the freespace in the VM (all drives).
3. Create a new ZFS Volume (good sized), map/mask through iSCSI to your ESX server and format VMFS version 3 with a different block size than your other datastores (2MB block or higher). This will be a temp datastore.
4. Use Storage vMotion (Virtual Disk format = Thin Provision) to move ALL virtual machines off of the original datastore (evacuate). This process will fix all the virtual allocated size within VMFS as it sits on the new VMFS datastore
5. Create a new Virtual Machine on the original VMFS Datastore (doesn't matter what OS), only that when it asks the virtual disk, you chose Eager Zero and choose a size that is about 99 % the size of the free space in the datastore. Let this process complete (may take time) as this zero's out the datastore - Fixing the ZFS Volume in the pool below.. check by performing a '#zfs list'
6. Delete the New Virtual Machine from disk (datastore)
7. Use Storage vMotion (Virtual Disk format = Same format as source) to move the Virtual Machines BACK to the original datastore as the VMs are not thin friendly, and the Datastore is also thin friendly.
8. Remove the Temp datastore (or leave it for future repeate of this process - but perform step 5 on the temp datastore to clear its ZFS data out of the pool.)
Now you should have a cleaned up ZFS pool as well as a cleaned up VMFS datastore.
In ESX(i) Virtual environments where it is necessary to worry about pooled type storage and all that accompanys them (Thin Provisioning, oversubscribing, etc) I have learned quite a bit when it comes to managing and maintenance for ESXi utilizing storage (iSCSI) on FreeNAS (FreeBSD) ZFS storage and I'm hoping that this is the proper place to document such experiences.
Audience:
This guide is intended for I.T. professionals who are versed in VMWare ESX(i) technology enough to be able to create / manage Virtual Machines in a small / medium scale. A lot of this guide references utilizing FreeNAS with ZFS pooled storage for the back end of said virtualization infrastructure, however portions are specific to ESXi and may generally help non FreeNAS users as well.
**If you don't know what server virtulaization is or even considerations around storage level management, then don't waste your time reading beyond this.***
What this guide does not do (it assumes you already know how to):
1. Explain how to set up FreeNAS on a physical machine and configure storage in ZFS
2. Explain how to set up iSCSI on FreeNAS
3. Explian how to install ESXi 4.x / 5.x on a separate physical machine
4. Explain how to configure ESXi to talk to iSCSI on FreeNAS and create VMFS datastores
5. Explain how to set up ESX to run or host virtual machines
6. Explain how to set up vCenter Server for ESX(i) environment
7. Explain day-to-day management of ESX environment through vCenter
8. Act as any sort of guide configuring FreeNAS to run WITHIN a virtual environment
Background:
I have build myself a nice sandbox utilizing FreeNAS 8.0.3-p1 where I have several larger drives set up in a RAID-Z1 pool that has both ZFS Datasets defined for NAS type storage, as well as ZFS Volumes to be used with the iSCSI Service for running the ESXi (currently 5.0) environment that I have a separate server built to run. Having somewhat a storage background, I prefered to be as "thin-active" as the technology allows (both ESXi 5 and ZFS). There is quite some capability here after researching out both technologies but they come with some caveats required to understand to successfully maintain thin storage with the combinations of these two infrastructre softwares.
Technology Required:
In order to be able to follow the methodoligies and best practices I'm including in this guide, the following infrastructure will need to be configured:
1. FreeNAS 7.X, 8.X with some type of ZFS pool that iSCSI volumes can be carved up for use by #2 below as VMFS datastores.
2. ESXi Server (4.1 or 5.0) - Supports of Thin Provisioned Virtual Machine datastores
3. Virtual Machines configured for use in the virtual environment (storage remaining perferably on FreeNAS/ZFS backend)
a. Windows 7/2003/2008 64 Bit to host ESX vSphere Server (required for VMotion)
b. Other ESXi Virtual Machines that are created with their storage (VMDK) as thin type.
3. Management machine (place to connect to FreeNAS via SSH, as well as connect to ESXi via vSphere client)
Terminology
-Thin Maintenance / Shrinking:
The concept of taking consumed space away from an object in the pool when the object has previously "consumed" a pools resources (space), but no longer needs that resources.
-Dual Layer Thin environment:
Both ZFS and VMFS provide thin provisioning type environments, one layered below the other. Maintenance done to one layer may not effect the other.
Technology Properties
-ZFS Provisioning - ZFS pool and ZFS Volumes (type of resource) configured in that pool
1. ZFS observed behavior around ZFS block volumes
a. ZFS will "reserve" a ZFS volume's space upfront and and subtract the volume's reserved space from all other objects in the pool (datasets, other volumes)
b. ZFS does not actually allocate that space from the pool until data is written to the volume (Thin-type property)
c. Consumed blocks that are zero'd out will be release from the volume and be given back to the pool (VERY good attribute of this technology)
-ESX / VMFS Filesystem
1. VMFS Filesystem can be considered to be a storage pool containing other filesystems . When it is created on a device presented from FreeNAS sitting on ZFS, it technically ends up being a "pool within a pool". The VMWare Guest OS filesystem then sits within this, creating many layers of storage allocation..
ZFS Pool
->ZFS Block Volume
-->VMFS Pool (via iSCSI)
--->Guest OS filesystem (VMDK) - NTFS/ext3/etc
2. ESXi VM provisioning (in a VMFS filesystem / datastore) can be configured 3 ways.
Eager Zero: (all blocks are zeroed out ahead of time and all VMFS space consumed up-front: (need 10 GB for a VM-regardless of how much data, it will take 10 GB out of VMFS)
Lazy Zero: Similar to Eager Zero, however data is zero'd out on first write to block
Thin provisioned (sparse file): Space is taken out of VMFS only as data is loaded into the VM (10 GB provisioned VM, but only 5 GB of data results in only 5 GB of space out of VMFS)..
Observed basic behaviors:
a. Eager Zero and Lazy Zero thick devices are NOT thin-devices, they are to consume the expected space, which means quickly can consume all useable space configured to VMFS
b. Non-Thin devices linearlly eat up all of VMFS capacity. 10 x 10GB non-ESX-thin VMDK devices will consume 100% of a 100GB VMFS datastore, even if all 10 VMDKs are EMPTY.
c. For Eager and Lazy Zero VMs (VMDK), ZFS in FreeNAS will still recognize blank space and adjust accordingly at the ZFS Thin Pool Layer. So a 10 GB thick device with 5 GB data will at the VMFS layer starts out consuming 10 GB of the VMFS datastore, however ZFS will only consume 5 GB worth of space out of the ZFS pool (even less if ZFS compression is enabled). So in essence, Eager and Lazy zero devices are THICK to VMFS, but are thin to ZFS.
d. VMFS thin devices start out at consuming less space at the VMFS and ZFS layers, as data is added VMFS will consume more space in the VMFS datastore (pool), as well as the ZFS Layer. These devices are THIN to VMFS and thin to ZFS.
e. Deleting files within a Virtual Machine (Windows recycle bin) do not FULLY delete data from the disk (allowing pool resources to be cleaned up). A full free-space wipe must be performed to Zero out deleted data. This can be done with multiple tools:
- ccleaner has a free space wipe tool built in
- sdelete.exe from systernals (Microsoft) can zero out free space
- precompact.exe from Microsoft / Virtual PC can be used to zero out free space
i. ZFS will instantly react to wiping free space (can confirm by running '#zfs list' within FreeNAS while the tool is running in the VM)
ii. ****VMFS (version 3 or 5) will NOT instantly react to wiping the data within the VM (within the VMDK) on ESX-thin-allocated VMs****
f. Performing free space wipe will inflate the VMDK to almost full capacity at the ESXi Level, at the same time it deflates the ZFS pool use.
g. VMFS can only reclaim (maintenance) zero'd out space within a VMDK by use of Storage vMotion or through CLI tools (if Storage vMotion is not available - ie no vSphere server)
h. VMFS will only reclaim space on ESX-thin-allocated VMs when source VMFS (Datastore) and target VMFS (datastore) use a different formatted block size (IE 1MB Block -> 2MB Block)
i. ESXi 5.0 with VMFS version 5.0 only supports one block size (1MB) so creating the second VMFS datstore with VMFS version 3.0 will be necessary to select a different block size)
j. Storage VMotion a VM off of a datastore (VMFS) will NOT Zero out the blocks on that data store - not cleaning up ZFS pool underneath.
Best Practice for Maintenance on ESX with VMFS/Thin and ZFS for reclaiming space given the plethora of information above:
A. If using ZFS for thin in the backend disk layer, but allocating full (non-thin) VMFS/VMDK based virtual machines
1. Simply run the noted free space tools above to wipe data from the ZFS pool
B. If using ZFS for thin at the backend disk layer, but are using ESX 4.X or 5.X thin type devices:
1. Make sure all VMs are created as type THIN and are not fully expanded (or planning on converting from thick to thin)
2. Use one of the formentioned tools above (unfortunately I haven't found tools for other than Windows that can do this) to zero out all the freespace in the VM (all drives).
3. Create a new ZFS Volume (good sized), map/mask through iSCSI to your ESX server and format VMFS version 3 with a different block size than your other datastores (2MB block or higher). This will be a temp datastore.
4. Use Storage vMotion (Virtual Disk format = Thin Provision) to move ALL virtual machines off of the original datastore (evacuate). This process will fix all the virtual allocated size within VMFS as it sits on the new VMFS datastore
5. Create a new Virtual Machine on the original VMFS Datastore (doesn't matter what OS), only that when it asks the virtual disk, you chose Eager Zero and choose a size that is about 99 % the size of the free space in the datastore. Let this process complete (may take time) as this zero's out the datastore - Fixing the ZFS Volume in the pool below.. check by performing a '#zfs list'
6. Delete the New Virtual Machine from disk (datastore)
7. Use Storage vMotion (Virtual Disk format = Same format as source) to move the Virtual Machines BACK to the original datastore as the VMs are not thin friendly, and the Datastore is also thin friendly.
8. Remove the Temp datastore (or leave it for future repeate of this process - but perform step 5 on the temp datastore to clear its ZFS data out of the pool.)
Now you should have a cleaned up ZFS pool as well as a cleaned up VMFS datastore.
