Backing up Virtual Machines Off Site - ESXi Users

[DaveFL]

Hello all,

My understanding is that Freenas supports vaai which makes taking ZFS + ESXi snapshots relatively easy, my question is that if one wants to backup offsite those snapshots won't help much (I don't think). So for those who are running VMs and are backing up off site what is your process for doing so?

Is there anyway to hook into having the backup program run when Freenas is asking ESXi to take the snapshot? Perhaps there is an easier way.

Dave

 

[kdragon75]

You can simply use the FreeNAS "replication" also known as zfs send and receive. to make this works there are three main things to setup in FreeNAS

1. Plan everything out. Because you are coordinating two separate tasks make sure you have plenty of time for the first to run. Do a few test run with just step one and ad 50% more time just to be safe. Until we have better integration for vmware backups or some sort of even hooks for running tasks, we will need to time things out manually.
2. FreeNAS triggered VMware-Snapshots - This will setup a schedule for FreeNAS to log into your ESXi host or vCenter and tell every VM on the selected datastore to take a VMware snapshot. Once the VMware snapshot is done it will take a ZFS snapshot of your dataset/pool. Once the ZFS snapshot is done, it wil the VMware snapshots. NOTE: Be sure to select the ZFS dataset/pool that your VMware datastore resides on.
3. FreeNAS ZFS Replication - This part can take a little bit more work depending on your desired settings but the important part here is that you check the box for "Recursively replicate child dataset’s snapshots". This is due to the fact that the FreeNAS VMware snapshots will not coordinate with the replication tasks.

The FreeNAS VMware snapshots are a great and flexible way to run backups on your VMs. There are a few obvious limitations though. The biog one I see is that you must do a full dataset/datastore at a time. This kinda makes sense due to the fact that we are ultimately just backing up a snapshot of the underlying ZFS but there may be cases where VMware snapshotting large VMs will cause noticeable performance degradation even with a full sett of VMTools and drivers installed as a VM with 64GB of RAM wired will need to write a 64GB file. This is compounded by the number of VMs on your datastore.

Best practice in VMware is to keep datastores relatively small and have fewer VMs each. This has a number of benefits in addition to keeping the number of VMware snapshot events per FreeNAS VMware snapshot task to a minimum. For example, if using iSCSI, each datastore will be a separate iSCSI session/connection allowing better use of multiple network interfaces and LAGGs. Another benefit comes in the form of more IO queues. This helps to alleviate issues with micro bursting causing IO to pause even under low throughput.

I know I got a little off the rails there at the end but I'm sure someone can use a few of those nuggets.

If any of this is incorrect, wrong, or there is a better way, I'm all ears!

 

[kdragon75]

Hello all,

My understanding is that Freenas supports vaai which makes taking ZFS + ESXi snapshots relatively easy, my question is that if one wants to backup offsite those snapshots won't help much (I don't think). So for those who are running VMs and are backing up off site what is your process for doing so?

Is there anyway to hook into having the backup program run when Freenas is asking ESXi to take the snapshot? Perhaps there is an easier way.

Dave

To more directly address your post, VAAI won't offer much here. As implemented in FreeNAS if you copy a VM to the same datastore like if you were deploying from a template (from and to the same datastore), ESXi would just tell FreeNAS "Hey just make a copy of those blocks for me". The benefit is that we don't need to move those blocks over the network/SAN. Its all done inside FreeNAS. If we are making a VMware snapshot, we still need to send the contents of the VMs RAM to FreeNAS to be written to disk.

Once you have the FreeNAS snapshoted (is that a word?) VMware snapshots, you can use ZFS send/receive (FreeNAS replication task) to export that anywhere. Just remember its just like incremental backups. If you lose and incremental, everything after that is lost until your next full. If you lose your only full, you have nothing.

As for other backup solutions for VMware?
I. Love. Veeam.
There is a free version to play with but you cant scheduled backups with it. They do offer a free 30 day trial and its worth looking at if you willing to spend some cash on a flexible and robust solution. Just don't expect to use FreeNAS as a linux target even though there i no reason for it not to work. Don't worry though, you can still use FreeNAS as a CIFS target and thats dead simple. Just remember to ALWAYS only set read only permissions on your backups outside of your backup service account as I hear lots of ransomware is targeting backup file types too.

 

[DaveFL]

You can simply use the FreeNAS "replication" also known as zfs send and receive. to make this works there are three main things to setup in FreeNAS

1. Plan everything out. Because you are coordinating two separate tasks make sure you have plenty of time for the first to run. Do a few test run with just step one and ad 50% more time just to be safe. Until we have better integration for vmware backups or some sort of even hooks for running tasks, we will need to time things out manually.
2. FreeNAS triggered VMware-Snapshots - This will setup a schedule for FreeNAS to log into your ESXi host or vCenter and tell every VM on the selected datastore to take a VMware snapshot. Once the VMware snapshot is done it will take a ZFS snapshot of your dataset/pool. Once the ZFS snapshot is done, it wil the VMware snapshots. NOTE: Be sure to select the ZFS dataset/pool that your VMware datastore resides on.
3. FreeNAS ZFS Replication - This part can take a little bit more work depending on your desired settings but the important part here is that you check the box for "Recursively replicate child dataset’s snapshots". This is due to the fact that the FreeNAS VMware snapshots will not coordinate with the replication tasks.

The FreeNAS VMware snapshots are a great and flexible way to run backups on your VMs. There are a few obvious limitations though. The biog one I see is that you must do a full dataset/datastore at a time. This kinda makes sense due to the fact that we are ultimately just backing up a snapshot of the underlying ZFS but there may be cases where VMware snapshotting large VMs will cause noticeable performance degradation even with a full sett of VMTools and drivers installed as a VM with 64GB of RAM wired will need to write a 64GB file. This is compounded by the number of VMs on your datastore.

Best practice in VMware is to keep datastores relatively small and have fewer VMs each. This has a number of benefits in addition to keeping the number of VMware snapshot events per FreeNAS VMware snapshot task to a minimum. For example, if using iSCSI, each datastore will be a separate iSCSI session/connection allowing better use of multiple network interfaces and LAGGs. Another benefit comes in the form of more IO queues. This helps to alleviate issues with micro bursting causing IO to pause even under low throughput.

I know I got a little off the rails there at the end but I'm sure someone can use a few of those nuggets.

If any of this is incorrect, wrong, or there is a better way, I'm all ears!

Can replication be done without ZFS send or SSH? Would like to use rclone or something similar if possible.

 

[DaveFL]

To more directly address your post, VAAI won't offer much here. As implemented in FreeNAS if you copy a VM to the same datastore like if you were deploying from a template (from and to the same datastore), ESXi would just tell FreeNAS "Hey just make a copy of those blocks for me". The benefit is that we don't need to move those blocks over the network/SAN. Its all done inside FreeNAS. If we are making a VMware snapshot, we still need to send the contents of the VMs RAM to FreeNAS to be written to disk.

Once you have the FreeNAS snapshoted (is that a word?) VMware snapshots, you can use ZFS send/receive (FreeNAS replication task) to export that anywhere. Just remember its just like incremental backups. If you lose and incremental, everything after that is lost until your next full. If you lose your only full, you have nothing.

As for other backup solutions for VMware?
I. Love. Veeam.
There is a free version to play with but you can't scheduled backups with it. They do offer a free 30 day trial and its worth looking at if you willing to spend some cash on a flexible and robust solution. Just don't expect to use FreeNAS as a linux target even though there i no reason for it not to work. Don't worry though, you can still use FreeNAS as a CIFS target and thats dead simple. Just remember to ALWAYS only set read only permissions on your backups outside of your backup service account as I hear lots of ransomware is targeting backup file types too.

I installed this yesterday. It seems to work quite well. What I did was exact this. Setup a CIFS target and and put the Proxy and Backup server on the same machine. Process time was about 130mb/sec not sure if that is good or bad for an ESXi setup. I had similar performance with both the virtual appliance and direct nfs settings.

I need to experiment more but I’m think that I can rclone the files to B2 or Wasabi afterwards.

 

[kdragon75]

Can replication be done without ZFS send or SSH? Would like to use rclone or something similar if possible.

You could use ZFS send and pipe it into gzip or similar and write it to a dataset then use rclone but that's all CLI and cron jobs you would have to set up.

 

[kdragon75]

I installed this yesterday. It seems to work quite well. What I did was exact this. Setup a CIFS target and and put the Proxy and Backup server on the same machine. Process time was about 130mb/sec not sure if that is good or bad for an ESXi setup. I had similar performance with both the virtual appliance and direct nfs settings.

I need to experiment more but I’m think that I can rclone the files to B2 or Wasabi afterwards.

Do you mean 130MB/sec or 130mb/sec? Veeam does a lot of processing of the data. It performs checksums, deduplication, and lots of compression and this is all extremely CPU & memory intensive. If your in the 130MB/sec range in a VM, make sure you not starving out other VMs for CPU cycles.

FreeNAS has built in B2 support that you can use with files/folder on FreeNAS. You may consider running FreeNAS based VMware snapshots and adding a cron command to ZFS send zpool/vdev | bzip backupfiel.bz then a few minutes later run the Cloud Sync.

Personally I would prefer Veeam and syncing the VIB files over to the cloud. If you configure all you VMs in one job using dedup, you save a TON of space. Also Veeam is just amazing for microsoft environments and restoring crap for users.