vmware snapshots not deleted

[toadman]

I am experimenting with vmware snapshots. I configured two, one for each of my two hosts. From reading the documentation I think the way it's supposed to work is:

1. Freenas sends command to esxi host to snapshot the vm(s) that use a freenas hosted datastore (i.e. a dataset or zvol) that is to be snapshotted on the freenas side
2. snapshot the freenas dataset
3. send command to esxi host delete the vmware snapshot created in step 1.

Things seemed to work as expected except for step 3. The freenas triggered vm snapshot on the esxi host remains.

Here is what the Freenas documentation says re: vmware snapshots: "Once this type of snapshot is created, FreeNAS® will automatically snapshot any running VMware virtual machines before taking a scheduled or manual ZFS snapshot of the dataset or zvol backing that VMware datastore. The temporary VMware snapshots are then deleted on the VMware side but still exist in the ZFS snapshot and can be used as stable resurrection points in that snapshot."

Is my understanding correct and there is a problem? Or am I misunderstanding the sequence?

 

[cyberjock]

What build of FreeNAS are you using? There's been a few builds that were bugged.

Also what version and build of ESXi are you using?

Your understanding is correct, so something is wrong. Whether it's on the FreeNAS side or ESXi side, I don't know.

 

[toadman]

Both ESXi hosts are VMware ESXi, 6.0.0, 2809209
FreeNAS is FreeNAS-9.3-STABLE-201506292130

I checked again this morning. Another FreeNAS created snapshot exists for each VM (as would be expected), but it's not deleted. (So now there are two.) I'll have to dig into the logs.

 

[cyberjock]

Yeah, I have zero experience with ESXi 6.0.0, so I can't really tell you if that should work or if it has been tested. I'd expect that if I asked the iXsystems devs they'd say it works on 6.0.0, but I don't have the knowledge to actually claim that.

 

[toadman]

I will test to see if it might be related to this issue. I do have a cluster and I don't have the vcenter server listed in FreeNAS. I'll update my entry and try again tonight.


https://bugs.freenas.org/issues/9104

Description

The FreeNAS docs explain VMWare Snapshots as shown below:

Setting Value Description
Hostname string IP address or hostname of VMware host

This should be expanded on to include a statement that if there is an ESXi cluster being used then the vcenter server for the cluster should be listed for the hostname.

 

[toadman]

Yeah, I have zero experience with ESXi 6.0.0, so I can't really tell you if that should work or if it has been tested. I'd expect that if I asked the iXsystems devs they'd say it works on 6.0.0, but I don't have the knowledge to actually claim that.

I suspect you are right, that the developers would say it works with 6.0.0. I know the watch commands haven't changed from 5 to 6. Not sure on the remove commands, but I doubt it.

 

[dlavigne]

One of the devs mentioned that it's a known bug with esxi 6 as the snapshots FreeNAS creates should get deleted by FreeNAS.

 

[toadman]

Thanks for that info dlavigne.

I checked this morning and there are no vm shapshots present in vmware. So this is different from my last attempt. This time I setup the vmware-snapshot in freenas to point at the vcenter server vs. the individual esxi hosts. (My two hosts are in a cluster, but I don't have DRS turned on.)

I can't tell yet if the vms were successfully snapped in vmware, then were deleted, or just never snapped at all. I will have to look at the freenas snapshot of the zvol later to find out. (or just check the logs.) I'll let you guys know what I find.

 

[lofwyr]

Hi,

I have run into this issue (I think at least) at work where we keep different versions of ESXi for testing. Our older 6.* versions stack up old snapshots that needs to be manually deleted but the latest version installed does not. Did you ever find out anything more @toadman?

 

[toadman]

I didn't make much headway. I implemented another solution when it wasn't working for me originally

Having just looked, I can tell you the vmware snapshots are just not happening now, at least based on cloning last nights snap of one of the datasets that has live VMs on it. Obviously a different issue than before, which was your case - snap created, not deleted. I haven't really checked in a while because I have another solution implemented, but I'll play with it and see if I can get it to work.

I've got a new vmware snapshot pointing at the vcenter 6.0, so we'll see what happens. The vcenter and the hosts are all on the latest patches. I'll check tomorrow.

 

[dlavigne]

If you end up creating a bug report, post the issue number here.

 

[toadman]

I can confirm the vmware snapshots work for me on an NFS datastore. I deleted the old vmware snapshot in FreeNAS and recreated it just to make sure it was correct. The dataset was snapped last night. I cloned the snap and confirmed the FreeNAS snapshot is there and that it is not present on the live filesystem. So it looks as if all worked as expected.

I pointed the vmware snapshot at the vcenter that managed a 3 host cluster. The vcenter and hosts are all patched to the latest available patches (as of last monday). FreeNAS is FreeNAS-9.3-STABLE-201602031011.

 

[lofwyr]

I just wanted to get back and thank you as well as say that it all seems to be working now with the latest versions of involved softwares. :)