Backup bhyve Windows VM

[revengineer]

After googling for an hour and not finding a satisfactory answer, I am posting a question that must have been answered somewhere and I am just not smart enough to find it. So here I go...

I have a Windows 10 bhyve VM on a 250GB ZVOL. The VM runs of a 1 TB SSD. I would like to make daily backups of the ZVOL on a spinning drive so that I can restore it incase the SSD goes south. The content of the Windows 10 VM changes daily. I only need to keep the last ~7 daily backups. Are snapshots and replication the way to go here? Or will the snapshots quick clobber up my SSD space? How would I restore the VM?

If someone could point me to where this has been answered or, even better, a tutorial, that would be fantastic.

Thank you.

 

[Samuel Tai]

Since bhyve doesn't have any mechanism to quiesce the zvol, you'll have to shutdown the VM before taking a snapshot. The first snapshot will be full-size; subsequent snapshots will only be deltas.

Once you have snapshots, many things become possible. You can rollback to a previous snapshot if the current zvol has a problem. You can replicate snapshots to another system for backup. Note all snapshot operations have to be done with the VM down.

 

[revengineer]

Understood regarding the need to shutdown the vm. With respect to the snapshot, if the ZVOL is 250 GB and I do a snapshot, does that mean that this VM then occupies 500 GB on the drive? I am asking because the SSD is only 1 TB so it will quickly fill up.

 

[Samuel Tai]

It depends how much of that 250 GB is actually in use. For example, I have a 25 GB zvol for a Docker VM, with 3 snapshots. zfs get all main/zvols/rancher-zvol shows:

Code:

main/zvols/rancher-zvol  usedbysnapshots          8.01G                    -
main/zvols/rancher-zvol  usedbydataset            20.9G                    -

The first snapshot used 3.5G; the second, 2.38G; and the third, 1.37G.

 

[revengineer]

Thank you, that is a useful command, I shall take that in my notes for future reference!

 

[Dan Tudora]

hello
or maybe can use a "pure" backup solution for windows like veeam agent, it's free, not need to shutdown the windows VM because use shadow copy
and make incremental backup
succes

 

[Patrick M. Hausen]

Snapshots are incremental, so the space should not be a concern for a while.

As for the need to shut down - I take snapshots of running VMs all the time. You need to consider which applications are running inside the VM and what their requirements with respect to persistent data are. Rolling back to a snapshot is like having pulled the power plug at that time. So you will definitely loose e.g. in flight database transactions. Essentially everything the guest OS has not flushed to its disk.
If you can live with that, fine. I can. An fsck, a mysql_check --auto-repair, and all is well for my specific requirements.

 

[revengineer]

hello
or maybe can use a "pure" backup solution for windows like veeam agent, it's free, not need to shutdown the windows VM because use shadow copy
and make incremental backup
succes

Interesting you bring this up. I was just thinking about this last night. I have urbackup running in a jail and image backups work. The change blocking tracking by urbackup client is not working but I think I can live with that. I have to try restoring the backup but think it should work. If not, veeam would be my second choice.

 

[revengineer]

Snapshots are incremental, so the space should not be a concern for a while.

As for the need to shut down - I take snapshots of running VMs all the time. You need to consider which applications are running inside the VM and what their requirements with respect to persistent data are. Rolling back to a snapshot is like having pulled the power plug at that time. So you will definitely loose e.g. in flight database transactions. Essentially everything the guest OS has not flushed to its disk.
If you can live with that, fine. I can. An fsck, a mysql_check --auto-repair, and all is well for my specific requirements.

Thanks for the feedback. I will keep that in mind. The VM in question is running emby, and when all setup, I expect very few changes. The biggest changes are with content which is store in SMB shares outside the VM. So I am actually ok with manual snapshots and replication. But it's good to know that live snapshots could work if needed.

 

[bal0an]

Given (using Freenas 11.3-U4.1) I want to take snapshots when the vm is shut down, I'd like to see a sequence of automated events (tasks | cronjob ?):
for each vm:
1. shutdown vm
2. take a snapshot (there is 'zfs snapshot' for that)
3. startup vm

I was looking for ways to stop and start the vm, but neither vm nor iohyve seem to be a valid approach. I have found one hint to use the api. And then, I played around with

Code:

curl -X POST "http://freenas.local/api/v2.0/vm/id/1/status" -H "accept: */*" -H "Content-Type: application/json" -d "{}"

which didn't really work.

Is there an example or recipe showing how to stop and start vms using the API?

 

[Samuel Tai]

Use midclt call vm.stop <VM ID> and midclt call vm.start <VM ID>.

 

[bal0an]

The name of my vm is mars. How do I find out the <vm id>?
I can't find the <vm id> in the WebUI...

 

[Samuel Tai]

The easiest is to look for the number in the VM's COM port: It'll be something like /dev/nmdm1B. The VM ID will be the number.

Another way is to run midclt call vm.query | jq, and look for the "id" line.

 

[bal0an]

Thanks. A hint to midclt with an example would make a nice and helpful addition to the API manual section... The current example is in python - which is a good choice for more complicated use cases - though much too verbose for my use case.

 

[Samuel Tai]

A hint to midclt with an example would make a nice and helpful addition to the API manual section...

I agree. Please submit a feature request to the developers. On the forum here, the moderators don't have the ability to alter the API documentation.

 

[seanm]

Use midclt call vm.stop <VM ID> and midclt call vm.start <VM ID>.

Does midclt call vm.stop <VM ID> do a clean shutdown of the guest OS? For that matter, does the "stop" button the web UI?

 

[seanm]

A hint to midclt with an example would make a nice and helpful addition to the API manual section

https://jira.ixsystems.com

 

[Samuel Tai]

Does midclt call vm.stop <VM ID> do a clean shutdown of the guest OS? For that matter, does the "stop" button the web UI?

From what I've observed, yes.

 

[bal0an]

I've created a script I use now to take cold snapshots from my vms:

Code:

#!/bin/bash
if [ $# -ne 2 ]; then
    echo "Usage: take_vm_snapshot <vm_name> <dataset>"
    exit 1
fi
VM_NAME=$1
DATASET=$2
RC=$(zfs list $DATASET)
if [ $? -eq 1 ]; then
    echo "Error: Dataset $DATASET not found. Aborting."
    exit 1
fi
echo $(date '+%Y-%m-%d %H:%M:%S') "Taking snapshot of $DATASET for VM $VM_NAME."
VM_ID=$(midclt call vm.query | jq ".[] | if .name == \"$VM_NAME\" then .id else empty end")
if [ "$VM_ID" == "" ]; then
    echo "Error: No VM found with name $VM_NAME. Aborting."
    exit 1
fi
echo $(date '+%Y-%m-%d %H:%M:%S') "vm $VM_NAME has id $VM_ID."
if [ $(midclt call vm.status $VM_ID | jq '.state') != "\"STOPPED\"" ]; then
    echo $(date '+%Y-%m-%d %H:%M:%S') "Shutting down VM $VM_NAME..."
    midclt call vm.stop $VM_ID
    while [ $(midclt call vm.status $VM_ID | jq '.state') != "\"STOPPED\"" ]; do
        echo $(date '+%Y-%m-%d %H:%M:%S') "Wait for vm $VM_NAME to terminate..."
        sleep 5
    done
fi
echo $(date '+%Y-%m-%d %H:%M:%S') "vm $VM_NAME stopped"
SNAPSHOT_NAME=$(date "+$DATASET@auto-%Y-%m-%d_%H-%M")
echo $(date '+%Y-%m-%d %H:%M:%S') "Taking snapshot $SNAPSHOT_NAME"
zfs snapshot $SNAPSHOT_NAME
echo $(date '+%Y-%m-%d %H:%M:%S') "Starting up VM $VM_NAME"
midclt call vm.start $VM_ID
echo $(date '+%Y-%m-%d %H:%M:%S') "Done."

The snapshots can be configured for remote replication. Automatically created snapshot tasks may be disabled/deleted.
Known issue: snapshot expiry and cleanup does not work.

 

[bal0an]

Does midclt call vm.stop <VM ID> do a clean shutdown of the guest OS? For that matter, does the "stop" button the web UI?

Yes, it does. When I call midclt call vm.stop $VM_ID with a VNC console open, I can see the vm properly shut down before the midclt call vm.status $VM_ID | jq '.state' changes to STOPPED.