Snapshots of VMS in the middle of a write?

[oguruma]

I'm trying to learn more about TrueNAS SCALE as it pertains to virtualization.

As I understand it, the main method of backing up a VM in SCALE is the zfs snapshotting of the VM's volume.

What happens if the snapshot task runs in the middle of a large file transfer, or any file transfer, for that matter?

Suppose the snapshot runs halfway through somebody moving a large video file to the machine, and that snapshot must be recovered from?

Is there a chance of anything being broken?

 

[jgreco]

Of course there is a chance of things being broken. This has been the bane of virtualization systems forever. You can never tell when an application is sitting there with stupid important data in core. The classic example are database servers.

There's lots of pros and cons to strategies that try to fix this by, for example, "quiescing" a guest OS for filesystem accesses, but this is relatively naive. If you're in the middle of a thirty day file transfer at 14.4kbps modem speeds, what is the right thing that should happen? Think carefully. It's complicated, and it can never be perfect.

In the cloud industry, this is often dealt with simply by virtue of having multiple restore points that can be rolled back to.

 

[sretalla]

Suppose the snapshot runs halfway through somebody moving a large video file to the machine, and that snapshot must be recovered from?

Depends heavily on the method of transfer... some can be continued from where they stopped, others not.

Imagine it as if the power was cut to a physical machine and then the machine restarted... rolling back the snapshot is the same as going back to the point in time when the power was cut.

 

[jgreco]

Depends heavily on the method of transfer... some can be continued from where they stopped, others not.

Well, that's true too. If we're strictly examining the impact on a single file copy, scp or ftp are not able to recover from an interrupted transfer, while rsync can (and rsync can even do it correctly, by validating the existing contents). This becomes more complicated once you get into stateful mechanisms such as web browsers, where the browser may be independently tracking state in a json or xml file, and isn't updating it on a to-the-second basis.

In general, you have the lack of atomicity of interrelated actions on the userland side, along with the cleanup effects of fsck or chkdsk on the operating system side, which tend to work together to create a bit of a disorganized mess.

Now, the upside is that hypervisors tend to be more careful about their treatment of VM's because these problems were understood early on. VMware ESXi, for example, is famous for "requiring" high end RAID controllers with battery backed write cache, because it is absolutely a goal of the hypervisor to be able to do all important writes synchronously, because VMware knows they have no visibility into which writes are actually important. People spend a lot of time sabotaging this, such as using a Syno or QNAP solution and turning on inappropriate async writes to "make it fast".

Overall, though, the problem with block-level snapshotting is that you're going to get that "moment in time" behaviour and @sretalla is right in that it's very much like someone went "Yoikes!" and pulled the plug on a running machine, that's the state you're left at.

However, even on-VM file-based backup systems (which often cannot capture "moment in time") suffer from these problems, as the underlying file system tends to change as the backup engine creates a copy of all the files. But even if we assume a system that's capable of filesystem-level snapshotting, such as ZFS or FFS+snaps, this doesn't actually fix the "in-progress" issue the OP is asking about.

Can you tell I've spent lots of my career wrestling with related issues?

 

[HoneyBadger]

Ah, the joy of explaining "crash-consistent backups" to customers, then they demand "application-consistent, quiesced backups" and wonder why it takes longer and/or times out, ca. the reference made by @jgreco to a 30-day transfer at 14.4kbps ...

TrueNAS has integration with VMware to use the guest OS/tools in Windows to attempt VSS (Volume Shadow copy Service) quiescing, but I'm uncertain if SCALE is integrated with qemu-qa in the same manner to call --quiesce through virsh. I also don't know what the timeout and fallback behaviors are there. Sorry.