SOLVED How do I backup a VM (not showing in FS)

[Phase]

Hi, I want to backup a TrueNAS VM after the initial setup to another TrueNAS machine in case the original server goes capult for some reason.

First thing I tried was replication, but while the VM has 304 GiB the replicated image was nearly empty. So then when I did a "du -h" on that mount point I found that the actual storage used by the VM is not accounted for in the file system. That is a 444 GiB pool and only 140 GiB is accounted for -- i.e. the 304 GiB used by the VM do not seem to be part of the visible file system.

$ cd /mnt/OVERVOLT $du -h <snip> OVERVOLT 140G 252K 140G 0% /mnt/OVERVOLT OVERVOLT/My-VM 140G 88K 140G 0% /mnt/OVERVOLT/My-VM <snip> $ cd /mnt/OVERVOLT/My-VM $ ls -la drwxrwxr-x+ 2 root vmking 2 Nov 8 2021 . drwxr-xr-x 4 root wheel 4 Nov 8 2021 ..

Any thoughts on what is going on, and more importantly, how can I make a copy of the VM for posterity?

Thanks!!

 

[Arwen]

First off, if you are trying to copy the TrueNAS OS, don't. Make a backup of the configuration, both off box and probably in your data pool(s). Then keep an ISO image of the TrueNAS install you used. Combined with the configuration, you can reload the "OS" part of the NAS easily.

Next, ZFS has the ability to have both snapshots & clones. Check with below and post output here, in CODE tags;
zfs list -t all -r

Last, ZFS block storage is done outside of datasets and mounted file systems. It is done throuh zVols, which act like pseudo disks.

 

[Phase]

Thanks, that move things along a bit, "zfs list -t all -r" shows the 305G that I'm looking to backup. When I tried replicate the snapshot shown below the replication engine did not pickup those 305G. How can I save a copy of the VM in another server?

NAME USED AVAIL REFER MOUNTPOINT OVERVOLT/MyUbuntuVM 305G 140G 88K /mnt/OVERVOLT/MyUbuntuVM OVERVOLT/MyUbuntuVM@manual-2022-05-20_10-18 56K - 88K - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp 305G 262G 182G -

 

[Patrick M. Hausen]

E.g.

Code:

# create snapshot
zfs snap OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07
# create backup in regular file in some suitable location
zfs send OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07 | gzip -c >/some/storage/place/MyUbuntuVM.gz
# or send backup to other server and create regular file there
zfs send OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07 | gzip -c | ssh some.server "cat >/some/storage/place/MyUbuntuVM.gz"
# destroy snapshot - won't hurt VM image, only snapshot
zfs destroy OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07

 

[Arwen]

Based on what I know now, it appears you tried to send the parent dataset and not the zVol. @Patrick M. Hausen has the correct commands.

One note about zVols, you can have a "refreservation" that makes the zVol look like it is using 305GB of space. But, could be actually using much less space. Basically a reservation reserves space for the dataset or zVol out of the pool.

So, next step, send us the output of;
zfs get all OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp

 

[Phase]

Yes! Progress.

zfs snap OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07 gives an OUT OF SPACE message (?)
cannot create snapshot 'OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp@2022-06-07': out of space

zfs get all OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp gives
NAME PROPERTY VALUE SOURCE OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp type volume - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp creation Fri Oct 23 14:01 2020 - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp used 305G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp available 262G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp referenced 182G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp compressratio 1.04x - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp reservation none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp volsize 300G local OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp volblocksize 16K - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp checksum on default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp compression lz4 inherited from OVERVOLT OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp readonly off default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp createtxg 339 - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp copies 1 inherited from OVERVOLT/MyUbuntuVM OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp refreservation 305G local OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp guid 3020975354493221296 - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp primarycache all default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp secondarycache all default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp usedbysnapshots 0B - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp usedbydataset 182G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp usedbychildren 0B - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp usedbyrefreservation 122G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp logbias latency default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp objsetid 102 - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp dedup off default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp mlslabel none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp sync standard default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp refcompressratio 1.04x - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp written 182G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp logicalused 190G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp logicalreferenced 190G - OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp volmode default default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp snapshot_limit none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp snapshot_count none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp snapdev hidden default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp context none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp fscontext none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp defcontext none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp rootcontext none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp redundant_metadata all default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp encryption off default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp keylocation none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp keyformat none default OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp pbkdf2iters 0 default

 

[Arwen]

Simply un-set the "refreservation" and remove the old snapshots. Then, create a new snapshot. Here is how to remove the old "refreservation";
zfs set refreservation=none OVERVOLT/MyUbuntuVM/MyUbuntuVM-xo4swp

I do highly recommend learning about ZFS if you intend to continue using ZFS. It's different enough from anything that came before, it will take time to pick things up.

A few years ago I had a co-worker who thought he knew enough about snapshots. And he did, VMWare VM snapshots, Linux LVM snapshots. But, he kept mis-understanding on how ZFS snapshots work. Those others work by acting like a transaction log. ZFS does not, it simply uses new space for changes. Thus, to delete a ZFS snapshot causes ZFS to free up space, incrementally. Those others have a tremendous disk I/O load applying those file system transactions to the base file system.

I describe that example to give you an idea of the differences. Plus, as an example even a professional Unix SysAdmin still had trouble understanding the differences.

 

[Phase]

Awesome. It is slow (270 Mbs based on network traffic) but it is doing it! Thank you!!

I googled for the reservation implication on snapshots and found this https://docs.oracle.com/cd/E19253-01/819-5461/gbdbb/index.html

If refreservation is set, a snapshot is only allowed if sufficient unreserved pool space exists outside of this reservation to accommodate the current number of referenced bytes in the dataset.

I don't understand why. Is it duplicating the information? Why would it? Thoughts?

 

[Arwen]

If I understand it correctly, (and it's entirely possible I don't), a "reservation" reserves space for a dataset or zVol. But, the "refreservation" is for snapshots and possible clones.

ZFS snapshots work as a copy on write. So a snapshot takes very little space when created. As the source dataset or zVol changes, those changes get written to new, free space on the pool. The snapshot blocks remains static, unchanged.

Another way to look at a ZFS snapshot is that it is a hard link to the original source dataset or zVol. But, any change splits the hard link, leaving the copy from the snapshot alone as R/O. And the change is written to free space.


As for your question about duplicating the information. No, ZFS does not duplicate information. But, ZFS can appear to do so as directory entries, (aka metadata), do have to be updated on any change. Thus, use more disk space as part of the copy on write.

ZFS takes the approach that metadata, (like directory entries), is more important that file data. This is because if you loose a directory entry that includes all the blocks assigned to a file, you loose the file data's proper sequence. The blocks may still be known to have been assigned to some file, but without the known order they should be in, the file can be considered useless.

This is why ZFS uses copy on write, leaving the old metadata, (for any snapshots or clones), alone, and any changes get written to new, free space on the pool. So, in theory, no "fsck" / file system check is ever needed with ZFS.

That's probably too much information...

 

[Phase]

That is how I thought it worked, but there seems to be some complications, with refreservation -- maybe it is that reservation must also be set in order to have a snapshot. The language is confusing: "If refreservation is set, a snapshot is only allowed if sufficient unreserved pool space exists outside of this reservation to accommodate the current number of referenced bytes in the dataset."

Regardless I'm gonna leave it alone as I got my backup, and we are not using refreservation anymore :)

Thanks!!