Weird problem with snapshots of 200gb .VHD files (Windows Image Backup)

[carl0s]

Hi. I am having some strange problems.

I have a Windows Sever 2008 r2 box doing a wbadmin backup to \\freenas\server
"server" is both a CIFS share, and a zfs dataset.

ZFS dataset is configured to do periodic snapshots.

The snapshots are taking place, but the ~227gb .VHD file, although showing up in each of the snapshots with the correct modified date/time, looks to be the same file - the size never changes. Although when I mount the file, it looks like the latest backup of the system. I am unable to mount the file from an older snapshot right now, but I think that's because the snapshot is read-only - I am in the process of copying to the Desktop to look at it from there.

In the list of snapshots in the FreeNAS gui, I can see that the "Used" column shows basically 0, or in some cases a few hundred kb, but never the ~227gb I am expecting.

Actually, the latest backups should be closer to 160gb now since some data was removed from this system, yet the files are still showing at 227gb.

It's weird. Any ideas what's going on?

To slightly confuse matters, the "server" share is actually accessible through a parent share, i.e. it is mounted as a subfolder of a different share, but that share does not have recursive snapshots enabled, and I am accessing the server share by its own CIFS share, which is how I can see the snapshots. i.e. :

\\freenas\NAPP = CIFS share & ZFS dataset.
\\freenas\NAPP\Server = shared separately & also a separate ZFS dataset. I am browsing it via \\freenas\Server - the separate share, so that the snapshots show up.

(see below for what I mean):

 

[carl0s]

Hmm. This looks a bit more promising:

Code:

[root@freenas] /var/log# zfs list -o space
NAME                        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
SATAStorage/NAPP            2.11T   490G     23.5G    106G              0       361G
SATAStorage/NAPP/SERVER     2.11T   361G      144G    217G              0          0
[root@freenas] /var/log# 

(I am expecting to see ~200gb old snapshot, and ~160gb latest), which looks close from those numbers.

 

[carl0s]

OK after moving one of the ~227gb .VHD files from the .zfs directory to a read-write directory, and mounting it, I can see that the snapshots are working fine.

However the filesizes are weird. simple ls -l shows what can only be the wrong file size - old and new .vhd files are identical size, which can't be possible.

the filesystem reports these wrong sizes: ls -l
and the FreeNAS gui totally reports invalid data - 0kb snapshot sizes and stuff (as above).

 

[JaimieV]

I may be off base here, but if .vhd files are like VMware's .vmdk disk files then they will not shrink by themselves - they'll expand to contain all the virtual machine's data, but you need to run a manual process to shrink them back down again. So it's perfectly possible for the old and new .vhd's to be the same size.

 

[cyberjock]

I may be off base here, but if .vhd files are like VMware's .vmdk disk files then they will not shrink by themselves - they'll expand to contain all the virtual machine's data, but you need to run a manual process to shrink them back down again. So it's perfectly possible for the old and new .vhd's to be the same size.

You are 100% correct JaimieV. I'm not sure if that's his problem though. As soon as a thread starts talking about snapshots and disk space I tune out completely. I don't know much about snapshots(my one experiment with them failed) but I also know that there are a bunch of people always complaining about snapshots and disk space used/free. 99% of them always end with the OP not understanding how snapshots work. If I knew someone in the forum that understands snapshots thoroughly I'd love to put together a presentation on it so that we can stop dealing with snapshot and disk space issues every few days :P

 

[carl0s]

You are 100% correct JaimieV. I'm not sure if that's his problem though. As soon as a thread starts talking about snapshots and disk space I tune out completely. I don't know much about snapshots(my one experiment with them failed) but I also know that there are a bunch of people always complaining about snapshots and disk space used/free. 99% of them always end with the OP not understanding how snapshots work. If I knew someone in the forum that understands snapshots thoroughly I'd love to put together a presentation on it so that we can stop dealing with snapshot and disk space issues every few days :P

There is a presentation, or at least a general ZFS one anyway, and I have been through it a little while ago.

I think JamieV may be onto something with the .vhd sizes, which is great (ish), since they are the same as Virtual-PC disk images, although I will Google it to check. In this instance I am simply running a Windows Backup of the C: drive, and as far as I know, this should certainly shrink. I've definitely never had to shrink or defrag NTFS volumes in the past for the purpose of backup image file sizes, and I use wbadmin to back up to RDX drives at quite a lot of sites. I'll look into it anyway.

The other part of my query though is why the GUI shows miniscule snapshot sizes, when I can clearly see through the .zfs directories that the snapshots are large. Also, the zfs list -o space shows something more like what I was expecting, but doesn't match up with either the FreeNAS GUI (showing roughly zero), or the filesystem (showing multiple 227gb files). If I was using de-duplication or something then I could understand it.. but I'm not, or I'm not supposed to be anyway.

I suppose the good news is that I was able to utilise the backup file from the snapshot (after moving to a r/w location because Windows won't mount read-only), and that means that the system is ulimately doing what I want it to.

 

[cyberjock]

There is a presentation, or at least a general ZFS one anyway, and I have been through it a little while ago.

I know.. I wrote it :)

I have a background in training individuals with basic skills, which came in handy writing that presentation.

 

[carl0s]

I may be off base here, but if .vhd files are like VMware's .vmdk disk files then they will not shrink by themselves - they'll expand to contain all the virtual machine's data, but you need to run a manual process to shrink them back down again. So it's perfectly possible for the old and new .vhd's to be the same size.

You are right about the .VHD files. (http://social.technet.microsoft.com...p/thread/3ccdcb6c-e26a-4577-ad4b-f31f9cef5dd7 )

They can be compacted after they are created, although the compacting only really works on zeroed data. I don't see any way to "compact" an actual physical disk volume, which is where the .VHD files are created from every time a wbadmin backup is run.

Hmmm. That is a pain. I need to somehow zero and deallocate unused sectors of the drive or something. It's wasting ~60gb per day, which is retained for ~2weeks, so quite a lot of wasted space.

Anyway, thanks for the help ;)

 

[carl0s]

I know.. I wrote it :)

I have a background in training individuals with basic skills, which came in handy writing that presentation.

I did wonder if you had your sarcasm mode on.. :D

Thanks for the presentation. I will look over it again, although I think I understand the concept reasonably well.

 

[carl0s]

Oh, you weren't being sarcastic, sorry ;)

(I just looked at your presentation again and it doesn't go into snapshots).

 

[cyberjock]

Nope. I really did write it. That's why its in my sig. :)

And I really would like to do a presentation on Snapshots :P

 

[carl0s]

Nope. I really did write it. That's why its in my sig. :)

And I really would like to do a presentation on Snapshots :P

LOL yes I know you wrote it, I just meant when you said "one day I'd like to write a presentation on snapshots, to stop all these questions"... then i realised you wrote the presentation on ZFS, so I thought you were being sarcastic about wanting to write a presentation...

Anyway nevermind :)

 

[bollar]

Snapshots are almost magical. I have tested snapshots extensively over the past few weeks and some of the questions you're coming across don't surprise me.

- First off, snapshots are block-based, so the size of the snapshot really only reflects the number of blocks changed, not the entire size of the file. if one of your VHD files only has a file or two added to it, the snapshot usually only requires tracking the blocks that have changes. If there have been no changes, the snapshot will have a size of -0-

- Second, I don't know for sure if the file sizes reported by ZFS for snapshots are correct. I think they are, but there is a lot of stuff going on under the hood that's not exposed to the user.

- Third you can view the files within a snapshot by "cloning" it. This will make a parallel directory with the state of the system as of the time of the snapshot. From there, you can copy out the files you want, or even delete the entire directory without harming your production files. If you want to replace the production system, you'll need to use "rollback" and be aware that it's destructive.

I'm very impressed with how snapshots (and replication) work and really for the first time, it's been feasible for me to offer a failover server without significant expense and configuration. The main downside from my perspective is that you don't have much control over how the process works, without investing in an enterprise-type backup solution.

 

[bollar]

...but I also know that there are a bunch of people always complaining about snapshots and disk space used/free. 99% of them always end with the OP not understanding how snapshots work.

Snapshots are like a creeping crud if you're not careful. In my first build, I did snapshots every 15 minutes on the entire NAS. Because of all of the mucking around I was doing, in a week, I had 12,000 snaps and about a 1/2 TB of data within them. There's no way to fix that problem from within FreeNAS and it requires some command line deletion work -- and since those files are hidden it's not straightforward to most users. BTW, if someone does want to delete all snapshots from their Zpool, here's a good script:

Code:

zfs list -H -o name -t snapshot | xargs -n1 zfs destroy

 

[cyberjock]

BTW, if someone does want to delete all snapshots from their Zpool, here's a good script:

Code:

zfs list -H -o name -t snapshot | xargs -n1 zfs destroy

Can you say "KABOOM!"

 

[carl0s]

Snapshots are almost magical. I have tested snapshots extensively over the past few weeks and some of the questions you're coming across don't surprise me.

- First off, snapshots are block-based, so the size of the snapshot really only reflects the number of blocks changed, not the entire size of the file. if one of your VHD files only has a file or two added to it, the snapshot usually only requires tracking the blocks that have changes. If there have been no changes, the snapshot will have a size of -0-

Ah, now that's interesting.
I actually did write "hang on, the snapshots are file based, not block based, right?", but then I deleted the post because I didn't want to sound stupid, and thought that I was confusing the idea of de-duplication.
I just can't see how ZFS can say that only a small block has changed, when the client is overwriting a whole 227gb file via CIFS, without de-duplication or something .. ?

 

[bollar]

Ah, now that's interesting.
I actually did write "hang on, the snapshots are file based, not block based, right?", but then I deleted the post because I didn't want to sound stupid, and thought that I was confusing the idea of de-duplication.
I just can't see how ZFS can say that only a small block has changed, when the client is overwriting a whole 227gb file via CIFS, without de-duplication or something .. ?

I did say it's magical...

I have been looking for the definitive explanation, but haven't come across it. My understanding is that ZFS maintains a checksum of each block and as data is coming in, ZFS compares checksums and only writes modified data. The old blocks are what are added to the snapshot.

This Oracle white paper covers how copy on write works and some other storage topics: Oracle Solaris ZFS Storage Management

 

[Stephens]

Let's say you have a file (F1) comprised of 4 blocks (F1B1 .. F1B4) and they have hashes of FB1H .. F1B4H? Now let's say you get ready to overwrite the file and start sending FreeNAS/ZFS the data. So it buffers up (it doesn't write a byte at a time) and finally you have the whole block 1 (F1B1). Let's say ZFS has been able to create a hash of that incoming data and knows it's exactly the same as the hash it already has for that block (part of metadata). No need to overwrite it. Same for F1B2 and F1B3. But you put a period at the end of your document, so now B4 has changed. ZFS doesn't overwrite F1B4.. Instead, it creates F1B4', creates a hash F1B4H' (I'm trying to keep it simple). For one version of the file, it'll point to F1B4. For another version of the file, it'll point to F1B4'. There are a lot of details involved in what ZFS does, but it does work without deduplication. And it does work at the block level.

 

[carl0s]

That's really interesting, thanks.

And if it means that a ZFS "diff" is really so small, it means offsite replication would work much better than I thought, considering the size of the file. I expected that basically the whole file would be seen as "different", so a ~220gb diff each day, but I guess not.

Hmmm. So VHD / wbadmin images & ZFS seem to play really nicely together :)

 

[cyberjock]

ZFS deduplication is block level, not file level. So you can have identical files that are NOT deduplicated. Also, SOME programs will screw up how Stephens explains it. He is technically correct in all respects from a ZFS standpoint. But some programs will actually create a new temporary file, write that whole file, then delete the "old" file and rename the temp. In this case, it would be completely different. But I think you can understand what would happen then.

Edit: Keep in mind that using a VHD file on your ZFS system can result in very poor performance over the long term. This is documented in a support ticket for iscsi at support.freenas.org. The same mechanism will degrade your VHD file due to excessive fragmentation of the file because of the copy-on-write characteristics of ZFS.