Snapshots not working

[mo100]

Hi,

I have looked around and cannot find anyone with a similar issue. My snapshots, as far as I can tell, do not work at all. They never have since installation. Currently I am using Freenas 8 RC1.

I have a periodic snapshot schedule set, which does not create any snapshots. Also if I manually create a snapshot then although the gui states "snapshot succesfully taken", I can not see any snapshots within the ZFS snapshot tab. It is an empty list.

Is this a hardware issue? (i.e. not enough memory). Are snapshots turned off by default and I need to turn them on via the command line? Are there any commands I can use to work out why snapshots are not working?

I would appreciate any help.
Thanks,
Mark

 

[mo100]

Anyone??? Am I the only one?

Are there any commands I can use on the command line to work this out?

Thanks.

 

[ProtoSD]

Currently I am using Freenas 8 RC1

I think it's an issue with FreeNAS 8, try upgrading to 8.01. I have snapshots setup and they work great.

 

[mikeyr]

Snapshots not working here either...

I just set up snapshots and can't seem to get it to make any. I'm running Freenas 8.0.1-RELEASE.
I set it up to make a snapshot every 15 minutes and waited for 25 minutes and still don't see any snapshots listed under the ZFS Snapshots tab...
Seems to be working for other people, don't know what I'm doing wrong, I guess I'll just have to play around with it some more and see if I can figure out what wrong

 

[ProtoSD]

8.02 just came out, not any snapshot improvements from 8.01 that I'm aware of, but there are other improvements.

Not sure what you could be doing wrong, it seems pretty straight forward. Sometimes the interface has trouble with certain browsers and doesn't display properly, maybe try that.

 

[mikeyr]

More detail on my snapshot woes:
I tried making snapshots via the WebGUI interface, and also via zfs snapshot command at the CLI.
I can ssh in as root and do #zpool history and see that is thinks it did the snapshots.

[root@freenas] ~# zpool status -v
pool: RaidArray
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
RaidArray ONLINE 0 0 0
mirror ONLINE 0 0 0
gpt/ada0 ONLINE 0 0 0
gpt/ada1 ONLINE 0 0 0

errors: No known data errors
[root@freenas] ~# zpool history
History for 'RaidArray':
2011-08-20.00:27:06 zpool create -o cachefile=/data/zfs/zpool.cache -fm /mnt/Rai dArray RaidArray mirror /dev/gpt/ada0 /dev/gpt/ada1
2011-08-20.00:28:44 zfs create RaidArray/Marvin
2011-08-20.00:29:37 zfs inherit compression RaidArray/Marvin
2011-08-20.00:29:37 zfs inherit atime RaidArray/Marvin
2011-08-20.00:29:38 zfs set reservation=0 RaidArray/Marvin
2011-08-20.00:29:38 zfs set refreservation=0 RaidArray/Marvin
2011-08-20.00:29:38 zfs set quota=none RaidArray/Marvin
2011-08-20.00:29:38 zfs set refquota=none RaidArray/Marvin
2011-08-20.00:29:48 zfs destroy RaidArray/Marvin
2011-08-20.00:30:06 zfs create RaidArray/Camarillo
2011-08-20.01:13:56 zpool import -o cachefile=none -f RaidArray
...
2011-09-18.10:35:43 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-09-19.03:01:29 zpool scrub RaidArray
2011-09-26.21:36:56 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-03.20:57:54 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-10.13:51:02 zfs snapshot RaidArray/Camarillo@auto-20111010.1351-2d
2011-10-11.22:55:07 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-12.02:00:02 zfs snapshot RaidArray/Camarillo@auto-20111012.0200-2d
2011-10-12.13:51:02 zfs destroy RaidArray/Camarillo@auto-20111010.1351-2d
2011-10-12.23:36:14 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-13.02:00:02 zfs snapshot RaidArray/Camarillo@auto-20111013.0200-2d
2011-10-14.02:00:02 zfs snapshot RaidArray/Camarillo@auto-20111014.0200-2d
2011-10-14.02:00:02 zfs destroy RaidArray/Camarillo@auto-20111012.0200-2d
2011-10-14.08:14:38 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-14.09:29:51 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-14.09:37:12 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-14.22:24:52 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.16:15:40 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.16:43:31 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.16:50:54 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.17:16:01 zpool import -R /mnt RaidArray
2011-10-15.17:16:02 zpool export RaidArray
2011-10-15.17:17:41 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.17:17:42 zpool set cachefile=/data/zfs/zpool.cache RaidArray
2011-10-15.18:01:02 zfs snapshot -r RaidArray/Camarillo@auto-20111015.1801-1w
2011-10-15.18:01:02 zfs destroy RaidArray/Camarillo@auto-20111013.0200-2d
2011-10-15.18:37:12 zpool import -o cachefile=none -R /mnt -f RaidArray
2011-10-15.18:37:13 zpool set cachefile=/data/zfs/zpool.cache RaidArray
2011-10-15.19:00:02 zfs snapshot -r -o freenas:state=NEW RaidArray/Camarillo@auto-20111015.1900-1w
2011-10-15.19:15:02 zfs snapshot -r -o freenas:state=NEW RaidArray/Camarillo@auto-20111015.1915-1w
2011-10-15.19:30:02 zfs snapshot -r -o freenas:state=NEW RaidArray/Camarillo@auto-20111015.1930-1w
2011-10-15.19:45:02 zfs snapshot -r -o freenas:state=NEW RaidArray/Camarillo@auto-20111015.1945-1w
2011-10-15.20:00:02 zfs snapshot -r -o freenas:state=NEW RaidArray/Camarillo@auto-20111015.2000-1w
2011-10-15.22:09:10 zfs snapshot RaidArray/Camarillo@testsnap
2011-10-15.22:44:26 zfs snapshot RaidArray@arraysnap

But then when I try to list to see the snapshots nothing is visible...

[root@freenas] ~# zfs list -r
NAME USED AVAIL REFER MOUNTPOINT
RaidArray 416G 1.38T 23K /mnt/RaidArray
RaidArray/Camarillo 416G 1.38T 416G /mnt/RaidArray/Camarillo
[root@freenas] ~#

My understanding is that the zfs list -r should show all the snapshots available.

Any thoughts about what's going on here? Does it have anything to do with the fact that I've only got 3GB of ram? Any other ideas what I can try or look at?
Thanks!

 

[mikeyr]

Uhhh,what just happened? I had given up on snapshots and was starting to figure out how to RSYNC my boxes... I went out for the afternoon (having left the periodic snapshot task set up-- forgot to delete it); and when I got home and logged into the WebGUI, voila!

There were all the snapshots that had been taken over the last 2 days of (seemingly unsuccessful) attempts and making snapshots... Why did they suddenly appear in the WebGUI? (Same browser, didn't reboot the machine I am browsing from, I'm pretty sure I didn't even reboot the FreeNAS box...)

I guess maybe I'll proceed with trying to debug replication next.

but I'm worried about counting on the snapshots-- if they can "magically" start working; then they can "magically" stop working again.

Anybody have any clues? I don't even know what logs or output I could be posting to help debug this...

And BTW-- what is the proper CLI command to list the snapshots? I thought it would be #zfs list -- but that still doesn't show any snapshots even though the GUI now does show them.

 

[ProtoSD]

And BTW-- what is the proper CLI command to list the snapshots? I thought it would be #zfs list -- but that still doesn't show any snapshots even though the GUI now does show them.

zfs list -t snapshot (shows a list of all snapshots)

 

[mikeyr]

OK, sorry to keep answering my own questions, but maybe someone else out there will be able to learn something from my Newbie Q&A.
I figured out that I can see the snapshots from the CLI by using

#zfs list -t snapshot

Some references I had read suggested that the "zfs list" would show snapshots, but apparently it doesn't unless the "-t snapshot" is added.

I think I have replication working, both boxes show messages on the console that look like something is happening, and both interfaces have been running maxed out (local source is max TX, remote reciever is max RX). I don't know how to check the status of the replication, so I am assuming that for the first snapshot it needs to replicate the entire 500 GB data set and this will take a while. I'm going to bed, will check in the morning if I have a replicated data set and post my results!

 

[mikeyr]

Status update: Both machines ran at full network bandwidth for about 8-10 hours. I never saw any completion message or status update; but the remote machine now lists all the snapshots; and the disk space shown as used on REMOTE corresponds with how much data was on the local machine-- so it looks like it made a replicated filesystem + snapshots on the remote machine.

Success! I think...

Now I need to figure out how to mount and/or browse through the replicated system just to make sure it is _actually there_. Some web pages (not FreeNAS, but general ZFS related) mentioned making sure to mount snapshots as read-only or simply the process of cd'ing to the snapshot can mess up some meta-data or something... not sure if this applies to FreeNAS or not.

I'm also a little mystified why all the snapshots made "accidentally" got replicated as well as the most recent snapshot that triggered the replication to run... but I really don't understand what going on "under the hood" with replication and snapshots. I assume if I did it would all make sense.

 

[mikeyr]

New updates:
I can ssh into the "remote" machine and use 'ls ' to browse through the remote filesystem; all the replicated data is there, so it looks good!

I have noticed that the "snapshots" page on the GUI doesn't always display properly. Yesterday I had a whole long list of snapshots that were created when I was experimenting; and they all displayed properly in the GUI. I deleted most of them and have since created a few more. Now only about half of the list shows up, it looks a lot like the bottom half of the snapshot display is somehow truncated.

In IE8, the snapshots do not display at all in the GUI. In Firefox, some of the snapshots display, but the bottom half of the list seems to be missing for some reason.

If I ssh into the "local" machine and do 'zfs list -t snapshot' then I see a list of all the snapshots.

The FreeNAS documentation on setting up replication didn't work too well for me; there were a number of small errors I had to work through. I will try to post my experience and what I did differently to get it working soon. (Mostly to do with setting up key based authentication; and also to get the paths and volumes set up correctly in replication...)

 

[mikeyr]

I just posted, but my post seems to be missing... so I'll repost and hope it doesn't double up somewhere...

Anyway, I was able to ssh into the remote machine and use 'ls' to browse the replicated filesystem. All my data is there, so replication looks good!

Broswer display issues seem to be part of my snapshot confusion. Yesterday I had a page full of snapshots displaying on the GUI on my local machine (created during experimentation with snapshots). I deleted most of them, and have since created just a couple of more. Only 3 of the 5 snapshots I have now show up in the GUI interface (using Firefox). If I use IE8 _none_ of the snapshots show up. It appears like the display is somehow truncated before the entire list gets rendered...? The display is similarly truncated on both the local machine and the remote machine.

I can ssh into the both machines and use 'zfs list -t snapshot' and see that all the snapshots are there, so I know the snapshots are working; they just don't display properly. I haven't tried upgrading to 8.0.2 yet, maybe that will fix it...

 

[mo100]

Just in response to my original post (in case it helps anyone else). After upgrading to recent FreeNAS 8.0.2 the snapshots would still not work. I had hit the cmd line and could see that nothing was happening. In the end I destroyed the dataset (via freenas gui) and started again. After I had setup a new zfs dataset (via freenas gui), I found that my snapshots were working without any problem. I don't know why it originally wouldn't work, but I suspect a permissions problem. Perhaps I had clicked a tick box where I shouldn't have.

/Mark.

 

[Durkatlon]

Now I need to figure out how to mount and/or browse through the replicated system just to make sure it is _actually there_.

In the root of your volume (or dataset if you are using those), there is a hidden directory called ".zfs". You can ssh into the machine and "cd" into this directory. Note that it will not be listed by "ls" even if you add the option to list hidden directories, but it'll be there when you change into it.

This directory will have subdirectories for all your snapshots, and if you go into one, you'll see the snapshot's "view" of the volume or dataset.

 

[bobrebelbob]

it has been about a month- what are you results with syncing two like boxes?

I am certainly a noob at this and followed your notes with interest. I built two 6tb servers (4x2TB drives in Raid5) that are just alike. I was hoping that some bright soul would put out a step by step HOWTO on either 'snapshot' or 'RSYNC' method to back up on FreeNAS 8 box to another one. I have been concerned about placing my important data on such a huge data store and not having it backed up. Also, I have been 'burning in' my boxes for 3 months now. I had one drive fail very early on and have the replacement in place and spinning. Sleeping did not work at first but they got that working and a few other bugs were fixed along the way. I am not just exercising the drives by copying 900 meg data store from one box to the other (over night) but I am not depending on the drives to safely hold all my data YET!


SO, how's is FreeNAS performing for you now and especially the backup process between boxes??

Thanks for anything you can contribute to the education of the community,

Bob

 

[cucu007]

Status update: Both machines ran at full network bandwidth for about 8-10 hours. I never saw any completion message or status update; but the remote machine now lists all the snapshots; and the disk space shown as used on REMOTE corresponds with how much data was on the local machine-- so it looks like it made a replicated filesystem + snapshots on the remote machine.

Success! I think...

Now I need to figure out how to mount and/or browse through the replicated system just to make sure it is _actually there_. Some web pages (not FreeNAS, but general ZFS related) mentioned making sure to mount snapshots as read-only or simply the process of cd'ing to the snapshot can mess up some meta-data or something... not sure if this applies to FreeNAS or not.

I'm also a little mystified why all the snapshots made "accidentally" got replicated as well as the most recent snapshot that triggered the replication to run... but I really don't understand what going on "under the hood" with replication and snapshots. I assume if I did it would all make sense.

Sounds like a good accomplishment, maybe you can share with others like me what you did to get this going. I am still in a very early state integrating FreeNAS 8.2 into our network. I have two boxes currently running for about 3 weeks, it looks rock solid. Both boxes have about 40Tb of space using a 4Gbs channel to transfer data across each other. I am still doing reading to get this going. The documentation is good, but it can actually explain things a bit further. I guess we will get there someday. Share what you did if you have a minute with us.