Backup and Restore datasets, quotas, ...

[DerJan]

I'm working on the backup concept for my infrastructure and there is a FreeNAS box among them which is used as fileserver. The backup is currently being done using bacula to encrypted hard drives which are stored externally. The FreeNAS box also has a file-daemon running and hence, all the data is already properly backed up. Furthermore there is also backups of the FreeNAS configuration.

I'm now simulating a full system failure (think theft or a server room fire) where the FreeNAS server and all the hard disks are gone for good. My original plan was:

1. Install fresh FreeNAS on new hardware
2. Restore backup configuration
3. Get bacula-fd running again from backups
4. Use bacula to restore all data

The main issue I encountered is in step 2) here. Restoring the old configuration leaves the pool untouched. While this is usually desired it seems like there is a big gap in FreeNAS here. I've not yet found a way to backup my pool configuration so that I can restore it later to a new pool and essentially get all my datasets, quotas, ACL settings back in an otherwise essentially empty pool. This appears to be a major problem when doing disaster recovery of a FreeNAS box.

The only solution I've found so far is to use zpool history and some grepping to filter out all the periodic snapshots, scrubs and other maintenance stuff that piles up. I could then, after some manual screening, just run the series of commands on the new pool. Afterwards, restoring from backup and doing the following steps seems to work ok.

However, that solution seems to be very hacky and probably also fragile. So I'm wondering if there is a better way. It appears to be quite a big missing feature if recovering from such a disaster essentially requires re-creating all datasets by hand.

 

[Arwen]

Yes, I ran across the Zpool and ZFS attribute backup issue. Those people using ZFS send
and receive get that taken care of as part of the process. But, I am using Rsync, and you
are using Bacula.

My solution was to run the following and re-direct the output to the backup media, (or
location that get's backed up);

Code:

zpool list
zpool status
zpool get all

zfs list
zfs get all

Not perfect. But, I would then beable to re-create all the pools and datasets perfectly.
Plus, this can be automated, (though I have not yet done so).

 

[DerJan]

Yes, automating this does not seem to be the issue. However, it seems to me that your approach does seem to be missing at least userquotas.

[root@strg] ~# zfs userspace Storage/timemachine
TYPE NAME USED QUOTA
POSIX User <user1> 107G 500G
POSIX User <user2> 13.0G 500G
...

(For the people who come here while searching for timemachine: Do not try to use userquotas to limit the space timemachine can utilize per user as shown above. Turns out, it does not work. Use individual shares for each user and set a refquota, that works)

However:

[root@strg] ~# zfs get all Storage/timemachine
NAME PROPERTY VALUE SOURCE
Storage/timemachine type filesystem -
Storage/timemachine creation Wed Sep 3 11:14 2014 -
Storage/timemachine used 1.67T -
Storage/timemachine available 11.6T -
Storage/timemachine referenced 1.43T -
Storage/timemachine compressratio 1.35x -
Storage/timemachine mounted yes -
Storage/timemachine quota none default
Storage/timemachine reservation none default
Storage/timemachine recordsize 128K default
Storage/timemachine mountpoint /mnt/Storage/timemachine default
Storage/timemachine sharenfs off default
Storage/timemachine checksum on default
Storage/timemachine compression lz4 local
Storage/timemachine atime on default
Storage/timemachine devices on default
Storage/timemachine exec on default
Storage/timemachine setuid on default
Storage/timemachine readonly off default
Storage/timemachine jailed off default
Storage/timemachine snapdir hidden default
Storage/timemachine aclmode passthrough inherited from Storage
Storage/timemachine aclinherit passthrough inherited from Storage
Storage/timemachine canmount on default
Storage/timemachine xattr off temporary
Storage/timemachine copies 1 default
Storage/timemachine version 5 -
Storage/timemachine utf8only off -
Storage/timemachine normalization none -
Storage/timemachine casesensitivity sensitive -
Storage/timemachine vscan off default
Storage/timemachine nbmand off default
Storage/timemachine sharesmb off default
Storage/timemachine refquota none default
Storage/timemachine refreservation none default
Storage/timemachine primarycache all default
Storage/timemachine secondarycache all default
Storage/timemachine usedbysnapshots 0 -
Storage/timemachine usedbydataset 1.43T -
Storage/timemachine usedbychildren 246G -
Storage/timemachine usedbyrefreservation 0 -
Storage/timemachine logbias latency default
Storage/timemachine dedup off default
Storage/timemachine mlslabel -
Storage/timemachine sync standard default
Storage/timemachine refcompressratio 1.39x -
Storage/timemachine written 1.43T -
Storage/timemachine logicalused 2.18T -
Storage/timemachine logicalreferenced 1.93T -
Storage/timemachine volmode default default
Storage/timemachine filesystem_limit none default
Storage/timemachine snapshot_limit none default
Storage/timemachine filesystem_count none default
Storage/timemachine snapshot_count none default
Storage/timemachine redundant_metadata all default#

While the zpool history seems to capture those as well.

[root@strg] ~# zpool history Storage | grep timemachine | grep -v "@auto"
2014-09-03.11:14:26 zfs create -o compression=lz4 Storage/timemachine
2014-09-03.12:29:34 zfs set userquota@<user1>=500G Storage/timemachine
2014-09-26.17:26:55 zfs set userquota@<user2>=500G Storage/timemachine
...

The good thing about history is that I can just paste it to a shell and wait. However, it might contain a lot of old datasets that were later deleted. I am still a bit confused that such a big topic for file servers seems to be a blind spot. Not just for FreeNAS but for ZFS as a whole.

 

[Arwen]

Hmmm, never knew about "userspace" and "groupspace", since many earlier Solaris 10
instances don't have them. Oh well, learn something new every day.

As for "zpool history", I'd prefer to save the state of my pool and datasets as they exist at
the time of the backup.

You are right in that it would be nice if FreeNAS kept a disaster recovery command file to
re-create the pool and datasets exactly as they are at the time of configuration saves. This
would be a nice RFE, (Request For Enhancement), for FreeNAS.

It would not be perfect, since some disaster recoveries end up with slightly different
hardware. Meaning an original NAS of 4 x 4TB disks might morph into 5 x 6TB disks when
performing the recovery. So, with a command file you could pick and choose which
commands to run on the the new system. In the above example, you manually create the
pool, but may copy and paste all the pool attribute settings, and the all the "zfs" commands.