Please validate my backup plan: Rotating offsite backup disks, from single FreeNAS primary storage

[dztrbd]

Thanks so much, Finally got it to work. That must have been painful for ya. :P I tried to copy paste but the IMPI wouldn't let me do it. And i need the typing practice, But now i know that the forum does that it will be handy to know.

 

[Michael Wulff Nielsen]

Might I suggest using the crashplan plugin, it really makes off-site backup a breeze, and it is very affordable. A normal setup is to have your machines backup to freenas and freenas to the cloud.

(Disclaimer I am just a happy crashplan customer, not affiliated with them in any way)

 

[mattyakel]

@MichaelWulffNielsen Exactly, How? that's what I want. I am a CrashPlan customer as well (very happy with their service). But now that I have moved to FreeNAS I need to get it all running right. I have 1 Linux laptop, 1 Mac Laptop, 1 Linux Desktop. Need to back them up to the FreeNAS and then back up the FreeNAS to my CrashPlan account for the offsite. How does this work with the jails? Ex: I have /mnt/tank as my main ZFS share so /mnt/tank/Photos needs to be backed up by the CrashPlan plugin/jail. Are there any how-tos for this scenario? Is this how you have yours set? I will gladly discuss Ideas Thoughts Processes and share more of my details. If I can get it running I'd like to do a blog post unless someone already has.

 

[Michael Wulff Nielsen]

You install the Crashplan plugin and add your zpool to the jail (mine is in readonly mode).

Then you configure crashplan to upload everything using the headless guide

After that you just wait for crashplan to upload everything from your freenas to their servers :)

 

[panz]

Is your current directory writable?

The geli backup syntax is "geli backup <provider> <filename>". If you did not change the FreeNAS default the provider is the second partition on the disk (the first partition is swap). So, to backup ada1 metadata to a geli_ada1 file you would run "geli backup /dev/ada1p2 geli_ada1". My script tries to use the serial numbers of the drives as the file names. I did consider using the GPTIDs but I want to be able to match the backups to physical drives even in case the partition tables get corrupted too. The partition table is easy to recreate, but you will get new GPTIDs.
I just also noticed that my script will fail if the serial number contains space(s) (my WD REDs' serial numbers don't). This should work better:

Code:

#!/bin/sh
for disk in /dev/ada*p2
do
  geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`".eli
done

I've done a little modification to your script, because my first 2 disks (da0 and da1) are not encrypted. The encrypted pool is da2, da3, da4, da5, da6 and da7; so my script is:

Code:

for i in 2 3 4 5 6 7; do
geli backup da${i}p2 "`camcontrol identify da${i} | grep serial | tr -s \  | cut -d \  -f 3-`".eli
done

Is this correct?

 

[Ef57uiKnN6]

Just played around a bit what resulted in two shell scripts.
One for geli metadata backup and another for restoring metadata.

Highly experimental and not tested at all!

Both scripts expect a FreeNAS volume name to be passed.
The passed volume name is checked for existence and is checked to be an encrypted volume against the FreeNAS configuration database.
The scripts lack of any exception handling. Some conditions are checked though.

The Backup script exports geli metadata for all partitions in parallel, encrypts them on-the-fly and finally creates a tarball out of them.
Encryption stuff removed in elifun-0.3.tar.gz.

For encryption the file name used for geli key (located in configuration database as well) is taken as password - not very secure :)
In elifun-0.2.tar.gz the geli metadata of the corresponding FreeNAS volume is taken as password for encryption.
Due to this change the use of named pipes could be completely eliminated.

It is always tried to export the metadata from the secondary partition. When there is no such, then the first partition is used.

The Restore script is more or less the same vice versa:
- Explode previously created backup/tarball
- Decrypt and restore on-the-fly in parallel

Note: The geli-restore command is commented out for security reasons. Instead a message is written to stdout to indicate whether the final data looks reasonable.

Any comment is welcome, of course. Anyhow, as mentioned before it's just a finger excercise and not well thought-out at all.
So, don't take it too seriously ;)

Argh! Just noticed that I have used some misleading variable and function names. Corrected in elifun-0.2.1.tar.gz.

While correcting this post as well regarding elifun-0.2.1.tar.gz , I notice that there is probably no reason to encrypt meta-data :D
How the hell did I actually thought it's all about backing up a master-key.... However, I had some fun.

And finally elifun-0.3.tar.gz without the useless encryption stuff.

 

[senior_hombre]

I tried your script (copied the last version and called it ge.sh) when calling it with: sh ./ge.sh i get the following error:
./ge.sh: 3: Syntax error: word unexpected

Whats the problem here?

btw: this is the script iam using:

Code:

#!/bin/sh
for disk in /dev/ada*p2
do
   geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`".eli
done

my drives are indeed named /dev/ada0 to /dev/ada5

edit: nevermind i used Ef57uiKnN6s script and it seems to work, thanks

 

[panz]

Did someone try a restore? Without knowing if a drive is faulty or metadata (for that drive) is corrupted is not so easy...

 

[R Bauer]

Where should the metadata backups be kept? Obviously not on the encrypted volume... :)

 

[panz]

Where? An USB key? Amazon S3? Another computer? On your cellphone? On a SD card?

 

[cyberjock]

Someplace safe. Whatever you want to call "safe".

 

[R Bauer]

Someplace safe. Whatever you want to call "safe".

What I meant was is there some place on the server itself? I am not familiar enough with it to know how various data persists (other than the storage volumes). What places, if any, may I stuff a little bit of data? Or is the recommended practice to get it off the server completely?

 

[cyberjock]

No. In particular you don't want it on the server since you'd be wanting it in the event of *serious* server problems.

 

[R Bauer]

No. In particular you don't want it on the server since you'd be wanting it in the event of *serious* server problems.

Thank you for the advice.

 

[looking.promising]

I do incremental replications. I have an automated snapshot task that snapshots the main pool and the snapshot is then replicated to the backup drive (incremental = only changed blocks are transferred).
Of course. I did not yet upgrade to 9.2.0, so the script does direct writes to the config DB. I plan to update it to use the FreeNAS API later. My removable backup pool is named backup, the disk is ada1… ...I run the script after I replace the backup drive and import the pool. It sets the idle timer, some smart options (-a is no longer needed in 9.2.0) and adjusts the scrub (it gets created automatically when you import the pool) schedule to my liking. It then regenerates config files / restarts services so that the changes take effect.

With my backup rotation schedule the system will perform at least one scrub before I swap the drives. This will detect any bitrot and I can redo a full backup if needed...

The one slightly annoying thing with this schema is that when you export/detach a pool (required before you remove it) FreeNAS will discard the related disk settings (SMART extra parameters, spin down timeout, ...) and its scrub schedule. When you connect the other drive you'll get default disk settings and a default scrub will be scheduled. So, you need to redo your changes (if any) every time you swap the drives (I have a script to do that).

Sorry to revive an old thread but my question seemed most appropriate here...

@Dusan, I have two new drives and a traylesss drive bay I plan to use for my backup strategy (like yours). I have the following questions that I'm hoping you can help with:

1) For reference, could you share your incremental, snapshot, and scrub schedules?
2) If you have upgraded to 9.2.x, would you be willing to also share your updated "post mount setting restore" script?
3) Do you use the same backup pool name on both external backup disks in the rotation?
4) Will zsh always "catch up" the rotated backup pool even if it is a month out of sync?

Hopefully these questions are reasonable, make sense, and have straight forward answers. Thanks!

 

[David E]

Just a follow up to this excellent thread, the post with the script earlier that backs up the geli metadata was not correctly backing it up for SLOGs/Caches, here is a slight tweak to capture those as well:

Code:

#!/bin/sh
for disk in /dev/da*p2
do
    geli backup $disk `camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`.eli
done
for disk in /dev/da*p1
do
    filename=`camcontrol identify ${disk%p1} | grep serial | tr -s \  | cut -d \  -f 3-`.eli
    if [ ! -f $filename ]; then
        geli backup $disk $filename
    fi
done

 

[Teeps]

Hello,

I am using an LSI RAID controller flashed with the SATA expander firmware and I get the following:

Code:

[root@freenas] /mnt/ # ll /dev/mfisyspd*
crw-r-----  1 root  operator  0x65 Sep 12 17:09 /dev/mfisyspd0
crw-r-----  1 root  operator  0xc4 Sep 12 17:09 /dev/mfisyspd0p1
crw-r-----  1 root  operator  0xc3 Sep 12 17:10 /dev/mfisyspd0p1.eli
crw-r-----  1 root  operator  0xc6 Sep 12 17:09 /dev/mfisyspd0p2
crw-r-----  1 root  operator  0x66 Sep 12 17:04 /dev/mfisyspd1
crw-r-----  1 root  operator  0x6b Sep 12 17:04 /dev/mfisyspd1p1
crw-r-----  1 root  operator  0xc2 Sep 12 17:10 /dev/mfisyspd1p1.eli
crw-r-----  1 root  operator  0x6c Sep 12 17:04 /dev/mfisyspd1p2
crw-r-----  1 root  operator  0x67 Sep 12 17:04 /dev/mfisyspd2
crw-r-----  1 root  operator  0x6d Sep 12 17:04 /dev/mfisyspd2p1
crw-r-----  1 root  operator  0xc9 Sep 12 17:10 /dev/mfisyspd2p1.eli
crw-r-----  1 root  operator  0x6e Sep 12 17:04 /dev/mfisyspd2p2
crw-r-----  1 root  operator  0x68 Sep 12 17:04 /dev/mfisyspd3
crw-r-----  1 root  operator  0x6f Sep 12 17:04 /dev/mfisyspd3p1
crw-r-----  1 root  operator  0xbf Sep 12 17:10 /dev/mfisyspd3p1.eli
crw-r-----  1 root  operator  0x70 Sep 12 17:04 /dev/mfisyspd3p2
crw-r-----  1 root  operator  0x69 Sep 12 17:04 /dev/mfisyspd4
crw-r-----  1 root  operator  0x71 Sep 12 17:04 /dev/mfisyspd4p1
crw-r-----  1 root  operator  0xc0 Sep 12 17:10 /dev/mfisyspd4p1.eli
crw-r-----  1 root  operator  0x72 Sep 12 17:04 /dev/mfisyspd4p2
crw-r-----  1 root  operator  0x6a Sep 12 17:04 /dev/mfisyspd5
crw-r-----  1 root  operator  0x73 Sep 12 17:04 /dev/mfisyspd5p1
crw-r-----  1 root  operator  0xc1 Sep 12 17:10 /dev/mfisyspd5p1.eli
crw-r-----  1 root  operator  0x74 Sep 12 17:04 /dev/mfisyspd5p2

Code:

# sh bkup
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mfisyspd0 doesn't exist
geli: MD5 hash mismatch: not a geli provider?
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or mfisyspd0p1 doesn't exist
geli: MD5 hash mismatch: not a geli provider?

any ideas?

Thanks for helping a noob! Let me know what else I can provide to clarify.


edit: I can get the serial like so but the tr/cut doesn't seem to like the output

Code:

# camcontrol devlist
<ATA WDC WD60EFRX-68M 0A82>        at scbus0 target 9 lun 0 (pass0)
<ATA OWC Mercury EXTR BBF0>        at scbus0 target 10 lun 0 (pass1)
<ATA WDC WD40EFRX-68W 0A80>        at scbus0 target 11 lun 0 (pass2)
<ATA WDC WD40EFRX-68W 0A80>        at scbus0 target 12 lun 0 (pass3)
<ATA WDC WD40EFRX-68W 0A80>        at scbus0 target 13 lun 0 (pass4)
<ATA WDC WD40EFRX-68W 0A80>        at scbus0 target 14 lun 0 (pass5)
<Kingston DT 101 G2 PMAP>          at scbus10 target 0 lun 0 (pass6,da0)

# camcontrol identify pass0 | grep serial
serial number         WD-WX11D259VYUD

Either way, geli doesn't like it:

Code:

# geli backup pass0 WD-WX11D259VYUD.geli
geli: Cannot open pass0: Operation not permitted.

 

[cyberjock]

Your problem is what exactly? You just post some info and say "any ideas"...


I have an idea.. you are using hardware raid with FreeNAS, which is a "do at your own risk" type of risky behavior. Nobody here supports that configuration.

 

[Ericloewe]

I am using an LSI RAID controller flashed with the SATA expander firmware

What does that even mean? Please use proper terminology, or nobody will be able to help - and most people won't bother to spend their time going back and forth trying to figure out what you mean.

 

[Teeps]

What does that even mean? Please use proper terminology, or nobody will be able to help - and most people won't bother to spend their time going back and forth trying to figure out what you mean.

Please excuse my imprecise brevity, it was extenuating circumstances that led to my pushing the post button without all the final details detailed out. I'm using the very popular IBM M1015 card cross-flashed to the LSI9211 HBA 6.0gbps SAS / SATA III adapter firmware as detailed, for example, by this post: http://www.servethehome.com/ibm-serveraid-m1015-part-4/

Given the popularity of this solution, I was hoping that someone might have run across this issue as well with attempting to backup the GELI key for the individual drives as they are represented to the OS via this SATA expander card.