Please validate my backup plan: Rotating offsite backup disks, from single FreeNAS primary storage

Status
Not open for further replies.

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
I intend to assemble a FreeNAS box using all of the great advice and guides I have seen on this forum. Thanks to everyone who contributes.

I have a backup scheme planned that seems very robust to me. I've researched threads of people who designed similar schemes, but I have a few unique questions and would just like to get some validation from the community.

My plan is to run a FreeNAS box (on server-grade hardware, ECC, etc.) which will be my home's main fileserver, holding all our data and precious memories. The data will be available to network machines (Windows and portable devices) via CIFS shares. Attached to this FreeNAS box, I would have an eSATA drive which does an rsync or something nightly to keep daily backups of our files. Once every few months, I would take this drive offsite and rotate it with a second backup drive. So, I'd have two backup drives which are continually rotated offsite.

Maybe I'd also keep a third "stationary" drive permanently attached via eSATA, just as another layer of daily backup protection.

Here's my concerns and questions.
  1. What are your first thoughts? Does this sound like a robust scheme? Or is basing it all off of FreeNAS risky?
  2. If I run ZFS on the backup drives, would they be adequately protected from "bit rot" and other nasties which affect lesser file systems, despite not being mirrored or in a RAIDZ? I don't want to go to a backup and find it's been corrupted.
  3. Would I be able to somewhat easily do a FreeNAS GUI-based encryption of my backup drives which rotate off-site? If it's too complicated I will be afraid of screwed it up and losing everything in a backup-reliant situation. Does this fall within the expected usage of the encryption features?
Thanks for reading my thread and being a part of this awesome FreeNAS offering.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
1. Seems fairly good for home data. Poor for a business that can pay for better stuff. But for home, very acceptable.
2. It won't protect you from bitrot, but it will identify it with just a single disk pool.
3. Doing encryption of disks that are being removed from the server doesn't work out so well. Several people have tried to use it, and it didn't end well for them. Not sure of the specifics since its been a long time since anyone thought of trying to do it.

Do keep in mind that you will need to either export the pool before disconnecting the external drives or shutdown the server to disconnect the drives.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
What are your first thoughts? Does this sound like a robust scheme? Or is basing it all off of FreeNAS risky?
That's almost the same as my backup "system". The only difference is that my backup drive is in an removable SATA tray instead of being an eSATA external drive. I also use ZFS replication instead of rsync.
If I run ZFS on the backup drives, would they be adequately protected from "bit rot" and other nasties which affect lesser file systems, despite not being mirrored or in a RAIDZ? I don't want to go to a backup and find it's been corrupted
Cyberjock already answered this one.
Would I be able to somewhat easily do a FreeNAS GUI-based encryption of my backup drives which rotate off-site? If it's too complicated I will be afraid of screwed it up and losing everything in a backup-reliant situation. Does this fall within the expected usage of the encryption features?
All my pools are encrypted, including the two backup ones. It works perfectly (I did not even have to hack anything ;)). Just make sure that you keep backup of both encryption keys and passphrases (it would be possible to force it via command line to use the same key for both backup drives, but I did not bother).

The one slightly annoying thing with this schema is that when you export/detach a pool (required before you remove it) FreeNAS will discard the related disk settings (SMART extra parameters, spin down timeout, ...) and its scrub schedule. When you connect the other drive you'll get default disk settings and a default scrub will be scheduled. So, you need to redo your changes (if any) every time you swap the drives (I have a script to do that).
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
Hmm. So I've got cyberjock saying that removing encrypted disks is risky and dusan saying it works perfectly.

Maybe I should just rely on my old NTFS bitlocker encrypted offsite backups instead of risking it with FreeNAS encrypted offsite backups.

I need to read up more on snapshots but I was hoping that using a ZFS backup would let me to keep snapshots on there, which would give me a proper backup that allows me to go back to a certain point in time.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
My evidence is based on the encryption system as it has been the last year or so. I've found quite a few serious bugs. 8.3.1 added encryption. But, FreeNAS wouldn't let you do a disk replacement of the pool. Yes, it had bugs like that in it. I had a failed disk in an encrypted pool, so I have a thread somewhere discussing how to do a disk replacement via editing the FreeNAS sql database. Not for the light hearted for soure.

Even after some bugs like that have been fixed, we have a bug in 9.2.0 that can result in a loss of pool. No newer RELEASE has been made.

My other reasoning is that we've had many users try to do disk removals and reinstallations like you are trying to do. It has failed for many/most of them.

Anyway, based on the history of the encryption system, I'm not quick to trust it to do many things right(and I'm dreading the day I need to do a disk replacement on my encrypted pool again). There's been a history of bad blood between users and the encryption system. While we've had many people use it, there have been significant hiccups along the way. Dusan can read the code, so he can somewhat vouch for how it is in its present condition. There's always the possibility for another bug we aren't aware of. I am not a coder(I dabble, but that's about it). In short, his view and my view differ, but neither is wrong. It's just different ways at looking at the picture. My words of warning is to tread lightly and keep backups if you want to do this. If you do set it up and use it, definitely report back. Its always nice to get feedback of what does as well as what doesn't work.
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
1. Seems fairly good for home data. Poor for a business that can pay for better stuff. But for home, very acceptable.

Could you briefly explain what a business might do differently? I am willing to spend more to do it "right" to a certain point.

One thing I'm considering is setting up a similar server at family's house to backup to. I know businesses often backup offsite like this.

Do keep in mind that you will need to either export the pool before disconnecting the external drives or shutdown the server to disconnect the drives.


Good point, thanks. I'm hoping this will be somewhat trivial after I look into how to do it and experiment with my test setup which is now running.
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
That's almost the same as my backup "system". The only difference is that my backup drive is in an removable SATA tray instead of being an eSATA external drive. I also use ZFS replication instead of rsync.
Can you hot plug your SATA tray? I need to learn more about hot plugging. And what the difference is to hot swapping. The internet seems confused about this.

Q1. When you say ZFS replication, do you mean copying snapshots from the original drive to the backup?

The one slightly annoying thing with this schema is that when you export/detach a pool (required before you remove it) FreeNAS will discard the related disk settings (SMART extra parameters, spin down timeout, ...) and its scrub schedule. When you connect the other drive you'll get default disk settings and a default scrub will be scheduled. So, you need to redo your changes (if any) every time you swap the drives (I have a script to do that).


Q2. Could you share your script with me? I would like to see how complicated it is if I have to create one.
Q3. How do you address the issue that something (ex: bit rot) could corrupt your backup and you wouldn't know it until you need to restore?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Could you briefly explain what a business might do differently? I am willing to spend more to do it "right" to a certain point.

One thing I'm considering is setting up a similar server at family's house to backup to. I know businesses often backup offsite like this.

Businesses often backup offsite with ZFS snapshot/replication. ZFS has its snapshot/replication feature solely for backing up data. My guess is you won't want to build a second FreeNAS server and keep it elsewhere so you can do syncs over your LAN or internet.
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
My guess is you won't want to build a second FreeNAS server and keep it elsewhere so you can do syncs over your LAN or internet.

I might not want to, but if my brother decides that FreeNAS will work well for him too, then I'm sure I could convince him to install a disk or two for receiving my backups! Could be a pretty sweet setup! Make use of that Comcast Blast internet connection overnight when no one in the house is using it!

Overnight. You know: when Comcast decides to have "unplanned downtime" aka "internally planned downtime" since consumers don't need reliable internet.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Can you hot plug your SATA tray? I need to learn more about hot plugging. And what the difference is to hot swapping. The internet seems confused about this.
It's actually a trayless bay (http://www.raidsonic.de/en/products/mobile-racks.php?we_objectID=6277). It supports hot swap, but I never tried it. It is a home system, so I don't care about small downtimes.
Q1. When you say ZFS replication, do you mean copying snapshots from the original drive to the backup?
I do incremental replications. I have an automated snapshot task that snapshots the main pool and the snapshot is then replicated to the backup drive (incremental = only changed blocks are transferred).
Q2. Could you share your script with me? I would like to see how complicated it is if I have to create one.
Of course. I did not yet upgrade to 9.2.0, so the script does direct writes to the config DB. I plan to update it to use the FreeNAS API later. My removable backup pool is named backup, the disk is ada1:
Code:
#!/bin/sh
sqlite3 /data/freenas-v1.db "update storage_disk set disk_hddstandby='60', disk_smartoptions='-a -M daily' where disk_name='ada1';"
sqlite3 /data/freenas-v1.db "update storage_scrub set scrub_threshold='25', scrub_minute='59', scrub_hour='01', scrub_daymonth='06', scrub_month='*', scrub_dayweek='*' where scrub_volume_id = (select id from storage_volume where vol_name='backup');"
service ix-ataidle quietstart ada1
service ix-smartd quietstart
service ix-crontab quietstart
service smartd forcestop
service smartd restart

I run the script after I replace the backup drive and import the pool. It sets the idle timer, some smart options (-a is no longer needed in 9.2.0) and adjusts the scrub (it gets created automatically when you import the pool) schedule to my liking. It then regenerates config files / restarts services so that the changes take effect.
Q3. How do you address the issue that something (ex: bit rot) could corrupt your backup and you wouldn't know it until you need to restore?
With my backup rotation schedule the system will perform at least one scrub before I swap the drives. This will detect any bitrot and I can redo a full backup if needed. Also, my main pool is RAIDZ2 so 5 drives need to fail for me to lose data -- 3 drives in the RAIDZ2 and the two backup drives. Of course, if somebody steals the server then I'm down to the one offsite backup drive. If some bitrot happens at such an unfortunate moment I should still not lose all data (in theory, that is :)). ZFS stores metadata in several copies so it should be able to correct smaller metadata corruptions even in a non-redundant pool (so the pool should still import). With data block corruptions I will only lose the affected files and not the entire pool.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Hmm. So I've got cyberjock saying that removing encrypted disks is risky and dusan saying it works perfectly.
I did read the code so I'm OK with using the encryption. I already swpped the drives several times. However, understanding the internals also prevents me from doing some "stupid" things that the GUI may still allow you to do. The latest bug cyberjock mentioned is I guess the one about trying to extend (add another vdev) an encrypted pool that is locked (the keys are not present). This is something I would never do, but I understand that somebody may try to do it and the GUI should prevent it. I'm also quite confident that I will be able to fix any config DB issues if needed. The one encryption bug I really care about is this one: https://bugs.freenas.org/issues/3206. The encryption introduces a single point of failure (the geli metadata). I use this script to quickly backup all the metadata into disk-serial-number named files. You just need to run it once after you create an encrypted pool (it assumes you use swap partitions and that all drives are encrypted, it may throw some errors for unencrypted drives):
Code:
#!/bin/sh
for disk in /dev/ada*p2
do
  geli backup $disk `camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`.eli
done

I need to read up more on snapshots but I was hoping that using a ZFS backup would let me to keep snapshots on there, which would give me a proper backup that allows me to go back to a certain point in time.
ZFS snapshots & replication allow you to do exactly that. My backup pool is an exact replica of the "critical" dataset (the one I backup) -- this includes the history of snapshots. This means I can go "back in time" via snapshots both in my main and backup pool.
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
Great info Dusan! Thanks for your help :) May have some more questions once I go to actually set this up.
 

dztrbd

Dabbler
Joined
Dec 6, 2013
Messages
36
Code:
#!/bin/sh
for disk in /dev/ada*p2
do
  geli backup $disk `camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`.eli
done



Hi, With this script, what and ahere is the output supposed to be? Ive run the script and it appears to do nothing, and i cant find the backed up data?
Thanks
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
It creates the backups in the current directory. However, the script was meant as an example -- it assumes lot of things (for example that your disk devices are named /dev/ada*).
 

dztrbd

Dabbler
Joined
Dec 6, 2013
Messages
36
Well my disks are ada0 -> 5 . Ive looked in the current directory and nothing comes up? I also looked at the geli man page, and wasnt sure about the $geli backup args.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Well my disks are ada0 -> 5 . Ive looked in the current directory and nothing comes up?
Is your current directory writable?
I also looked at the geli man page, and wasnt sure about the $geli backup args.
The geli backup syntax is "geli backup <provider> <filename>". If you did not change the FreeNAS default the provider is the second partition on the disk (the first partition is swap). So, to backup ada1 metadata to a geli_ada1 file you would run "geli backup /dev/ada1p2 geli_ada1". My script tries to use the serial numbers of the drives as the file names. I did consider using the GPTIDs but I want to be able to match the backups to physical drives even in case the partition tables get corrupted too. The partition table is easy to recreate, but you will get new GPTIDs.
I just also noticed that my script will fail if the serial number contains space(s) (my WD REDs' serial numbers don't). This should work better:
Code:
#!/bin/sh
for disk in /dev/ada*p2
do
  geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`".eli
done
 

dztrbd

Dabbler
Joined
Dec 6, 2013
Messages
36
Is your current directory writable?

Yep $ ls -ld drwxrwx--- root wheel. I ran the script as su. Im currently in /mnt/Freenas/user.

I just also noticed that my script will fail if the serial number contains space(s) (my WD REDs' serial numbers don't). This should work better:
Im also running Wd reds


Code:
#!/bin/sh
for disk in /dev/ada*p2
do
  geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \  | cut -d \  -f 3-`".eli
done

That code is the exact same as the one above.
  1. #!/bin/sh
  2. for disk in /dev/ada*p2
  3. do
  4. geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \ | cut -d \ -f 3-`".eli
  5. done
Not quite, I missed that.




The output i got was.
geli: Cannot open 'camcontrol identify /dev/ada0 | grep serial | tr -s \ | cut -d \ -f 3-'.eli: No such file or directory.
Repeated for all drives
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Ah, did you type or copy & paste the code? I assume you did the first.

geli backup $disk "`camcontrol identify ${disk%p2} | grep serial | tr -s \ | cut -d \ -f 3-`".eli

Notice that the red character is a back quote, not an apostrophe -- ` vs. '
 

dztrbd

Dabbler
Joined
Dec 6, 2013
Messages
36
I typed. And im an idiot. I completely missed that. That would explain some other issues ive had. Thanks so much for your time.

After the change, It returns cut: bad delimiter
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
It's better to just copy & paste the code ;).
There are two spaces after each backslash:
tr -s \[space][space]| cut -d \[space][space]-f 3-
The forum silently compresses them into one space in the post discussing the back quotes, but they are present in the [ CODE ] enclosed code.
 
Status
Not open for further replies.
Top