Question on creating a failsafe/rotating offsite Backup with FreeNAS

Status
Not open for further replies.

Micheal

Cadet
Joined
Nov 3, 2016
Messages
9
Hello Everyone,

I'm fairly new to the FreeNAS community, I've been using it for a couple of months. I'm truly loving it. At home I've been using NAS4Free due to my NAS being an Aspire Home Server with an Intel Atom, 4TB storage (with not the best retention), and 2GB RAM (I've had it for like 6 years). I've been wanting to replace it. I have a budget "infrastructure" project I'm working on and I was wondering if I could get some help. This is my first post, so go easy on me lol.

What I would like to do is take a 12TB FreeNAS Volume/dataset (8TB Usable on a RAIDZ2) and clone the whole data-pool/data-set to an external USB Drive (8TB) when connected.

1) I would like the entire data-pool to be intact so that, in a worst case scenario, I could do something like launch a clone of the USB (FreeNAS OS) drive as an ISO image (which I already have using ImageUSB) in say VirtualBox on any machine and mount the external drive, rebuild the volume and be up and running as a temporary solution (though performance would be slow.. but again as a fail-safe scenario).

2) I would like the entire data-pool to be synced incrementally, so that it's technically always a full backup/clone of the FreeNAS server volume but only take the time to transfer/sync what has changed (for speed and efficiency). I would also like it to retain all of the snapshots points/shadow copies.

3) I don't want anything graphical as I'd like to run it as an automatic job when one of the externals are mounted (which I would like to be done automatically and the USB devices verified against a config file to assure that only the drives designated can be written to). I would like the backup to take place automatically when a verified drive is mounted (one of at least two for rotating/offsite backups) so that there does not need to be a defined schedule.

4) I would like to keep logs of pretty much everything.

I can do most of this, really the things I'm not clear on is a program (such as RSync) and/or switches that can do #1 and #2.
Update: I was suggested to use ZFS replication and that's probably the direction I'll go but I'd also appreciate some help, thoughts, or considerations. I can make a smart script as I've already began to but again help is much appreciated.

I'd probably also add a status light to the server (programmable LED) and notification (email, logs & reports, and/or SMS).

For simplicity I considered using blinkstick or blink:
https://www.blinkstick.com/
https://blink1.thingm.com/

Any and all help would be appreciated.

P.S.

Just for conversation, this is my current setup.. technically for production, but it's complicated. Any suggestions or insight would be appreciated. Thanks

Currently my setup is as follows:

  • Hardware:
    • Dell Poweredge R710 ($405 on eBay, plus I got an additional $52 discount with a fun hack lol) with:
      • 12TB (8TB Usable on a RAIDZ2)
      • 72GB ECC Memory
      • Dell H310 Raid Card (Flashed and set to NO-RAID)
      • 2x High Speed USB (for OS fault tolerance)
    • Netgear 24-Port Gigabit Managed Switch ($100 on Amazon)
    • TP-LINK Loadbalancer (acts as VPN server, DHCP, DNS, Loadbalancer/failover of 2 internet connections, and firewall) ($150 on Amazon)
    • TP-LINK Wireless Router (only job is to hand out Wireless, low priority) (Already had)
    • Netgear 2000 Wireless N (flashed with DD-WRT and used as repeater Bridges) (Already had 8 of these)
    • 6U Vertical Rack Mount (replaced R710 screws with ones for rack to secure) ($50 on Amazon)
    • 4 Port USB 3.0 Card (added to server for faster transfer speeds to USB backups) ($15 on Amazon)
    • 2x 8TB External USB Drives ($220 each with OneDrive Cloud storage for a year)
  • Software:
    • Jail Running PhpVirtualbox Appliance
    • FreeNAS acts as Primary Domain Controller (PDC), NAS/SAN, Backup Server, and Virtualization Server (using PhpVirtualBox)
    • VM Running "Windows 10 Professional" (on & used to manage FreeNAS PDC Server and AD, GPO, etc)
    • Seafile (used as personal Dropbox server, desktop sync, and/or cloud drive and it has Cross platform apps for iOS, Android, Linux, Windows, etc)
    • ImageUSB.exe (on Windows machine to make bit level backup of USB Boot Drive)
    • I use a script to sync my recovery files (FreeNAS USB Backup, Config files, Software, etc) to a Free 50GB Mega.nz account (encrypts both ways)
    • TBD
  • Setup:
    • Retention is 2 months with 3 periodic snapshots per day (until I start running out of space lol).
    • I have a few other VMs but I'm just testing things right now (one I really like though is Eclipse Che for development and SnipeIT for Asset Tracking)
    • I use Seafile to backup a few computers (one remote) and provide secure file access, etc.

Other than documentation for setting up FreeNAS as a PDC being non-existent/lacking (which is sad because it's really just one line of code and a few steps, provisioning with CIFS enabled --no install needed), and a version issue with running LACP link aggregation (I bonded the 4 ports on the server to one interface and IP for better throughput), I've had few hiccups with the setup. FreeNAS is a very well built platform and now I have a server that "does it all". I love it.

WARNING RANT (you can skip this, lol): It's a legit/legal way to have a Domain Controller and manage Windows/Linux/Mac machines (from Linux and/or MS Admin Tools through another Windows box, which can be a VM) at a fraction of the cost.. the cheapest Windows server OS is like $700, then you need CALs for each machine that connects to it, then SA agreements, and a bunch of other hoopla, then you need another server/machine for Virtualization because of how heavy a Windows PDC is (overhead) and get another box if you want a SAN/NAS. FreeNAS is all that and more with low overhead with a web interface (awesome)... it work amazingly. FYI: bonding two crappy connections gives you an amazing almost business class connection with amazing reliability (low latency and no single computer can crap out the network--like by simply using YouTube and yes it happens) on the cheapness. The goal of this project was to build the best setup for the least money (minus time). Food for thought for anyone interested. I occasionally rant and for that I apologize. I do love me some tech.

Also, is it okay if I just make a FreeNAS PDC How-To article? I think it be helpful to someone out there?

Much appreciated.

Thank you,

Micheal
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
I could do something like launch a clone of the USB (FreeNAS OS) drive as an ISO image (which I already have using ImageUSB) in say VirtualBox on any machine and mount the external drive, rebuild the volume and be up and running as a temporary solution (though performance would be slow.. but again as a fail-safe scenario).
That's a silly approach. Cloning boot drives has also yielded mixed results, historically. The right way of doing this is to keep an updated backup of the config file and upload it to a new install.
2) I would like the entire data-pool to be synced incrementally, so that it's technically always a full backup/clone of the FreeNAS server volume but only take the time to transfer/sync what has changed (for speed and efficiency). I would also like it to retain all of the snapshots points/shadow copies.
So you want snapshots and replication.
3) I don't want anything graphical as I'd like to run it as an automatic job when one of the externals are mounted (which I would like to be done automatically and the USB devices verified against a config file to assure that only the drives designated can be written to). I would like the backup to take place automatically when a verified drive is mounted (one of at least two for rotating/offsite backups) so that there does not need to be a defined schedule.
That's going to take some hacking.
4) I would like to keep logs of pretty much everything.
What kind of logs?
Also, is it okay if I just make a FreeNAS PDC How-To article?
Sure.
 

Micheal

Cadet
Joined
Nov 3, 2016
Messages
9
I would still need a VM to do a USB restore but I'll do a test with the config backups, thanks (I use plpbt to boot to USB using VirtualBox and testing).

I don't personally think making a script to auto-mount a drive based on the hardware ID is all that "hacky", the script would just call from a config file that can be updated as needed. I'm not new to linux, programming, or scripting. I'm just a bit new to ZFS commands and doing specifically what I want with them. I just want to make sure I can get a mountable clone to the external USB drive. I want there to be two rotating drives for offsite and fail-safe backup purposes and I just want to monitor them and restrict that not-just-any-drive can be plugged in and backed up, but then again that probably would be fine.. I just want a level of control. But if you know of a better way, please do tell.

As for logging, I want to keep track of when a External Backup to USB was performed, what USB drive it was performed on, a verbose log perhaps, reporting, failure and warning notifications (email and SMS), etc.

I do appreciate the input.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
I backup my FreeNAS server using something less automated. It has 4 x 4TB in a RAID-Z2
vDev, so my external 8TB disk can catch everything. The backup process is scripted, including
the scrub performed before the backup. And I do get great logs out of it, (now, not so much
when I was writing the script).

Here are some things different from what you need, or I don't need:
  • Mine uses an eSATA enclosure, (eSATA is thought to be more reliable that USB)
  • Manually import the pool / disk
  • Manually launch backup
  • Mine uses Rsync, since I don't want to replicate client backup snapshots
  • Manually export the pool / disk
Your hardware list shows a USB 3.0 PCIe card. Yes, USB cards and hard drives are cheaper.
But, you may get what you pay for. It might work perfect. I just trust eSATA more, (and SAS
even more).

Next, you can probably do everything you listed, including auto-detect and import pool. But,
it may take you a while to debug the scripts. Mine took at least 8 months to get where it is now,
since I only performed monthly backups.

Even with all the automation, verification of backups is important. Any backup that you can't
perform a restore from, was wasted time and energy.
 
Status
Not open for further replies.
Top