How to: Backup to local disks

[Arwen]

Arwen submitted a new resource:

How to: Backup to local disks - Command line driven backups to direct attached disks

How to: Backups to local disks

Overview:
---------
This subject comes up often enough that I thought I would start a post of
how I do it. Specifically I back up my FreeNAS to a hot-inserted, locally
attached disk, then when done, hot-remove and take off-site.

The discussion whether hot-swap works okay or not, is up to the person
implementing the solution. You have to decide if your SATA or SAS ports are
up to the challenge of hot-swap. And if your drive slot, (internal or
external), can...

Read more about this resource...

 

[Arwen]

Attached is the backup Bash script.

 

[Arwen]

Updated the formatting to make it look prettier. Specifically, added CODE tags in the appropriate
places, used real under-lining, not dashes and removed the hard carriage returns.

Plus, clarified some of the details.

 

[Glorious1]

Thanks for this. It's interesting to see how it works. I do something similar, but the backup pool stays in the box rather than being stored off site. The main pool gets backed up daily to the backup pool via replication (using the target hostname 'localhost'). Maybe I'll consider swapping the pool out, although I can't hot-swap with my hardware, and don't have esata.

I wonder if it's possible to have two sets of disks for two backup pools with the same name, and trade them from time to time. Would FreeNAS see them as different for replication? Would the replication get messed up because it was the other backup pool that received the previous replication?

 

[Arwen]

...
I wonder if it's possible to have two sets of disks for two backup pools with the same name, and trade them from time to time. Would FreeNAS see them as different for replication? Would the replication get messed up because it was the other backup pool that received the previous replication?

Since I use Rsync it's probably possible for me to use backup disks with the same Pool name, (as long as they are not inserted at the same time).
I simply don't know about ZFS replication to 2 different Pools that have the same name, (but not inserted at the same time).

 

[Arwen]

Updated resource to mention procedure can work without hot swapping disk, by simply shutdown and performing the disk change cold, (without power).

 

[Redcoat]

A quick question: You mention "Dual SATA to eSATA PCIe back panel adapter is wired to 2 x Intel SATA ports, not the Marvell SATA ports" that you use in your mini. Am I right in concluding that device is not a PCIe card plugged into the mobo slot, but a basically "just" a bracket carrying two eSATA ports with cables to the mobo ports?

 

[Arwen]

Arwen updated How to: Backup to local disks with a new update entry:

How to: Backup to local disks

Added link to eSATA to PCIe back panel. Updated the wording a bit.

Read the rest of this update entry...

 

[Arwen]

A quick question: You mention "Dual SATA to eSATA PCIe back panel adapter is wired to 2 x Intel SATA ports, not the Marvell SATA ports" that you use in your mini. Am I right in concluding that device is not a PCIe card plugged into the mobo slot, but a basically "just" a bracket carrying two eSATA ports with cables to the mobo ports?

Yes, it's just a cable / bracket. Here is a link;

https://www.amazon.com/gp/product/B005KKCP26

I've also updated the resource with the link too.

 

[Redcoat]

Thanks for the confirmation. I think the real key piece of info is the use of "known good performance" SATA ports ...

 

[Arwen]

Arwen updated How to: Backup to local disks with a new update entry:

Updated script

I've added features to the script and am now publishing the changes.

Read the rest of this update entry...

 

[NASbox]

External USB drives for backup weren't great with UFS (but I did get away with them for over 2 years), but once support for the UFS file system was dropped external USB drives were a disaster waiting to happen, so I struggled for some time to find an affordable and practical backup solution. Input from @Arwen (Thanks!) I got me stared down this path.

As a result of the assistance I've received I'd like to offer this resource back to the community now that I've been using it for several months with great success.

I found an inexpensive and durable removable drive system that makes external drives for backup very easy/convenient/safe.

Full disclosure: I do not benefit from this in any way financially or otherwise. This is such a good product, I'd like to see it stick around and I think the community would benefit from knowing about it. The manufacturer has done a crappy job of marketing, and the web site sucks! (The brain dead search engine on the web site didn't find the item even when given the exact part number! FYI The following page has the model number info and current price info http://www.addonics.com/products/ddcssas.php ). Other details can be had here: http://www.addonics.com/products/ddcssas.php

It took me almost a day of serious searching find this unit, and then once I found it several days of back-and-forth with the support staff on their ticket system to clarify important details such as rated life of the connector and the correct part numbers to order.

FYI the units I purchased were:
Complete Unit: Model DDCSSAS
Extra Drive Cady: DSACSB

For the benefit of those who haven't researched the life span of the contacts on a SATA drive, I know it surprised me how very fragile the they are and how very few plug/unplug cycles they are rated for (Between 50-100). That's not going to work out very well for frequent changes over an extended period of time. (Murphy's law states your connection will wear out just when you need the drive to restore your pool!) All the other Hot Swap racks I was able to find did not protect the delicate SATA connectors because they are not designed to use the drive like a backup cartridge. In most normal application, you swap out a dead drive, or remove drives during system maintenance-that is only going to happen a few times during the life of the drive/system, so it's no big deal to just use the bare drive. As a removable backup that is changed 2 or more times a week, that connector is going to be badly worn in a year or less.

If you have a case that has an open 5 1/4" inch drive bay, and an extra SATA port that supports hot swap, the Diamond Drive Cartridge System will convert any SATA drive into a robust backup cartridge. (If you need an external device, that is possible as well, but I don't have the details - go to the web site and/or contact Addonics).

What makes this unit so good for a removable backup system are:

1. It has it's own high endurance SATA connector that connects the cartridge and the backplane that is rated at 10,000 insertion cycles.
2. It is very well manufactured to tight tolerances so that it inserts and removes from the backplane very smoothly. (Back in the IDE Drive days I used several removable drive systems, and they all were very tricky to insert and remove due to poor manufacturing tolerances.)
3. It well ventilated and is made of aluminum, so it does an excellent job of keeping the drive cool. There is no fan, so it is silent. (WD RED 8TB runs at about 10C over ambient, and a 10TB 7200 HGST drive was about 12C over ambient after days of burnin with badblocks-Mounted in a Corsair Obsidian Full Tower with stock fans)
4. Reasonable Price - Complete unit is $49US, and the cady only is $25US in 2017.
5. Doesn't require an extra cabinet - fits right in the case - no cables to accidentally get disconnected. Locking key to make you think before unplugging.
6. It was easy to install, and came with a decent SATA cable.

I've been super happy with the system-great for off-site backups and keeping a backup that is safe from catastrophic events such as a lightening strike on the power line or power supply failing and frying the whole system. Very easy to use:

1. Just slide the cartridge in, turn the key and the drive powers up.
   
2. Wait for the drive to spin up and be recognized by FreeNAS.
3. Sign in to the GUI and import the drive (pool) or if it's the first time with a new disk create your pool.
   
4. Run your backup.
   
5. Log into the GUI again and detach the drive.
   
6. Once detached, turn the key, and the drive powers off - wait a few seconds for the drive to spin down and park (to be on the safe side).
7. Pull out the cartridge and store the drive safely.

(For the benefit of fellow Canadians (Americans can easily get it directly from Addonics and possibly other major suppliers) the most affordable way to get this in Canada (Summer 2017) is from NCIX, but it was not on the NCIX web site. I had to contact NCIX and arrange a special order, and I also had to contact Addonics to contact NCIX to advise them of the availability of the Drive Cady: Part DSACSB)-NCIX orignally told me it was unavailable. The only delivery method offered by Addonics for international orders was courier and the brokerage fees for customs clearance were outrageous. It was cheaper to have NCIX import to Vancouver and then ship to me customs cleared in the Toronto area.)

I hope this helps others looking for a simple backup system for home/small office use where a second system for replication is not practical, and that it will save other from wasting all the time that I had to waste to find and acquire this solution.

 

[Kevin Horton]

I'd like to set up offsite backups for several datasets, and I'm pondering using rsync for this. The script in this resource would back up the whole pool, but I want to only do selected datasets.

If I understand the script correctly, line 255 has a list of things to exclude. Is it as simple as editing that line to list the various datasets I don't want to backup?

I'd also edit line 251 so the log matched what I was actually doing.

Thanks for any advice.

 

[Arwen]

I'd like to set up offsite backups for several datasets, and I'm pondering using rsync for this. The script in this resource would back up the whole pool, but I want to only do selected datasets.

If I understand the script correctly, line 255 has a list of things to exclude. Is it as simple as editing that line to list the various datasets I don't want to backup?

I'd also edit line 251 so the log matched what I was actually doing.

Thanks for any advice.

Yes, in someways you can modify the script to do what you want. Mine was designed around 2 locally attached disks, 1 x 8TB for all, and a 750GB for all except media. Thus, the detection of the destination pool name.

My suggestion is to perform the task manually for the first few times, documenting the details clearly. Then you can write up a script for some / most of it, (but still manually run).

For example, my Linux backups to either my FreeNAS or locally attached backup disks, is based on file systems. Then it's easy to exclude file systems named Media. I have Rsync skip any children file systems, (using -x), and simply loop on the file system list.

Last, you might simply look at the FreeNAS GUI and see if you can setup a Rsync task to perform what you want.

 

[Kevin Horton]

Yes, in someways you can modify the script to do what you want. Mine was designed around 2 locally attached disks, 1 x 8TB for all, and a 750GB for all except media. Thus, the detection of the destination pool name.

My suggestion is to perform the task manually for the first few times, documenting the details clearly. Then you can write up a script for some / most of it, (but still manually run).

For example, my Linux backups to either my FreeNAS or locally attached backup disks, is based on file systems. Then it's easy to exclude file systems named Media. I have Rsync skip any children file systems, (using -x), and simply loop on the file system list.

Last, you might simply look at the FreeNAS GUI and see if you can setup a Rsync task to perform what you want.

I cobbled together an rsync command, based on my reading of your script, plus a healthy dose of Googling. It is churning away now, apparently happily, and I'll look closely once it has finished to confirm it did what I expected. I think I can get a basic script set up eventually, to handle my planned two sets of backup disks.

Thanks for your script. It didn't seem to handle finding the backup path correctly (I'm on FN 11.1-U6), but it did get me pointed in the right direction.

 

[Arwen]

Glad I was able to help.

Even from the command line, you can add log file using;

| tee -a

Scripting is one reason I like *nix. Almost anything can be scripted.
MS-Windows is catching up, and allowing good scripting. But MS-Windows was never designed for the command line, so esoteric things can be hard or impossible to do. (OpenZFS may actually come to MS-Windows!)

As for backup scripting, it's going to be the extremely rare case where my script can be useful without serious modification. It's mostly just an example. There are just too many options people have for backups. Trying to write a general purpose backup tool that users can plug in values, (like destination disk or host), is beyond what I can do without a lot of prodding, (or money).

 

[pro lamer]

OpenZFS may actually come to MS-Windows!

Some attempts: zfs-win (read only), zfsin (tl;dr)

Sent from my mobile phone

 

[Ericloewe]

ZFS for Windows is coming along nicely, but slowly.

 

[Arwen]

Arwen updated How to: Backup to local disks with a new update entry:

Corrected spelling

Updated the spelling of ASRack to ASRock Rack, which is what the company uses.

Read the rest of this update entry...

 

[Jatrabari]

I am planning to implement finally a local disk replication. Is this script still viable to use? What about zfs send receive and/or straight GUI replication task?