USB Hard Drives for BackUP

[angoll]

Hello everybody, I'm reading posts for over a year but this is my first post :).
I read a lot of how USB hard drives shouldn't be used in Freenas, mostly because they were using it as a primary HDD.

But my idea is to use it as a backup solution, connect a usb hdd once a week make a copy and disconnect the drive. As in other NAS solutions software they have a utility to copy the files from the NAS to an external hard drive. To my knowledge there is no advice in this forum for that. Some people just connect the hard drive to another computer and transfer all the files. But I think we should be able to connect the usb drive to the server and make a backup of the data.

So I want to start a discussion about "best practices" or how we can achieve having this feature.

My questions are:
1. Is possible to make it ? Or there're other usb related problems?
2. Which filesystem we should use on the drive?

What I thing it should do is as a pseudo-code
1) Detect and mount the USB drive
2) Perform some analysis of the drive, SMART test? Check corruption? Integrity?
3) Perform a backup using ZFS or RSYNC
4) Unmount the drive and let us know that we can unplug the drive.

What do you think?
thanks.

 

[Robert Trevellyan]

1. It's possible, but eSATA would be better.
2. ZFS.

1. Yup.
2. Best you can do via USB is ZFS scrub.
3. Yup.
4. In the GUI, detach the pool first.

 

[Mirfster]

Or there're other usb related problems?

Side note, as far as I know USB 3.0 is NOT supported in FreeNas 9.3. So you are going to have to bear with USB 2.0 speeds. However, if you connected the USB 3.0 device (if it is 3.0) to a machine that does support it, then copied over the network you will have much faster speeds.

 

[DrKK]

Side note, as far as I know USB 3.0 is NOT supported in FreeNas 9.3. So you are going to have to bear with USB 2.0 speeds. However, if you connected the USB 3.0 device (if it is 3.0) to a machine that does support it, then copied over the network you will have much faster speeds.

I think this "9.10" FreeNAS they are in the process of releasing solves the USB 3.0 issue.

 

[Mirfster]

I think this "9.10" FreeNAS they are in the process of releasing solves the USB 3.0 issue.

Methinks you are correct, just would like to see true USB 3.0 support/speeds and a "Stable" release before declaring so.

 

[angoll]

Thanks for the replies,

It would be better to use Rsync or ZFS replication?
Based on this thread (Rsync vs Zfs snapshot Replication vs other -over wan), I fount that ZFS replication make a ssh connection. But with Rsync will I have the snapshots?

The ZFS scrub check for integrity of the data and disk problems, it should be done it after the copy of the files, right?

Is there is any way of check that the copy didn't have errors? I read that if I use ZFS replication and lost connection somehow it can't resume and complete the backup.

I'm hoping for the usb 3.0 support, maybe is off-topic but since version 8.2 of FreeBSD usb3.0 is partially supported, why version 9 of FreeNAS is not partially supported too?
For my understanding and please correct me if I'm wrong but FreeNAS is a personalized/embed of FreeBSD for use in NAS systems? shouldn't be the core be the same?

EDIT: I found this thread, about usb3 and how can be enabled (in the documentation), enabled to check if it works.

Thanks in advance

 

[Robert Trevellyan]

It would be better to use Rsync or ZFS replication?

With rsync, you can resume an interrupted transfer. With ZFS replication the entire filesystem structure, including nested datasets and snapshots, is preserved.

Is there is any way of check that the copy didn't have errors?

That's what the scrub is for. With eSATA, you can do SMART checks and run SMART tests too.

 

[Ericloewe]

I think this "9.10" FreeNAS they are in the process of releasing solves the USB 3.0 issue.

It does, as far as I can tell. I successfully installed FreeNAS 9.10 from a USB 2.0 drive to a USB 3.0 drive on my Skylake system.

I haven't tried the IPMI virtual devices yet, but those should now be working, too.

 

[Arwen]

I agree with Robert Trevellyan's response 100%;

1. It's possible, but eSATA would be better.
2. ZFS.

1. Yup.
2. Best you can do via USB is ZFS scrub.
3. Yup.
4. In the GUI, detach the pool first.

I perform something similar. My backup drives are connected via eSATA.
They are a Seagate 8TB SMR, (Shingled Magnetic Recording), for the entire
data. And a Seagate 750GB for all except the media. Both are formated ZFS
as a single disk pool and I scrub them once a backup. Stored in an anti-static
bag, which is stored in a Seahorse padded & water proof case, and rotated
off-site.

One note: I run the entire backup process via command line as "root" on my
FreeNAS. Partly since I have the commands pre-staged, then copy and paste
them. Partly because I am very familair with Unix command line.

My suggestion is whatever you do, document it, test it and update the document
as needed. Backups are worthless unless you can perform a restore. Thus,
reliability to top priority. With USB being somewhat less reliable than eSATA,
testing is important.

 

[mike360x1]

I agree with Robert Trevellyan's response 100%;
One note: I run the entire backup process via command line as "root" on my
FreeNAS. Partly since I have the commands pre-staged, then copy and paste
them. Partly because I am very familair with Unix command line.

Sorry to Hijack this old post. I figured it would be easier than making a new one. :)

Would you be able to share your code by any chance? :)

I see solutions to backup data from one machine to another in the FreeNAS documentation but nothing about backing up specific datasets to independent drives(Just a single drive in stripe).
I assume one just clones a snapshot from a dataset and copies that to the aforementioned drive. I just wanted to confirm that. Or do you have another method?

 

[Arwen]

Sorry to Hijack this old post. I figured it would be easier than making a new one. :)

Would you be able to share your code by any chance? :)
...

Well, it's not so much code, as a plain text document I can copy and paste from. Main source pool is named "Galadriel" and the backup pools start with "GBackup".
Caveats;

• I don't show how to create the single disk pool, but that should be trivial.
  
• Since I have the space, I create multiple copies, at least until my backup takes more that 50% of the space of the backup disk.
• This uses Rsync, not ZFS send and receive, because it's a full copy. Later will use ZFS send and receive when backups take >50% of the backup disk(s).

Code:

                FreeNAS backup to external disk
                ===============================

Import disk:
----------------
- For StarTech dual disk enclosure:
        - Make sure backup disk is in top bay
        - Dis-connect eSATA cable
        - Turn enclosure on
        - Wait til drive is ready, (10 seconds, maybe more)
        - Connect eSATA cable
        - Examine front panel's drive LED. Should be solid blue.
- Run;
        zpool import
- If import fails, it may be needed to run the following and repeat import;
        camcontrol rescan all
- Run;
        zpool import GBackup
                or
        zpool import GBackup750
- Scrub it before new backup
        zpool scrub GBackup
                or
        zpool scrub GBackup750
- Wait till scrub is complete before starting backup;
        zpool status GBackup
                or
        zpool status GBackup750


Setup backup:
-------------
zfs create GBackup/backups/Galadriel.`date +%Y%m%d`
        or
zfs create GBackup750/backups/Galadriel.`date +%Y%m%d`


Start backup:
-------------
zfs list GBackup Galadriel
time rsync -aAHSX --stats /mnt/Galadriel/ \
  /mnt/GBackup/backups/Galadriel.`date +%Y%m%d`/
zfs list GBackup Galadriel
        or
zfs list GBackup750 Galadriel
time rsync -aAHSX --stats /mnt/Galadriel/ \
  --exclude='Media/Videos' \
  /mnt/GBackup750/backups/Galadriel.`date +%Y%m%d`/
zfs list GBackup750 Galadriel


Export backup disk:
-------------------
cd /
sync
zpool status GBackup
zpool export GBackup
zpool list
        or
cd /
sync
zpool status GBackup750
zpool export GBackup750
zpool list

 

[Mirfster]

Well, it's not so much code, as a plain text document I can copy and paste from.

Well, you have all the commands there, just throw it into a script already girl... :P

 

[Arwen]

Well, you have all the commands there, just throw it into a script already girl... :p

Yeah, well, since my data does not change much, I run backups perhaps once a month. So it's not a problem.
I'll script it later. And auto-detect the backup disk and type of backup, (full or full without media).

 

[Robert Trevellyan]

I assume one just clones a snapshot from a dataset and copies that to the aforementioned drive.

If you decide to use snapshot replication, this isn't quite right. You don't clone a snapshot for this purpose, you send and receive it. Best to study the ZFS man page and practice using the -n option before doing it for real.

 

[Majo]

Hmm, you all recommend ZFS for the backup volume. Idk - the big drawback with ZFS on a single disk volume is data integrity. If the volume gets corrupted you'll lose the whole subtree if not the whole disk. With UFS you'll start an fsck and (hopefully) you'll only use a few blocks. When in doubt i'd go with UFS here. Sure, you'll lose the snapshots but that should be handled by your backup software. On the minus-side, you'll never discover bit-rot, which is bad especially for backups.

Opinions?

I'm in the process of setting up (additional) backups for two FreeNAS systems: the first with a 6*4TB RAID-Z2, the second with a 6*2TB RAID-Z2. We already use bacula with LTO, but over the time the tapes got too small for all the data.

On a sidenote, but still within the topic: Which backup software do you use?

For the more complex tasks we use bacula on a dedicated backup server. The bacula client runs in a FreeNAS jail and has read access to all the datasets in the backup.

For the simple tasks we use dirvish. In fact i use it privately, too. Simple, fast, reliable and gets the job done. And it comes without GUI, which i see as a plus ;)

root@FreeBSD-Jail:~ # psearch dirvish
sysutils/dirvish Network backup system based off of rsync

Otoh, there are plenty of backup tools in the ports tree:

root@FreeBSD-Jail:~ # psearch backup | wc -l
74

I'm open for suggestions.

-Walter

 

[Arwen]

One thing about using a single disk with ZFS for FreeNAS backups. Even though you may get corruption due to
lack of redundancy, in my opinion I'd rather know about bad blocks, than risk UFS's blind acceptance of bad data.
Until ZFS, (and Linux's BTRFS), corrupted data blocks could not be found by FSCK. Even corrupted metadata in
UFS can result in lost data.

Remember, you get AUTOMATIC redundancy of meta-data with ZFS. So a single bad block in meta-data is not
fatal. (Or multiple bad blocks in different meta-data, also not fatal.) Futher, you can, if so inclined, use "copies=2"
for certain data sets, (which then sets metadata copies to 3!). Or even set the entire backup copies to 2, to overcome
some bad block situations.

My backup scheme is 3 fold;

1. NAS, (currently a FreeNAS, but use to be an Infrant ReadyNAS 1000S). Done with Rsync.
   
2. External backup disks, usually with USB connections, but now some have eSATA. Done with Rsync.
   
3. Burned DVD or Blu-ray discs, for the archival copies, (perhaps 2 to 3 times a year). Done with DAR, (Disc Archiver).

Now the NAS has it's own backup disks, (1 x 8TB for all, and 1 x 750GB for all except media). So I am a bit lazy
with the other 2 backups. (This is home only... never lazy about work backups.)

 

[Majo]

Thanks for your thoughts...

Remember, you get AUTOMATIC redundancy of meta-data with ZFS. So a single bad block in meta-data is not
fatal. (Or multiple bad blocks in different meta-data, also not fatal.) Futher, you can, if so inclined, use "copies=2"
for certain data sets, (which then sets metadata copies to 3!). Or even set the entire backup copies to 2, to overcome
some bad block situations.

Ah, i didn't know that. But then, i'm no ZFS expert. I'm a Unix/Linux/BSD admin for about 20 years now, but didn't use ZFS up to now. Sounds good. In that case ZFS makes sense even on a single disk. After all: If you scrub after backup you should be on the safe side - if you have time enough for scrubbing between backups.

Now the NAS has it's own backup disks, (1 x 8TB for all, and 1 x 750GB for all except media).

That's what i planned, too. I know it sounds a bit paranoid (RAID-Z2, backup on LTO, second backup on Disk), but over the years i saw too many disks die. So I bought several Seagate SMR 8TB for an additional backup. They're cheap and ideal for backup purposes. :D

-Walter

 

[Arwen]

...
That's what i planned, too. I know it sounds a bit paranoid (RAID-Z2, backup on LTO, second backup on Disk), but over the years i saw too many disks die. So I bought several Seagate SMR 8TB for an additional backup. They're cheap and ideal for backup purposes. :D

-Walter

You are only paranoid IF they are not out to get you :).

Though I would not say the Seagate SMR 8TB disks are ideal for backups.
Backups are a write mostly usage, and writes on SMR disks being slower
due to re-allocation, (and defragmenation of free space), of the shingles.

But, if the user can tolerate the slower backups, they should be fine. Plus,
with ZFS, (or Linux's BTRFS), scrubs, it's actually possible to verify the SMR
drive's firmware is writing the data correctly. (One of ZFSs design goals.)

 

[Bigtexun]

I first started using ZFS under the FreeBSD 8 beta /because/ I was being paranoid when I found that yet another very expensive enterprise tape system was writing unreadable tapes.

But USB to a single 8tb disk doesn't sound very solid to me...
and slow...

How much data are you saving there? How long does it take?

My advice is to build a second freeNAS system, using raidZ3 (the paranoid level of ZFS ;) and learn how to sync the systems, and develop a method of using snapshots as an added precaution.

In my first ZFS FreeBSD system, I did this very thing. I could roll the backup back to any day, for any of my 300 systems being backed up. Backing up 300 systems took about 5 minutes once the initial backup had been made (the first backup took over a week). The sync is so fast that you can easily do it over a minimal internet connection to a remote site, so backups could be off site. I actually was not allowed to do a live off-site at that job, so instead we used a off-site service to swap drive sets stored in a padded pelican case. We had a large drive chassis, and one row of drives was rotated off-site every Monday when the courier arrived. the off-site location stored two cases of drives, bringing only one in each week for swap. We synced to the server, then it synced to the rotation drive set. this was cheaper than tapes, cheaper than the enterprise-grade crappy hardware/software, and rock solid. And backups were granular by the day, for recent dates, and as the dates got older they were reduced to weeklys and monthlys... It's too bad the snapshot support in FreeNAS doesn't allow a mix of daily to weekly to monthly snapshot aging... That way
I can do daily snapshots, and thin the space consumed for the older snaps...

A single drive will work fine using this method, and really it won't be so slow once the first backup is made. But if you were really paranoid, you would want more protection than a single external drive...

Paranoid is good when the subject is backups. You are NOT paranoid enough. Said by someone that is so paranoid as to backup his backups of backups using snapshots and excessive redundancy... Can you be excessively redundant and paranoid with your backups? It's not excessive if it is so cheap... I replaced an $80k backup system with a $5k box and a couple pelican cases... with greater capacity, speed, security, and granularity. Just no fancy gui.

 

[Majo]

But USB to a single 8tb disk doesn't sound very solid to me...
and slow...

Yes, well, there are better solutions. As you wrote, with a second FreeNAS system you're more flexible of course. And as you and Arwen pointed out, SMR will be slow when the disk begins to fill up. But for the price of one additional FreenNAS (~1500-2000$) i can buy 8-10 8 TB archive disks (~200$ each). That's what i meant by "ideal for backups": The Seagate SMR disks are dirt cheap. You can't beat the price per MB and with several rotating archive disks you got plenty of redundancy. Right now i'm testing with only one disk but i plan to add rotating backups once i got it running and understood all the ZFS import/export stuff. :) Scripting this is just a small step then.

How much data are you saving there? How long does it take?

Yesterday i plugged in the SMR archive disk via USB 3.0 (no more SATA lanes on the board), configured a single disk ZFS and configured replication tasks for most of the data sets. Initial replications wrote with a constant rate of 120 MB/s. I replicated 5.6 TB of data and the backup disk is at 76% now:

[root@freenas] ~# zpool list backup
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
backup 7,25T 5,58T 1,67T - 55% 76% 1.00x ONLINE /mnt

The rate of 120 MB/s was constant during the whole replication. I can live with this, especially since the data doesn't change much and the snapshots tend to be small. Naturally, the disk will be much slower when it fills up and when it doesn't write a continuous stream due to the SMR design, but i can absolutely live with that.

Tested with a Supermicro board, Xeon 1230 (Haswell) and FreeNAS 9.10.

My advice is to build a second freeNAS system, using raidZ3 (the paranoid level of ZFS ;) and learn how to sync the systems, and develop a method of using snapshots as an added precaution.

Three words: Way too expensive. For these backups, speed isn't an issue. I can absolutely live with a downtime of an hour until i mount one of the backup disks, find the correct snapshot and restore the data. Reliability is an issue here. And with several rotating disks with 4 weeks of ZFS snapshots on each i have at least the reliability of a second Z3 system. But granted, if downtime is an issue, you're right. Plus, with your setup you can script nearly everything, which i can't.

In my first ZFS FreeBSD system, I did this very thing. I could roll the backup back to any day, for any of my 300 systems being backed up. Backing up 300 systems took about 5 minutes once the initial backup had been made (the first backup took over a week). The sync is so fast that you can easily do it over a minimal internet connection to a remote site, so backups could be off site.
...
padded pelican case.
...

Ok, that setup is enterprise level. I'm talking about 20 TB data on two FreeNAS systems. Not quite the same :)

But yes, i know what you mean. In 20 years of IT i saw all kinds of backup strategies, disk/raid crashes, failed recoveries etc. And i'm convinced that rotating off-site media (disks or tapes) is the best solution regarding data security. I'm not a big fan of incremental backups either, but most of the time the backup media are way too small and you have to use incremental backups. But for a small FreeNAS this isn't necessary, so i still think rotating full-backups on single disks is the way to go here.

A single drive will work fine using this method, and really it won't be so slow once the first backup is made. But if you were really paranoid, you would want more protection than a single external drive...

Yes, with a "single external drive" i'd have a bad feeling, too. :)

Paranoid is good when the subject is backups. You are NOT paranoid enough. Said by someone that is so paranoid as to backup his backups of backups using snapshots and excessive redundancy...

I'm mostly with you here. At work i didn't have a serious crash yet, but on my private computer i lost a lot after 2 or 3 disk crashes ("ah, damn crash, but that won't happen again. What are the chances that it will happen again?"). I lost things like personal emails from Terry Pratchett, photo galleries, etc - so i learned the hard way, like most of us. That was in the 90s and now i'm old, grumpy and hopefully a little wiser. :)