Backup Dataset to USB HDD (local)

[Thund3rDuck]

Good afternoon and happy Saturday to you all.

In part of my setting up my FreeNAS box, one of my main concerns is backing up its Data. I do not have the funds nor the availability of additional hard drives for another FreeNAS install to rsync to. The only alternative it seems is to backup to an external hard drive via USB. If there is another alternative please enlighten me. My end goal would be to backup to one drive locally (once or twice a day) and then backup to another external drive ($$) monthly and keep it offsite. As of right now I'd like to just setup the daily backup so I at least have that.

I know I probably could figure out how to set it up with rsync in the FreeNAS shell and then schedule it via cron, but I would prefer to make sure I know how to do it in the GUI as well. Please keep all solutions in regards to setting it up in the GUI.

The second part of that question would be how would I restore from that USB hard drive?

Thank you all for your assistance.

 

[DrKK]

Let me tell you what I do, since it may be what you're after.

I have a FreeNAS with about 4TB of space on it. I have a Windows box on the LAN, which has an appropriate external hard drive on USB on it. I run a program *ON THAT WINDOWS BOX* called "syncbackfree" by 2brightsparks software (a very elegant piece of software btw), and every night at 1230AM, it simply "mirrors" anything that's on the NAS to itself (because I have a read-only windows share set up on the NAS, so that's just another network drive to Windows). Then, every so often, I have *THAT* external hard drive mirrors to an off-site hard drive.

This way of doing hard-core backups may not be elegant, but it alleviates any necessity of messing around with rsync's, or any other services in FreeNAS.

 

[gpsguy]

I've performed backups using a method similar to DrKK's, as well as a rsync script local to my FreeNAS box. Since you've said that you're not interested in a non-GUI solution, I won't post the script I use.

If you are a Windows user, a benefit of using a solution like DrKK suggested (I use Robocopy), is that the backup is on a NTFS formatted disk and easily accessible from Windows, should your FreeNAS box fail. I limped along in that mode for awhile last year.

FreeNAS + USB hard disks, might be a bit dangerous. If external, is eSATA available? You'll need to use either UFS or ZFS with FreeNAS, which might limit your ability to easily access the data in an emergency.

If you needed to restore from your USB drive (backup), it depends on the situation. I would recommend that you backup your FreeNAS configuration on a regular basis. You can do it manually from the GUI, or automate with a cron job.

Restoring from your backup would depend on the situation. It could be as simple as reinstalling FreeNAS along and your configuration file. Or, copying / rsync'ing data from a backup.

 

[DrKK]

Restoring from your backup would depend on the situation. It could be as simple as reinstalling FreeNAS along and your configuration file. Or, copying / rsync'ing data from a backup.

Right. My intention with *my* setup, in the event of a failure, is to either restore a small number of bad files, or to wipe out the NAS, get a fresh install, and transfer everything over from the external HDD on the Windows box. That's what I figure. I thought setting up a "pull" mirror *FROM* the FreeNAS by the Windows box was just easiest.

BTW gpsguy, I haven't seen "robocopy", but definitely this "syncback free" from 2brightsparks is on my top 10 list of most elegant, and most useful, and most polished, software for Windows that I've seen. I heartily recommend it.

 

[robotmonkeys]

Right. My intention with *my* setup, in the event of a failure, is to either restore a small number of bad files, or to wipe out the NAS, get a fresh install, and transfer everything over from the external HDD on the Windows box. That's what I figure. I thought setting up a "pull" mirror *FROM* the FreeNAS by the Windows box was just easiest.

You're doing what I want to do. Sure it works, but it's excruciatingly slow. It's a terrible hack that comes down to, "I'm using two computers connected across a network, because it's nigh-impossible to use the local bus on my NAS." Sure for a few files it's fine, but for a major failure it's just inefficient. I know that this forum's opinion of USB is that it's a horrible technology for horrible people because ZFS doesn't play well with it, and anyway "USB drives are not portable", and that the proper way to backup a NAS is through dedicated fiber lines to climate controlled mountains in Svalbard Norway and Inchon Korea. (Because with sea level rise and super volcanos these days...) Or if you insist on having your backup, then you should drop another $1500 for a second NAS to keep on a shelf at work. But that's just frankly overkill for home use. For home, monthly backups kept across the street are fine.

I really thought syncing to a local external drive would be easy.

 

[DrKK]

Excruciatingly slow?

Not really. Takes a few hours to do the initial mirroring, then nightly, it takes about 1-3 minutes to back all of the new/changed files up. At least in my use case.

And the USB drive is plugged INTO A WINDOWS BOX, not into a FreeNAS. USB drives do not play nicely with FreeNAS for many cases.

 

[cyberjock]

Well, USB drives really aren't recommended for critical data at all. You can never run SMART tests on the drives on any OS, and you can't monitor SMART parameters. So its pretty much impossible to really see how healthy your disk is until it starts having problems. And generally, if you have any problems the first indicator is the disk disconnecting and reconnecting from the host.

So in a hypothetical situation you lose your primary copy. So you start copying all of the data off your USB drive back to your new pool on your server. And partway through the copying it may fail because the disk is bad. So how do you skip the bad part? Well, for starts lets hope you even know what file had the error. If you do and you can easily skip it, then great. If not, well, you are in for some serious trouble.

But do you think your disk is going to have just one error? Probably not. So you'll have tons of files that you'll have to skip. And each time your disk disconnects you'll have to unplug the power supply from the USB drive, plug it back in and wait for it to reconnect. Then restart the transfer while skipping the file it was working on.

Sounds like hell to me. Not something I want to get involved with except in my enemy's nightmares.

This is why nobody who actually does file servers professionally uses USB.

 

[Dusan]

You can never run SMART tests on the drives on any OS, and you can't monitor SMART parameters.

I see this mentioned a lot on the forum. It may have been true in the past, but the situation is improving quickly. At this moment I have three different USB HDDs (fairly new, the oldest is ~2 years old) within my reach and smartctl is able to work with all of them both in Windows & FreeBSD. Almost all of the current USB SATA chips support the SCSI/ATA Translation pass-through standard (-d sat) + smartmontools also supports ATA pass-trough for three older chips (usbcypress, usbjmicron, usbsunplus). BTW, all of my drives (three different manufacturers) support the SAT standard. Also, FreeBSD is one of the three OS'es with best USB support in smartmontools: http://sourceforge.net/apps/trac/smartmontools/wiki/USB

 

[cyberjock]

I have yet to see a single USB drive work with smart tools in any OS personally. And every person I've talked to that had an external disk couldn't run smartmontools either. I'm not sure what is different for your situation than mine. It could be that the people that sell in the USA are cutting corners wherever they can and not where you are.

There is also a slight different between the store-bought external USB drives and the USB->SATA converters you can buy. If you use the converters you might have better luck(I've never tried them with SMART tools).

But by far most people that talk about using external USB drives are talking about drives that are pre-built as external devices. The applicable manufacturer uses some converter chip to make that drive a USB drive, and they are looking at cost as they want to sell them for as cheap as possible.

 

[Dusan]

The three drives are all pre-built (WD Elements Portable, Transcend StoreJet and Seagate Expansion Portable). Try to add the -d sat option to smartctl and I think you will be pleasantly surprised (if that doesn't work, try the other three options).

 

[DrKK]

Just for the record, I run a lot of external drives that I made myself---i.e., I buy an internal drive, then I buy a decent enclosure---and I have 100% of those recognizing SMART via USB 2.0, 3.0, and eSATA, without any problem.

 

[Thund3rDuck]

Let me tell you what I do, since it may be what you're after.

I have a FreeNAS with about 4TB of space on it. I have a Windows box on the LAN, which has an appropriate external hard drive on USB on it. I run a program *ON THAT WINDOWS BOX* called "syncbackfree" by 2brightsparks software (a very elegant piece of software btw), and every night at 1230AM, it simply "mirrors" anything that's on the NAS to itself (because I have a read-only windows share set up on the NAS, so that's just another network drive to Windows). Then, every so often, I have *THAT* external hard drive mirrors to an off-site hard drive.

This way of doing hard-core backups may not be elegant, but it alleviates any necessity of messing around with rsync's, or any other services in FreeNAS.

I appreciate your response and I thought of doing that too with an rsync through the shell and then scheduling it. My only problem is that I do not want to use another machine (mac, windows or linux) to backup my NAS. I want it to be completely independent of any other systems.

 

[Thund3rDuck]

I've performed backups using a method similar to DrKK's, as well as a rsync script local to my FreeNAS box. Since you've said that you're not interested in a non-GUI solution, I won't post the script I use.

If you are a Windows user, a benefit of using a solution like DrKK suggested (I use Robocopy), is that the backup is on a NTFS formatted disk and easily accessible from Windows, should your FreeNAS box fail. I limped along in that mode for awhile last year.

FreeNAS + USB hard disks, might be a bit dangerous. If external, is eSATA available? You'll need to use either UFS or ZFS with FreeNAS, which might limit your ability to easily access the data in an emergency.

If you needed to restore from your USB drive (backup), it depends on the situation. I would recommend that you backup your FreeNAS configuration on a regular basis. You can do it manually from the GUI, or automate with a cron job.

Restoring from your backup would depend on the situation. It could be as simple as reinstalling FreeNAS along and your configuration file. Or, copying / rsync'ing data from a backup.

I'm an everything that works user. If you have a script that works well let me know. If using a script, how do I make sure that it actually backs up successfully? I don't want to compare file created/modify dates or compare volumes sizes on a daily basis. I also really don't want to use another system and rely on that chain backup and it crap out due to a bad link. I could possibly install an esata card if FreeNAS would recognize it and connect it to one of my esata dock's. Does E-SATA work that much better when connected to a FreeNAS box?

I would most likely make it another ZFS Volume and recover with rsync should the box die or whatever.

Thank you for the reminder to backup the config. I usually only do this when I make changes.

 

[Thund3rDuck]

You're doing what I want to do. Sure it works, but it's excruciatingly slow. It's a terrible hack that comes down to, "I'm using two computers connected across a network, because it's nigh-impossible to use the local bus on my NAS." Sure for a few files it's fine, but for a major failure it's just inefficient. I know that this forum's opinion of USB is that it's a horrible technology for horrible people because ZFS doesn't play well with it, and anyway "USB drives are not portable", and that the proper way to backup a NAS is through dedicated fiber lines to climate controlled mountains in Svalbard Norway and Inchon Korea. (Because with sea level rise and super volcanos these days...) Or if you insist on having your backup, then you should drop another $1500 for a second NAS to keep on a shelf at work. But that's just frankly overkill for home use. For home, monthly backups kept across the street are fine.

I really thought syncing to a local external drive would be easy.

My thoughts exactly. I don't have money for a new box nor bigger drives. I'm running 6 x 500GB drives in a Z1 just to get what I can out of the box. The thing is, if I want (and have the money) to spend 500 to 800 bucks on a QNAP device, I get the option to plug in a HDD via USB or E-SATA and back the damn thing up. The option to back up is also through the GUI, which gives you a clean look as to the status of the backup and works amazing if you need to restore. Just don't get why I can't do that with FreeNAS. Even if I had to do it from the shell, at least make it easier to do. I have a hard time believing that everyone using FreeNAS is rsync'ing to a duplicate box as a backup.

 

[Thund3rDuck]

Well, USB drives really aren't recommended for critical data at all. You can never run SMART tests on the drives on any OS, and you can't monitor SMART parameters. So its pretty much impossible to really see how healthy your disk is until it starts having problems. And generally, if you have any problems the first indicator is the disk disconnecting and reconnecting from the host.

So in a hypothetical situation you lose your primary copy. So you start copying all of the data off your USB drive back to your new pool on your server. And partway through the copying it may fail because the disk is bad. So how do you skip the bad part? Well, for starts lets hope you even know what file had the error. If you do and you can easily skip it, then great. If not, well, you are in for some serious trouble.

But do you think your disk is going to have just one error? Probably not. So you'll have tons of files that you'll have to skip. And each time your disk disconnects you'll have to unplug the power supply from the USB drive, plug it back in and wait for it to reconnect. Then restart the transfer while skipping the file it was working on.

Sounds like hell to me. Not something I want to get involved with except in my enemy's nightmares.

This is why nobody who actually does file servers professionally uses USB.

If using FreeNAS professionally, I wouldn't be using USB drives. Of course, I would have professional $$ to spend on a concrete solution. What would be the proper professional backup and recovery life cycle if using FreeNAS in a production environment? (This is an actual question.)

Also if I was having any kind of connectivity issues with the drive, I would pull it out of the enclosure and test it on one of my stations. While doing so I would got get another 1 to 3TB drive to begin another backup. The backup drive dying is not that big of a deal as long as you can get or already have another backup as part of your process.

 

[gpsguy]

Here's the script I use. I lifted it from one of pirateghost's messages. It's not polished, if I were running it from a cron job, I'd include a date/time stamp in the log filename.

Code:

rsync -rvuAth --log-file=/mnt/vol1/home/scripts/gpsguy.log /mnt/vol1/home/gpsguy/ /mnt/backup1/gpsguy

r - recursive
v - verbose
u - update
A - preserve ACLS
t - preserve modification times
h - output numbers in human-readable format

 

[TheSmoker]

I have a Seagate Expansion 3TB and it works just fine for me, using the following command;
smartctl -a -d sat,12 /dev/da1

Sent from my iPad using Tapatalk HD

 

[cyberjock]

Well, smartctl -a -d sat /dev/sdd (run from smartmontools for Windows) did work for my Kingston Hyper X(basically an Intel SSD in an external Kingston labeled box).

But it didn't work for my WD "something" 1TB. Not sure what it is since the sticker letters are rubbed off and I got no results from the smartctl.

 

[Thund3rDuck]

So... still looking for a solid backup for my NAS data without relying on another computer to do the backup. Thoughts?

 

[TheSmoker]

I am using 2 external 3tb usb drives so i can have 2 external backups. Practically i am backing up the same thing to two different drives.

Sent from my iPad using Tapatalk HD