USB Hard Drives for BackUP

Status
Not open for further replies.

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
i read zfs single drive is pointless since there is no redundancy,
It's not pointless, it just doesn't take advantage of everything ZFS can do.
 

Majo

Dabbler
Joined
Apr 23, 2013
Messages
14
Quoting myself:

... I lost things like personal emails from Terry Pratchett, photo galleries, etc - so i learned the hard way, like most of us. That was in the 90s and now i'm old, grumpy and hopefully a little wiser.

In the back of my mind i had a Pratchett quote about backups. Just looked it up again (quote is from the 90s aswell):

Currently there's five machines permanently networked here. They all contain the serious core stuff. A couple of the machines are pensioned off 486s, with little other value now. Plus there's two Jaz drives in the building and the portable also carries a fair amount of stuff. Plus every Friday a man comes around and carves all the new stuff onto stone slabs and buries them in the garden... I think I'm okay.
-- (Terry Pratchett, alt.fan.pratchett)​

If you don't know the man, go ahead and buy his books. :D He died a year ago. :(
 

Bigtexun

Dabbler
Joined
Jul 17, 2014
Messages
33
So what I described is a lot like incremental backups, but it is must more like a full backup, minus the time spent transfering data that has already been transfered. And if you want to skip building another FreeNAS system, you can create Zx filesystem using any sort of drive attachment /including/ USB. Some of my earliest testing on FreeNAS was done with a usb hub with a lot of ports, and flash sticks.

Incremental backups have their place, and they can be a pain to restore from. But if you can do file by file backups rather than system images (this works for some backup applications and not others) you can get the speed of incrementals, with the security of full backups. Basically if you use rsync, it updates the backup by only transfering the files that change. That is not an incremental, it is a full backup, but with less data moving. AND if you snapshot your backup volumes between rsync operations, the backup volume with have copies of the various states of your system at the time of each backup.

For example, let's say you have 3T of data in use. You have a 4T backup drive. The data only has small changes day to day. Let's assume that you have 100MB of data change every day (not really that small, but this is an illustration). I will also assume that you keep the backup drive less than 90% full. I will also assume that the 100MB of change is sometimes added data, sometimes changed data, so your original storage does not grow very fast.

So first you back up the full data set, and you have 3T in use on the backup drive. You generate a snapshot on the drive. All the snapshot does is create a structure where you can see all of the files from the moment you made the snapshot.
The next day you sync the data to the backup disk, it takes a minute or so, and you generate a new snapshot. So now you have two snapshots that are a full copy of the data from both backups, but instead of taking up 6T of disk space, it takes up 3T plus the 100MB of changes. You do this every day for a year. At the end of the year you are using 3.0365T of space on the backup drive, but you have 365 different views of your data set, one for each time you produced a snapshot. You can go back to any day of that year, and look at the data as it was on that day, and instead of taking up 1095 T of disk space, it is taking up 3.0365T of space... barely larger than the original data set, and you have a record of every day's worth of change. You can roll back to any "full backup" without doing a dance of restoring a complex incremental.

Obviously this does not work with system images, as in a system image the entire dataset is treated as a single ever-changing blob of data. But this is not an incremental, it just resembles one. It is 365 full backups, taking seconds of time each, giving you a lot of access to every days worth of change. Obviously this example assumed big chunks of the filesystem aren't changing. But you can accommodate larger change deltas if you buy several USB drives, and build a ZFS volume from them.

My point is that you can still do what I did without a second FreeNAS system, and without anything more than the "cheap" drives you planned to use in the first place... What I did applies just as well to buying individual drives.

And FreeNAS backup servers don't cost any more than you make them cost for performance reasons. One of my FreeNAS systems is built into a machine I was going to litterally throw away. I did add some hardware to it to support a stack of drives, but for low performance that could be a $100 LSI9211-8i clone and 8 cheap SATA drives. If you pop open an external USB drive, you are likely to see a SATA drive, with a cheap SATA to USB adapter. The LSI9211-8i controller can handle 8 drives as-is, or 64 drives if you buy a couple of SAS expanders... SAS expanders are the only pricy part, a 32 drive expander costs about $380 from amazon. That is barely more than $10 per drive, about the cost of that slow USB adapter on the 8T drives you have.

I spent $5000 on one FreeNAS system using really sweet parts, and one was a throw away machine with about $480 worth of parts to give it access to up to 32 drives... that is more drives than I can fit into the box, but the box held 12 drives using a pair of cheap 6-drive 2.5" bays... and I'm using cheap drives, and it can be easily expanded if I get creative on how to hold more drives.

I know I talked about big "enterprise" things, but I was using commodity drives. You should learn to use ZFS scrubs, SMART tests, and the output of smartctl to manage your cheap drives to stay ahead of failing drives, but it isn't too hard. I'm the king of large arrays of dirt cheap drives. Keep your drives cool, keep them scrubbed (which does take I/O speed to do it very often).

So my iSCSI "SAN" is a $5000 FreeNAS box. But my other FreeNAS box is yesterdays trash, only I spent some money on drives. Now I went cheap on drives there, $20 each for 2.5" 1T drives. That price was because I bought a bunch of them. So far I'm 2+ years into using them and haven't lost one yet, but I have spares and RaidZ3 combined with scrubs and smart to keep the data whole. Eventually I'll need to buy more drives, and by then the cheap drives will be 4T or 8T. I bought more drives than I needed because I was expecting them to fail frequently. Surprise! They aren't failing yet. I'll probably upgrade them later for capacity reasons rather than failures.

What I do can cost a lot, or cost very little. I paid slightly less for storage volume than your 8T drives cost. My drives are cheap, slow, junk drives off of Ebay... this time. I have actually done this several times over the years, and the only time I have a problem is when I screw up and fill a backup disk to 100%. When I hit 100%, the data on the drives is ok, but the boot flash of the FreeNAS system has problems and needs to be reinstalled. I had the same problem under FreeBSD.
 

gdreade

Dabbler
Joined
Mar 11, 2015
Messages
34
Status
Not open for further replies.
Top