Offsite backup suggestions

[NASbox]

I'm hoping someone with experience can give me some guidance/point me in the right direction (sorry if I'm using the right terminology, but I hope my background info/question will still make sense.)

Background:
I am a home user, running FreeNAS 8.3-RELEASE-p6 and has a RAIDZ2 pool (4x6TB Disks) with a total of about 4TB of data (3 data sets), and a RaidZ pool (2x4TB Disks) with about 3TB of data (1 data set). (When I get the time, I plan to upgrade to v11, replace the processor and add another 16GB of memory when I get a chance so that I can run a couple of jails.)

After discussions with the community my plan is to do off-site backups using a removable disk caddy (Hot Swap SATA3). If I understand correctly, I should be able to create a single volume ZFS Pool, mount the pool, make my backup and then export the pool and remove the disk from the system for safe keeping. I can import the volume to update the backup and/or restore data.

My plan is to create one large single volume ZFS Pool on an 8TB removable drive and then do an rsync to copy the data from the production pools. I know I will eventually need to use 2 drives, but I want to avoid that until the backup won't fit anymore.

I don't know much about replication... if that's even a option... or if it makes sense given that I will want to efficiently update the backups. I was using rsync to external UFS formatted USB drives. The USB option while not great was workable for a home environment. I don't have the space for a second system to use as a backup, and UFS over USB is not a viable option for v11. I will be relying on backups made using this technique when I upgrade from v8.3 to v11.

Questions:
Should I just go with the rsync option, or is it worth my time/effort to research/learn about some other form of replication?
If so, can someone point me in the right direction so I can do some reading.

How should I set up the backup pool... I don't know if I did something wrong, but for some reason I ended up with swap partitions (slices?) on my USB drives... I'd like to avoid that if possible.

Thanks in advance for any input/assistance.

 

[lopr]

• If you just need an occasional incremental backup of some files I would do a manual rsync.
• both the rsync and the replication tasks in the freeNAS GUI are more suitable for automated backups.
• zfs replication (zfs receive/send) can only deal with snapshots, not with live data. IMHO zfs send/receive is great for a one time backup (done manually) or automated recurring backups (via replication tasks and snapshot tasks). It is also great to retain your filesystem structure, especially if you have many nested datasets. If you use replication tasks for replicating a whole pool not a dataset, be aware that there is currently a bug in freeNAS that does create a special dataset that is not accessible that will give you error messages (https://bugs.freenas.org/issues/23871).
• AFAIK freeNAS adds the swap space when you add a disk, so you'd have to add the disk via command line if you don't want that see below

 

[Chris Moore]

I do backups in a way that you might want to try. It is very similar to what you are looking for. When I get to my office, I will be able to share more details with you. Short answer, it can totally be done.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[danb35]

I don't know if I did something wrong, but for some reason I ended up with swap partitions (slices?) on my USB drives...

Yes, FreeNAS does that automatically (by default) when you create a pool.

I'd like to avoid that if possible.

Why? Why does it matter to you if there's a 2 GB swap partition on the disk? But it's easy enough to do if you want; just set the swap partition to 0 in the web GUI.

 

[danb35]

AFAIK freeNAS adds the swap space when you add a disk, so you'd have to add the disk via command line if you don't want that

Not at all; there's a GUI setting for the size of the swap partition. Set that to 0, and none will be created.

 

[Chris Moore]

Not at all; there's a GUI setting for the size of the swap partition. Set that to 0, and none will be created.

I would not do that, out of an abundance of caution, I just wouldn't worry about 2GB (a tiny fraction) of a multi Terabyte drive. It just isn't worth the potential problems down the road. Preserving your data is paramount.

 

[danb35]

It just isn't worth the potential problems down the road.

What "potential problems"? I agree that 2 GB out of an 8 TB drive is lost in the noise (1/40 of 1%), but I also don't see any problem in recreating the pool--especially a single-disk pool--without it.

 

[Chris Moore]

I'm hoping someone with experience can give me some guidance/point me in the right direction (sorry if I'm using the right terminology, but I hope my background info/question will still make sense.)

Background:
I am a home user, running FreeNAS 8.3-RELEASE-p6 and has a RAIDZ2 pool (4x6TB Disks) with a total of about 4TB of data (3 data sets), and a RaidZ pool (2x4TB Disks) with about 3TB of data (1 data set). (When I get the time, I plan to upgrade to v11, replace the processor and add another 16GB of memory when I get a chance so that I can run a couple of jails.)

Thanks in advance for any input/assistance.

When you upgrade to version 11, are you going to reconstruct your storage pool by off-loading your data onto the backup device and bring the data back once the pool is reconfigured? I ask that because the way you have your storage pool configured is less than optimal.

Anyhow, the solution I use to backup my storage involves some hardware and you might want to have a look at it. It has been working fantastically for me since I got it over three years ago.
I use this device to hold a set of drives: https://www.newegg.com/Product/Product.aspx?Item=N82E16817182327
That product is built pretty well and I have not had any difficulty working with it. When I first started using it, I put 4 x 3TB drives in it and I just this year upgraded to 4 x 5TB drives because I needed more space. I set the drives up as a RAID-z1 pool and import the pool when I connect the device and export the pool when I disconnect the device. I use eSATA to connect it and I will get a link to the card I use in a moment but the nice thing about it for me is that the eSATA can be plugged in while the NAS is running and disconnected the same way. No shutdown or restart needed. Just plug it in, wait for FreeNAS to detect the drives, import the pool, run rsync to update the backup, export the pool and unplug the enclosure. Done. I have been using it that way for at least three years and I make a backup every weekend.
The interface card I use is this: https://www.newegg.com/Product/Product.aspx?Item=N82E16816124063
I originally tried a different controller and it worked under Windows, but would not work properly in BSD / FreeNAS. This one works. I liked it well enough that I went back and bought one for each of the three systems I had at the time so I could use this to move data between my NAS units and that works too. I made a backup on one NAS and restored that backup on the other NAS. Importing and exporting the pool is smooth and easy even though it is a multi drive set. The only thing you have to watch out for is that you need to set the mount point. That is a pool property that is not set when the pool is created by FreeNAS so when you import the pool the next time, it will show up in the / directory. There is a zpool command to set the mount point and once you do that, export the pool and re-import the pool and it should remember where to mount in the future.
The Rosewill product says it supports drives up to 5TB so if you think you need bigger drives, you might try this product which says it is tested with drives up to 6TB.
https://www.newegg.com/Product/Product.aspx?Item=N82E16817707364
I have not used the StarTech unit but it looks very similar except I believe that it just has slots for the drives with no need to use trays.

 

[Chris Moore]

What "potential problems"? I agree that 2 GB out of an 8 TB drive is lost in the noise (1/40 of 1%), but I also don't see any problem in recreating the pool--especially a single-disk pool--without it.

For a single disk, it probably wouldn't be an issue, ever, I was thinking that if you leave the GUI setting at 0 and add another vdev / pool, that it would have no swap space.

I don't know if it is true, but I was told by someone a while back that the swap space on the drive also provides some "slop" so that if a replacement drive is not the exact same size (smaller) as the drive being replaced, the replacement can still work because the data space can be the same and the difference can come out of the swap space. Really only applicable to replacing drives in a multi-drive vdev.

 

[danb35]

I don't know if it is true, but I was told by someone a while back that the swap space on the drive also provides some "slop" so that if a replacement drive is not the exact same size (smaller) as the drive being replaced, the replacement can still work because the data space can be the same and the difference can come out of the swap space.

Yes, I understand that to be a big part of why FreeNAS does swap the way it does. I also understand that it's no longer relevant, as drive capacities are much more standardized than they once were--a 4 TB drive from WD, for example, should have exactly the same capacity as a 4 TB drive from HGST.

 

[Chris Moore]

Yes, I understand that to be a big part of why FreeNAS does swap the way it does. I also understand that it's no longer relevant, as drive capacities are much more standardized than they once were--a 4 TB drive from WD, for example, should have exactly the same capacity as a 4 TB drive from HGST.

Like I said, it is out of an abundance of caution on my part. I have reduced my setting from 2 to 1, but I don't feel safe going to 0.
I run ZFS on Linux at work and I think there is a little space that is left as a buffer on them by default. The swap space on that server is all on another set of disk.

 

[Stux]

When you upgrade to version 11, are you going to reconstruct your storage pool by off-loading your data onto the backup device and bring the data back once the pool is reconfigured? I ask that because the way you have your storage pool configured is less than optimal.

Anyhow, the solution I use to backup my storage involves some hardware and you might want to have a look at it. It has been working fantastically for me since I got it over three years ago.
I use this device to hold a set of drives: https://www.newegg.com/Product/Product.aspx?Item=N82E16817182327
That product is built pretty well and I have not had any difficulty working with it. When I first started using it, I put 4 x 3TB drives in it and I just this year upgraded to 4 x 5TB drives because I needed more space. I set the drives up as a RAID-z1 pool and import the pool when I connect the device and export the pool when I disconnect the device. I use eSATA to connect it and I will get a link to the card I use in a moment but the nice thing about it for me is that the eSATA can be plugged in while the NAS is running and disconnected the same way. No shutdown or restart needed. Just plug it in, wait for FreeNAS to detect the drives, import the pool, run rsync to update the backup, export the pool and unplug the enclosure. Done. I have been using it that way for at least three years and I make a backup every weekend.
The interface card I use is this: https://www.newegg.com/Product/Product.aspx?Item=N82E16816124063
I originally tried a different controller and it worked under Windows, but would not work properly in BSD / FreeNAS. This one works. I liked it well enough that I went back and bought one for each of the three systems I had at the time so I could use this to move data between my NAS units and that works too. I made a backup on one NAS and restored that backup on the other NAS. Importing and exporting the pool is smooth and easy even though it is a multi drive set. The only thing you have to watch out for is that you need to set the mount point. That is a pool property that is not set when the pool is created by FreeNAS so when you import the pool the next time, it will show up in the / directory. There is a zpool command to set the mount point and once you do that, export the pool and re-import the pool and it should remember where to mount in the future.
The Rosewill product says it supports drives up to 5TB so if you think you need bigger drives, you might try this product which says it is tested with drives up to 6TB.
https://www.newegg.com/Product/Product.aspx?Item=N82E16817707364
I have not used the StarTech unit but it looks very similar except I believe that it just has slots for the drives with no need to use trays.

You should make sure to regularly scrub those drives...

 

[NASbox]

Thanks everyone for the wonderful replies...

Yes, FreeNAS does that automatically (by default) when you create a pool.
Why? Why does it matter to you if there's a 2 GB swap partition on the disk? But it's easy enough to do if you want; just set the swap partition to 0 in the web GUI.

My reason was not to save the disk space. My concern (maybe out of ignorance) is that FreeNAS might actually use the drive to swap something to and I might have difficulty exporting/unmounting it.

Not at all; there's a GUI setting for the size of the swap partition. Set that to 0, and none will be created.

Did I miss something, or is this just available on the latest versions? As I mentioned I am currently on FreeNAS 8.3-RELEASE-p6.

When you upgrade to version 11, are you going to reconstruct your storage pool by off-loading your data onto the backup device and bring the data back once the pool is reconfigured? I ask that because the way you have your storage pool configured is less than optimal.

I wasn't planning to recreate the pools unless I need to because it's a lot of work (and if I screw up really badly I could lose data). My thinking as to configuration was that I use a different RaidZ level depending on the amount of redundency that I need, and the larger a pool gets the more difficult it is to work with for backup/recovery etc. That said, keeping in mind my objectives I'm very open to suggestions. (When I need more space I'll likely add 2 more 6TB drives to the RaidZ2 (if that's possible) rather than going with larger drives. After my rebuild I will have available SATA ports and space in the case. If I understand correctly 6 drives is much better for RaidZ2.)

Anyhow, the solution I use to backup my storage involves some hardware and you might want to have a look at it. It has been working fantastically for me since I got it over three years ago.....
The interface card I use is this: https://www.newegg.com/Product/Product.aspx?Item=N82E16816124063
I originally tried a different controller and it worked under Windows, but would not work properly in BSD / FreeNAS. This one works. I liked it well enough that I went back and bought one for each of the three systems I had at the time so I could use this to move data between my NAS units and that works too.

Thanks for taking the time to document this... Reasonable cost backup solutions with FreeNAS are not easy to come by. If I'd had this suggestion earlier I might have gone this way... I ended up with an LSI HBA... The consensus in the the forum was that was the way to go, so instead of just worrying about backup, I decided to really build out my FreeNAS... big case, faster processsor, more memory, more drives, multiport NIC as it would allow me to really improve my backups, and run a jail or two. I need automated workstation backup and FreeNAS is the easiest way for me to get that, and my current box doesn't have the storage capacity. When we get docker running/stable, it would also be helpful to run a couple of applications and get them off my daily use PC as well.

Anyhow, the solution I use to backup my storage involves some hardware and you might want to have a look at it. It has been working fantastically for me since I got it over three years ago.....
I use this device to hold a set of drives: https://www.newegg.com/Product/Product.aspx?Item=N82E16817182327
The Rosewill product says it supports drives up to 5TB so if you think you need bigger drives, you might try this product which says it is tested with drives up to 6TB.
https://www.newegg.com/Product/Product.aspx?Item=N82E16817707364
I have not used the StarTech unit but it looks very similar except I believe that it just has slots for the drives with no need to use trays.

Are you using multiple drives in that box, or just 1? If so do the multiple drive show up as separate drives or just one large drive (JBOD?)? When I was researching solutions I came to the conclusion (and maybe I was wrong), that multi-disk external drive enclosures would not allow FreeNAS to access the individual disks. My concern with this was much the same as using hardware raid 0 (a really bad idea)... if the hardware dies, the pool may be unrecoverable, and the failure of one disk will destroy the whole set.

After a lot of searching I decided to go with a removable tray from Addonics (I'll provide an update once I've tested to make sure things work as expected), but from what I can see the build quality of the units is great, and they are one of the few solutions that allows frequent removal/reinsertion of a SATA drive (Those PCB fingers on SATA drives have a rated life of about 125 insertion removal cycles, so they aren't meant to be used like a data cartridge!!!) The addonics unit has a connector that bridges the drive to the caddy thus taking the wear and tear off the drive.

If things work as expected, I should be able to just shove a drive into the front of the case, mount the pool and run a backup, and then unmount (export) and remove the drive. Also no concern about drive size. It the HBA can accept the drive, then the drive will work. Almost like a Tape cartridge on steroids... way faster and fast random access.

Hopefully hot swap works properly in FreeNAS/FreeBSD. (Anyone have experience with this? I'm planning on using an LSI SAS2008-8I PCI-E HBA Controller Card which I recently bought off eBay.)

 

[danb35]

Did I miss something, or is this just available on the latest versions? As I mentioned I am currently on FreeNAS 8.3-RELEASE-p6.

See page 78 of the manual.

My concern (maybe out of ignorance) is that FreeNAS might actually use the drive to swap something to and I might have difficulty exporting/unmounting it.

Understandable, but not a problem--FreeNAS will swapoff that swap partition before exporting the pool.

 

[Chris Moore]

You should make sure to regularly scrub those drives...

I scrub them and do a smart long test on them as part of my weekend backup process.. :)

 

[NASbox]

See page 78 of the manual.

Thanks I missed swap size in the manual.

Understandable, but not a problem--FreeNAS will swapoff that swap partition before exporting the pool.

Good to know about the automatic swapoff.

The only question that remains is should I get rid of that swap space? 2GB out of 8TB isn't worth worrying about, and it does provide a cushion for size differences if I need to clone the drive to another unit that is slightly smaller or if for some reason this was the only pool mounted (say during recovery after hardware failure).

Is my reasoning valid or are these non-issues?

Just wondering - does anybody know how FreeNAS decides where swap goes when there are lots of different disks and multiple pools?
I assume that the swap space is just "raw" block space like Linux and that there is no redundancy.
I don't think my current install ever swaps, but jails/VMs may well change that.

 

[NASbox]

I scrub them and do a smart long test on them as part of my weekend backup process.. :)

How many drives? Does FreeNAS allow access to the individual disks in that unit, or is it just one large block of storage?

 

[Stux]

Thanks I missed swap size in the manual.


Good to know about the automatic swapoff.

The only question that remains is should I get rid of that swap space? 2GB out of 8TB isn't worth worrying about, and it does provide a cushion for size differences if I need to clone the drive to another unit that is slightly smaller or if for some reason this was the only pool mounted (say during recovery after hardware failure).

Is my reasoning valid or are these non-issues?

Just wondering - does anybody know how FreeNAS decides where swap goes when there are lots of different disks and multiple pools?
I assume that the swap space is just "raw" block space like Linux and that there is no redundancy.
I don't think my current install ever swaps, but jails/VMs may well change that.

If the swap is in use, and the drive fails, your system will probably crash....

 

[danb35]

does anybody know how FreeNAS decides where swap goes when there are lots of different disks and multiple pools?

Every disk, by default, gets a 2 GB swap partition at the beginning.

I assume that the swap space is just "raw" block space like Linux and that there is no redundancy.

Correct, though I think FreeNAS is moving to mirrored swap in an upcoming release (maybe it's planned for 11.1?).

 

[Chris Moore]

I am a home user, running FreeNAS 8.3-RELEASE-p6 and has a RAIDZ2 pool (4x6TB Disks) with a total of about 4TB of data (3 data sets), and a RaidZ pool (2x4TB Disks) with about 3TB of data (1 data set).

The thing you have here, in the RAIDZ2 with four drives, effectively, two drives are storage and two drives are redundancy, that is not very space efficient. Normally, you would want to have a minimum of five drives (obviously you can do it with four) but it works better with five or six or more up to around 10 but ZFS will let you do even more. The way you have it, this should only be giving you about 10.2TB of usable space
Then with the RAIDZ pool, effectively, one disk is data and the other is parity so it is working like a mirror. Normally with RAIDZ you would want to have three drives at the minimum and again it will let you do what you did, but that is not optimal. The way you have it, this should only be giving you about 2.6TB of usable space
You can't add more drives to an existing vdev, but you could add another vdev to an existing pool. I would suggest that if you are interested in adding more drives, that you should copy all the data off your device, destroy the existing pool and create a new pool using six 6TB drives at RAIDZ2, which would give you about 20TB of usable storage and that might allow you to totally eliminate the 3TB drives and simply create datasets inside your main storage pool for everything to be.
This would be more economical of the storage space and power consumption would be more efficient as you would be spinning two 6TB drives instead of two 3TB drives.

Are you using multiple drives in that box, or just 1? If so do the multiple drive show up as separate drives or just one large drive (JBOD?)? When I was researching solutions I came to the conclusion (and maybe I was wrong), that multi-disk external drive enclosures would not allow FreeNAS to access the individual disks. My concern with this was much the same as using hardware raid 0 (a really bad idea)... if the hardware dies, the pool may be unrecoverable, and the failure of one disk will destroy the whole set.

I put four 5TB drives in the external enclosure and they each show up as separate drives to FreeNAS. It works as if each drive were directly connected to a SATA port and as I said, I have been using this for years with no trouble. I created a RAIDZ1 pool on those four drives which gives me about 12TB of storage. When I connect it to make my backup I can do a zpool import just the same way you were talking about doing with a single drive. It works just the same, you just have to wait for FreeNAS to detect all four drives. Then I run my rsync from the command line that synchronizes the data on my NAS such that any changes are copied to the external pool. Once I have done all the operations that I have in mind to do, like scrubbing the pool and smart tests of the drives etc. Then I can do the zpool export and shutdown the external enclosure.

I am confused by why you bother to ask a question when you have not only already decide on something else, but it sounds like you have already procured the hardware to implement that solution.

Hopefully hot swap works properly in FreeNAS/FreeBSD. (Anyone have experience with this? I'm planning on using an LSI SAS2008-8I PCI-E HBA Controller Card which I recently bought off eBay.)

This is the same model SAS controller that I use and it works fine with hot-swapping drives.