Backup spanning multiple external drives

[ovig]

I’d like to take a backup of my datasets to external HDDs for safe keeping offsite. I know there are several well documented ways of doing that (rsync; zfs send | zfs receive).

Has someone got first hand experience to share of doing that with a size of dataset much larger than the external disks, i.e. the backup spanning two or more disks: my datasets are typ. 3-5TB and i have a bunch of RDX disks (500GB-1TB) which i'd like to use, but can't just get my head around the spanning/disk change?

If at all possible i'd like to. have the disks plugged in one by one both. at creation and during backup. - and have an easy way of doing incremental/differential backups.

Any suggestion welcome!

 

[Samuel Tai]

Outside commercial backup software, I'm not aware of any backup tool that does this. I can see doing this in tar and dd with bs, skip, and count, but it would be very tricky.

 

[kiriak]

could splitting the data of the datasets or making datasets inside datasets be a possibility?
then you could replicate snapshots of specific datasets to the external HDDs

 

[ovig]

@Samuel Tai - Bacula/Amanda is what I was thinking to look at as a fallback but would rather not have to install 3rd party software
@kiriak - thought about re-arranging datasets so each fits on an external HDD but balked at the effort... is it possible to create a dataset inside a dataset? and can that be done on a non empty dataset (e.g. on my current dataset with folder1/ and folder2/ can I convert folder1 to a dataset and folder2/ to a second dataset - that’s not something I came across before but looks promising)?

 

[kiriak]

I'm very new to FreeNAS,
I think you can create a new dataset but cannot convert a folder to a dataset
and don't know if moving these data will take additional disk space

probably someone can help on this

 

[Samuel Tai]

Yes, you can nest datasets, and add datasets inside a non-empty dataset:

Say your structure is like this:

/mnt/FreeNAS top-level dataset

/folder1​

/folder2​

​

In Storage->Pools, this looks like

FreeNAS top-level dataset

folder1​

folder2​

​

You can create new datasets

FreeNAS top-level dataset

folder1​

new-nested-subfolder 1​

new-folder 1​

folder2​

new-nested-subfolder 2​

new-folder 2​

 

[argumentum]

..I would copy using RAR or the like, that can make parts/chunks. And copy those to the USB drive. If you find yourself without your familiar environment, this way you can have your data. I would try to make the data as portable as possible. ( my 2 cnts. )

 

[K_switch]

@ovig If i am understanding you right then you just want to move a copy of all your data across a few USB drive to store offsite... you are not expecting live data changes to the USB devices. If that is the case then i wouldn't over complicate... just connect the USB devices to a machine connect to your datasets and move the data using the most comfortable method... cp, robocopy, FileZilla, etc.

If you are looking for live data changes then that is an entirely different conversation.

 

[ovig]

@K_switch : you are right - I just want to take a full backup of my dataset to store offline. The difficulty I have with something like rsync/cp is when ”spilling over. With a 3TB dataset and a bunch of 1TB external drives, I can’t just rsync <src> <DST> as when the first external HDD is full it falls over. The alternative is to trawl through folder and find a bunch that take “almost but not quite 1tb to be safel and copy those; and the find another bunch to copy on the second drive... error prone and not particularly efficient. I’m just trying to automate away from these niggles... pretty much like using a multipart tar/dump/whatever that prompts you for a second tape when the first one is full, but for a bunch of external HDDs instead.

 

[Apollo]

Looking at some youtube video about RDX backup, it seems they are 2.5in disk with SATA ports. So you could use a bunch of those disk and mount them as RAIDZ2 and size them to get enough storage space for your datasets. This way you can use replication and be done with it.

You can have multiple pools to replicate too and have them stored in different locations.

You will retain redundancy through RAIDZ and you will get fast backup when using incremental replication.

I think this is the best solution.