New to FreeNAS and ZFS... Best way to copy/sync?

[Zaaphod]

I'm very new to FreeNAS and the ZFS file system. I am wondering what the best way to copy and sync large amounts of data with a FreeNAS share?

I have FreeNAS set up and running, and I have a small test hard drive running on it. after I understand it better I will put my real hard drives on it and start moving my data to it, and I'm wondering the best way to to the actual transfer.

There are two types of transfers I am interested in, One is the initial transfer of all the contents of all the current storage devices that are located on local computers (this is my first NAS), and also this same process in reverse.. from the NAS to a local device such as a laptop for off-site and off-line access.. so this question is bulk transfer to and from local computers.
I've been using windows robocopy to backup onto external hard drive, I'm wondering if that is the best option for copying to and from the NAS from local computers, or is there a better way. I turn on logging with robocopy, and use it for bulk backup copies as well as syncing with devices such as laptops

The second is how to bulk transfer folders to either external USB drives (plugged into the NAS motherboard) for off-site backup, or to something like or other internal NAS drives within the same box, or even internal drives in an SATA Dock. So these deal with transfers to other drives physically connected to the NAS system.
It seems silly to me to run something like robocopy on a local computer to copy from one network resource to another network recourse on the same NAS system, why send all my data through the network to the local computer and then send it all the way back to the NAS, it seems it would be better to just go from one location to the other on the NAS system directly, and so I'm asking, what is the best way to achieve this?

Thank you in advance for any advice anyone has to offer on this subject

 

[Arwen]

Can't answer the first question, (I don't use MS-Windows). But there are several threads about how people
backup there FreeNAS. Here are some hints;

1. Don't use USB attached anything, (flash, or spinning), for long term attached storage. SATA, eSATA or SAS is best.
2. Some use a dedicated ZFS pool for backups, that can be exported and physically detached.
3. Some have Seagate 8TB SMR disks, (Shingled Magnetic Recording), which work okay for backups, but may not for other uses
   
4. Some use another FreeNAS, off-site for backups.

One advantage of using a separate ZFS pool for backups, (even if it's a single disk), is that you can run scrubs
against it to be certain backup data is good. HOWEVER, if you use non-ECC memory in your FreeNAS server,
corruption could end up in your backup pool during the copy process. That's because most copy methods read data
from one set of disks and write to another set of disks, with memory as the intermediate step.

I use a single Seagate 8TB SMR disk as a ZFS pool for backups. It's used in a hot swap eSATA enclosure and taken
off-site. That covers everything, including media files. I also rotate that with a 750GB disk that skips the media
files, but gets 100% of everything else.

 

[depasseg]

1) FreeNAS won't be able to read the windows disk directly, so you will need to use windows to copy your files to a CIFS share on freenas.

2) in terms of backup, the best way is to use snapshots and zfs replication built into freenas. The other part of your question is the type of media. As @Arwen mentioned, avoid USB and use SATA, eSATA or SAS. There are a couple posts kicking around that talk about something similar to what you are trying to accomplish.

 

[Zaaphod]

Thank you for the help, So much of this is new all the the same time.

1) FreeNAS won't be able to read the windows disk directly, so you will need to use windows to copy your files to a CIFS share on freenas.

ok, I suppose my current method of using robocopy on windows will work for this

Don't use USB attached anything, (flash, or spinning), for long term attached storage. SATA, eSATA or SAS is best.

2) avoid USB and use SATA, eSATA or SAS.

just to clarify... with FreeNAS can I hot plug in an SATA III spare drive, perform the replication, then shut the drive down and remove it? or if I want to do be removable do i need to get a card with eSATA ports on it?

Some use a dedicated ZFS pool for backups, that can be exported and physically detached.

That sounds like a great idea

Some have Seagate 8TB SMR disks, (Shingled Magnetic Recording), which work okay for backups, but may not for other uses

I have one Seagate Archive 8 TB drive ST8000AS0002 I wanted to use for a backup then remove from the system.. is that the same type of drive you mention here?

I also have two HGST Ultrastar HE8 8TB Helium 0F23267 drives I intend to use for my main storage drives. I'm wondering if I should get a third one and use RAID-Z1 or just stick with a mirror copy, seems either way if one drive failed everything would be on the other drive, or is there an additional benefit to RAID-Z1 that I should consider?

Some use another FreeNAS, off-site for backups.

That's an interesting idea, does FreeNAS have a funtion which would backup to another FreeNAS box from directly within it?

One advantage of using a separate ZFS pool for backups, (even if it's a single disk), is that you can run scrubs against it to be certain backup data is good. HOWEVER, if you use non-ECC memory in your FreeNAS server, corruption could end up in your backup pool during the copy process. That's because most copy methods read data from one set of disks and write to another set of disks, with memory as the intermediate step.

I like the idea of validating the backup. I have all ECC memory now, I'll make a point to stick with it.

I use a single Seagate 8TB SMR disk as a ZFS pool for backups. It's used in a hot swap eSATA enclosure and taken
off-site. That covers everything, including media files. I also rotate that with a 750GB disk that skips the media files, but gets 100% of everything else.

That sounds like the kind of thing I want to do, but perhaps with a third group.. which would be my photographs. so I would want one huge backup of 100% of everything, including media and photos, one with everything except media, and one with just photos and nothing else. How does this work with ZFS, do I install one of these drives and assign it to the pool I want it to end up backing up and when it sees that drive again it mirrors the pool onto that drive? Also will it end up copying all the files every time or does it just change the files that have changed since last backup, add new ones, and delete deleted ones? Sorry if these are extremely newbie questions, but after all.. I AM a newbie :)

 

[Arwen]

...
just to clarify... with FreeNAS can I hot plug in an SATA III spare drive, perform the replication, then shut the drive down and remove it? or if I want to do be removable do i need to get a card with eSATA ports on it?

Yes, you can hot-plug a drive into FreeNAS for backups. You do need to have a clear procedure, of how to
import, perform the backup, then export the drive(s). Testing never hurts either.
No, you don't need an eSATA card. I used an internal SATA port, with an eSATA PCI bracket. But a free
hot-swap disk slot can be used instead.

...
I have one Seagate Archive 8 TB drive ST8000AS0002 I wanted to use for a backup then remove from the system.. is that the same type of drive you mention here?

Yes. Just be aware that these archive drives can have slow writes. When I do my backups, (a TeraByte at a
time), it runs at 30MBps average. But, small writes probably can run 90MBps or higher. That's because these
archive drives have a small, faster access disk cache, which may be 20GBs for the 8TB model.

...
I also have two HGST Ultrastar HE8 8TB Helium 0F23267 drives I intend to use for my main storage drives. I'm wondering if I should get a third one and use RAID-Z1 or just stick with a mirror copy, seems either way if one drive failed everything would be on the other drive, or is there an additional benefit to RAID-Z1 that I should consider?

With drives larger than 1TB, you want to avoid RAID-Z1. Stick with mirrors or RAID-Z2 or RAID-Z3. The
issue with RAID-Z1, (and RAID-5), is that all the other disks in the RAID group get read during a re-build.
Larger disks have a higher statistical chance of an un-recoverable read that can occur during a re-build.
Thus, RAID-Z2, (or RAID-6), with 2 disks worth of parity / recovery.

...
That's an interesting idea, does FreeNAS have a funtion which would backup to another FreeNAS box from directly within it?

Yes, it's replication. I can't walk you through it, as I don't use it. But, it's there and works if you have enough
network bandwidth between the 2 FreeNAS servers.

...
I like the idea of validating the backup. I have all ECC memory now, I'll make a point to stick with it.

Great.

...
That sounds like the kind of thing I want to do, but perhaps with a third group.. which would be my photographs. so I would want one huge backup of 100% of everything, including media and photos, one with everything except media, and one with just photos and nothing else. How does this work with ZFS, do I install one of these drives and assign it to the pool I want it to end up backing up and when it sees that drive again it mirrors the pool onto that drive? Also will it end up copying all the files every time or does it just change the files that have changed since last backup, add new ones, and delete deleted ones? Sorry if these are extremely newbie questions, but after all.. I AM a newbie :)

There are multiple ways to deal with the issues you describe.

Some people have large enough media data that they have a separate pool with limited recovery. For
example, they may use the not-recommended RAID-Z1.

Or even just striped without mirror or RAID-Zx. Since ZFS uses checksums, and checksums are validated
on every read, plus during scrubs, ZFS can report which files now have bad blocks. Thus, a user can re-rip
that specific movie or TV show, (or restore from backups). So they have a plan to deal with the data's loss.
Of course, if they have a full disk failure, the entire pool is lost... Full restore or full re-rip :-(.

Now for your case. You can use a single mirrored pool of your 2 x HGST 8TB disks. Then inside create at
least 3 separate datasets, (what could be considered file systems). One for your media, one for your photos,
and one for everything else. Thus, you get disk usage for each dataset and can treat them separately for
backups. Or CIFS shares.

Now for backups, if you have space on the target, you can simply ADD a new dataset for the new backup.
That's what I am doing. My 8TB disk currently has at least 4 full backup images on it. When it's full, I delete
the oldest to make room for the newest.

In reality, you can have more than 3 ZFS datasets. As many as make sense for your applications.

 

[depasseg]

I believe your system must be hot swap capable. Not all are.

Snapshots combined with replication is a great solution.

 

[Zaaphod]

Thank you very much for taking the time to answer all my questions. I've been doing a lot of reading and I think I'll stick with just a straight mirror because from what I've been reading the other raid schemes are nice for smaller drives but at the size of these drives it could take days or ever weeks to re-build an array when one drive fails, and during that time the remaining drives are stressed out... which could cause another to fail, but re-duplicating the surviving drive to another new drive is relatively quick and easy. I'm in the process of testing everything with three old 250GB drives I have with just a little data.. just to get a feel for how it all works.

I do have one more question about the hot swap thing.. With FreeNAS how does one do the equivalent of 'Safely Remove Hardward and Eject Media' as one does with windows? The purpose of this on windows it to finish all the writes to the drive and make sure nothing is still in the cache before powering down the drive for removal. I think FreeNAS would need this option so you could remove the drive after making the replication, but I think I'm just missing it.

 

[anodos]

Thank you very much for taking the time to answer all my questions. I've been doing a lot of reading and I think I'll stick with just a straight mirror because from what I've been reading the other raid schemes are nice for smaller drives but at the size of these drives it could take days or ever weeks to re-build an array when one drive fails, and during that time the remaining drives are stressed out... which could cause another to fail, but re-duplicating the surviving drive to another new drive is relatively quick and easy. I'm in the process of testing everything with three old 250GB drives I have with just a little data.. just to get a feel for how it all works.

I do have one more question about the hot swap thing.. With FreeNAS how does one do the equivalent of 'Safely Remove Hardward and Eject Media' as one does with windows? The purpose of this on windows it to finish all the writes to the drive and make sure nothing is still in the cache before powering down the drive for removal. I think FreeNAS would need this option so you could remove the drive after making the replication, but I think I'm just missing it.

I think you'd probably be better off replicating to a second NAS than swapping out drives. Eventually the connectors on either the drive or the backplane will wear out. Hotswap doesn't mean "constantly swap".

 

[Zaaphod]

Don't they still have hard drive trays where you plug the drive into the tray and there is a docking bay you push the tray into so all you are wearing out is the connectors between the tray and the dock.. then when you wear that out, just just get a new tray/dock and you are at least not wearing out the plug on the hard drive. I had about 20 of these for IDE drives back in the day. The idea with swapping in and out a few drives is that while they are out, they are protected against power related distasters. as long as hard drives or anything else is plugged into anything at the same location, a lightning strike can take out everything... all in one shot, so could a fire, or a flood, etc.. If you have just a few GB of data, you could transfer it on-line to get it offsite, but online transfer is not fast enough for massive data, so the best solution is to have at least two drives so you can hot swap one in, duplicate the data, then take it home, then the next day bring the other drive in, so at all times you have at least one drive or two drives in a different physical location.

 

[Scharbag]

Snapshots and replication works great either on the same system or remote systems. Nice thing is the "shadow copies" that are available for use in your windows systems.

Another option for backups is rsync. It works great if you do not want to deal with snapshots and replication. I use a combination of both in my system.

:)

 

[Arwen]

Yes, you can hot swap, if the disk controller supports it. I would think a SATA disk should survive
at least 100 swaps. Back planes might or might not support more, depending on the quality.

At 1 swap per backup, and one backup per month, that would be 100 months of backups. With 12
months per year, that's 8.3 years. If we do weekly swaps of the same disk, and only get 100 swaps,
we get a little less than 2 years.

 

[anodos]

Yes, you can hot swap, if the disk controller supports it. I would think a SATA disk should survive
at least 100 swaps. Back planes might or might not support more, depending on the quality.

At 1 swap per backup, and one backup per month, that would be 100 months of backups. With 12
months per year, that's 8.3 years. If we do weekly swaps of the same disk, and only get 100 swaps,
we get a little less than 2 years.

You can see relevant SATA spec here: http://www.calgreg.com/circuit-assembly/sata-specification.pdf
Minimum durability requirement is 50 cycles for internal cabled connections and 500 cycles for backplane connections. I think I read somewhere that the esata durability requirement is 5000 cycles.

 

[Zaaphod]

looking at the way the connector is designed, the fingers on the drive end of the connector cannot move, they are just little gold plated tabs. the cables have the little springy contact in them.. so if anything is going to wear out, it's going to be that spring tensioned contact.. the spring is the part with movement, and it will break first.. so just replacing cables or trays with connectors in the back of them every few years should be more than adequate.