Replication usb drive

[Zachary Miller]

Hello All!

I'm new to FreeNAS and am excited to get started with what seems to be a great solution. I've had a lot of experience in the enterprise realm and have worked with NetApp in the past, so I have some storage sense. I've got CIFS working, NFS, iSCSI, all that is pretty standard.

My big question comes down to replication. I want to use ZFS replication for some clients. I'm going to put a FreeNAS device at their place and one at my office and do off site for them. The replication is pretty straight forward, however I don't want to seed all their data over the WAN. I'm going to setup site-to-site VPN's with them and that is also pretty straight forward. I need a way to get their data to my office for the initial seed without using the VPN.

I didn't know if there was a way to do it with a USB drive or if anyone has experience with this. I know we should create a snapshot right before the seed, that way its a common starting point. If someone could shed some light on this, it would be greatly appreciated!

Thanks,
Zach

 

[cyberjock]

I'm not experienced enough to tell you if a USB drive will work. You could send ZFS snapshots to files and put those on a USB stick, but you'll lose the ability to verify the data integrity from the snapshot file.

Everyone else I've seen has just put the 2 servers on the same LAN for the initial copy, then transported one to the offsite location.

 

[Zachary Miller]

Thank you for the quick response. I have seen the local replication suggestions. The problem is I'm hoping to use my bigger FreeNAS device in my office to host several clients. It would be a pain to have to un-rack it, take it to a place and pause the other replications until the volumes are caught up. I was hoping to do something similar to an enterprise where you can do a tape drive, write the volume to tape, and restore it. I don't really want to fiddle with tape drives, so I was hoping an easier way to use USB drives or something more portable.

 

[cyberjock]

Personally, the only way I can think of is snapshot to file on your USB drive. Keep in mind that if you have a 10TB server your 2TB USB drive isn't going to hold your initial snapshot.

Other than that, I think you are at a loss for other options. ZFS is an enterprise class service. They don't deal with limitations like you and I have with small internet pipes and such. So there isn't much in terms of options for how to deal with the situation.

 

[Zachary Miller]

Hey Cyber,

Most companies I deal with are Sub 500 GB drives so it isn't a huge deal. When you say snapshot to a file, is that GUI or a command line thing? As I said just started using FreeNAS a couple days ago, so pardon my ultimate noobness. You said there are repurcussions to doing it this way though? If you could elaborate it would be greatly appreciated.

Thanks,
Zach

 

[Dusan]

Personally, the only way I can think of is snapshot to file on your USB drive.

You can also create a zpool on that USB drive and do a proper ZFS replication.
Btw.: zfs receive will detect corrupted replication streams/files.

 

[Zachary Miller]

Ok so if I do a ZFS replication from the appliance to an external USB device, then take that USB device back to my office, attach it my appliance, how will my appliance detect it? Do I just mount it and it will see the data? If so I just setup a replication from that to my zpool. Then I can do a replication from my zpool and pull from the client's side? This would advert any problems with the corruption from doing a snapshot file? Once I figure all this out and start playing with it I intend to make a document on how to do this to help everyone out! Thanks for all the input thus far!

 

[Dusan]

Both options are able to detect corruptions (the zfs checksums are included in the replication stream/file).
Rough outline of what you could do:

1. Create a ZFS volume on the USB drive (you can do this at your office).
2. Detach the volume.
3. Take it to the remote location and Auto Import it.
4. Do a snapshot & replication of the data.
5. Detach, take the drive to your office, connect & Auto Import.
6. Replicate the snapshot to your bigger device.

You can do the snapshot from the GUI, but you would need to do the replication from the command line.

 

[cyberjock]

But it won't be able to repair it if the data is corrupted, right?

 

[Zachary Miller]

Is it pretty common for the data to get corrupted? Also, does anyone have the command line syntax for the replication to get me started? The help has been awesome and I really appreciated it!

 

[cyberjock]

Common is not a good word for it. Depends on many factors. But ZFS is supposed to be immune to corruption normally, so its important to know when you don't have that protection.

zfs manpage is your best command line guide. http://www.freebsd.org/cgi/man.cgi?...h=FreeBSD+9.1-stable&arch=default&format=html

 

[Dusan]

But it won't be able to repair it if the data is corrupted, right?

Correct. It's similar to a single drive zpool -- it is able to tell you that there is a problem, but as there is no redundancy it is not able to fix it. I tested it once by randomly flipping bits in a snapshot file and trying to receive it. It fails with "cannot receive new filesystem stream: invalid stream (checksum mismatch)" or "cannot receive new filesystem stream: invalid backup stream" depending on where the corruption is.

 

[cyberjock]

Hmm.... so it won't integrate(which is good) but you'd obviously have to go get another uncorrupted copy to update your "remote" server. Is there a way to do a data integrity check of one of these snapshot files without actually applying it?

 

[Dusan]

Actually it is a bit more complicated. If the file contains multiple streams (multiple snapshots or multiple filesystems if you used the recursive option) it will definitely integrate all valid snapshots up to the corrupted one. I'm not sure if it also processes any valid snapshots that may be present in the stream after the corruption.

zfs receive -n won't help you verify the file as it will only tell you the names of the stream it would receive, but it does not completely process the data. Luckily, there is the zstreamdump command :). If the stream metadata is corrupt if fails with something like this: "INVALID record found: type 0x400", if a data checksum doesn't match it outputs for example:
[PANEL]Expected checksum differs from checksum in stream.
Expected checksum = 172947509f5/21a26cc078cd80/e104a375a5a1497a/ab208c6e3fa5556c
END checksum = 172947508f5/21a26cc0480e80/e104a3710171a97a/ab204100adad756c[/PANEL]Be aware that it continues processing the file, so if the file contains multiple streams the error could be buried somewhere in the middle of the output.

 

[Zachary Miller]

Ok I'm revisiting this because I"m almost done with the replication task I'm looking to achieve thanks to all your help! So far so good as far as the instructions. Now I'm at the point where the replicated data is on my office's FreeNAS and I want to setup the schedule replication for every night. Is it possible to do this through the GUI so the replication task shows or is this something I will need to do through crons? Not really sure how the snapshots will sync up and know of each other on the two appliances. I haven't tried it yet, getting ready to, but thought I would reach out beforehand and see if there are any gotchas I should worry about.

 

[Dusan]

Is it possible to do this through the GUI so the replication task shows or is this something I will need to do through crons?

You can do it in the GUI: http://doc.freenas.org/index.php/Replication_Tasks

 

[Zachary Miller]

Hey Guys,

You have all been a great help and I'm almost there! I got the initial seed into my local freenas device and did a manual increment snapshot replication via command line and this worked great. When I went into the GUI to schedule the replication task, I have it all setup, but now it tries to send the full snapshot instead of the incremental. Shouldn't the replication task pickup that a snapshot already exists and try to do an incremental or is there something I'm missing? The local seed is there, a manual replication worked, but he scheduled replication via the GUI acts like it doesn't know its already there. Any help would be greatly appreciated.

Thanks,
Zach

 

[Dusan]

FreeNAS uses a custom ZFS snapshot property (freenas:state) to keep track of snapshots it already replicated. Try this to fix your situation:
zfs set freenas:state=LATEST <name of the last snapshot that exists on the remote side>

 

[Zachary Miller]

I have tried this but the replication still appears to be trying to send the original snapshot instead of the most current one. Is there anything I can look at as to why this is happening? Any help would be greatly appreciated.

Thanks,
Zach

 

[Zachary Miller]

Correction, I had to list the freenas:state to realize for some reason the state was NEW on the initial snapshot. I used the command:

zfs list -Ht snapshot -o name,freenas:state

Once I saw the initial snapshot was set to NEW, I used the set command you showed me to set it to "-" instead. Then I went back through and made the latest one was set to LATEST. I kicked it off manually with the following command:

python /usr/local/www/freenasUI/tools/autorepl.py

This seemed to kick off the replication task. I have a little to catch up on so hopefully in a few hours I'll post an update on how everything is working fantastic and how cool freenas is! Thanks again for all your help on this.

Zach