How I did site to site replication

[Zachary Miller]

Hi All!

Over the past few months I've been going through the forums and gathering a lot of information. I am a managed service provider in PA and love using open source products. One venture I just recently delved into is doing off site backups for some of my clients. This has not only opened additional revenue for me, but allowed me to play with FreeNAS, such a great product! I wanted to give you all the experience I had so you can easily use it however you see fit, giving back how I can to the community. Below are the steps I used to get my system setup to accept seeds from remote hosts and finally get the site to site replication working.

1. Obviously the hardest thing for me was getting the initial backup of data to my FreeNAS at my shop. These backups ranged from 100 GB up to 300 GB for a full system backup for me. I bought a 1 TB external USB drive.
2. The external drive was used as my seed device, all I had to do was plug it and create a new volume as FreeNAS saw it as an additional disk.
3. Once this volume is created, you can create a snapshot of the volume you created to store the user's backups. I found iSCSI a necessity for using Windows Server 2008 R2 native backup solution as you can't do incremental across a network drive. With the iSCSI attached using iSCSI initiator it looks like a local drive to Windows so you get full functionality with the built in backup, which is quite nice.
4. To replicate locally is fun, took me some trial and error, here is how I do it, please correct me if someone knows a better way.
   1. On the FreeNAS box, I got to services and ssh, check allow root login.
   2. Open up the storage tab, go to replication and view public key. Copy that key.
   3. Now go into the users tab, open up root, and paste the key into the ssh key area under root.
   4. Now ssh into your FreeNAS box as root, I use putty.
   5. type mount -uw / (This puts your drive into a writeable mode)
   6. Type in ssh -i /data/ssh/replication 127.0.0.1 (This adds the key to known_hosts and saves it because it is in writable mode)
   7. type exit (This exits the ssh session you just made to yourself, talk about inception :P )
   8. Type in mount -ur / (This can take some time as it puts the system back into read only mode)
   9. Now you can replicate locally using the following:
      1. The initial code looks like this: zfs send volume/dataset@snapshotname | ssh -i /data/ssh/replication 127.0.0.1 zfs receive seed-volume/dataset@snapshotname
      2. The first one will take time depending on the size. I noticed you can go to storage tab and see the new dataset created on your seed device when this starts so you know its working
      3. After the first one is done you can do an incremental which looks like the following:
      4. zfs send -i volume/dataset@snapshotname-old volume/dataset@snapshotname-new | ssh -i /data/ssh/replication 127.0.0.1 zfs receive seed-volume/dataset@snapshotname-new
      5. As you can see you have to specify the older snapshot first and then the newer one so it knows what to build from
5. Ok so now you have replicated all your snapshots to the seed device. You can detach this volume from the storage tab, make sure you don't select the new disk, you will lose everything you just did!
6. Take the device to your office FreeNAS and plug it in.
7. Go to storage tab, click the auto-import and let it find your volume. Import your seed volume.
8. Now you will repeat the steps above on your local system, just inverse, taking it from the seed device to your local FreeNAS volume.
9. Once you have your FreeNAS caught up, now you get to setup the replication across remote nodes.
10. This is where it starts to get a little more fun. I used the PFSense router, an open source solution, and created an OpenVPN site to site vpn. This has worked for me to get the connection established, but any vpn or peer-to-peer link will work, as long as you can talk to your end host, I guess you could even do it straight across the internet without a vpn using public IP's, but I like the security blanket of having it inside a vpn.
11. Once you decide how you would like to do it, you will need to break out putty again and follow these steps:
    1. Putty into your local FreeNAS box. Again we are going to put it into writable mode to take care of a few ssh things
    2. So type in mount -uw /
    3. We will do a cd ~/.ssh
    4. This will put us into root's ssh directory
    5. There will be two files in there, you can see them by entering ls, authorized_keys and known_hosts
    6. type in vi authorized_keys (VI is a text editor in linux, a little cryptic but I'll walk you through it)
    7. Once in there, type A (A is a shortcut to append to the end of a file)
    8. Now press enter to create a new line
    9. Here you will past in the public key from your remote host, you can obtain this from the replication tab on the remote node
    10. Once pasted in press esc (This will take you out of edit mode on text file)
    11. Now type in :wq (: will allow you to put in command, w is write, q is quit)
    12. You should be back to a standard command prompt. You just added the public key from the remote host to the authorized_keys allowing for key authentication via ssh
    13. Now type in ssh -i /data/ssh/replication remotehostip
    14. When prompted to add it to known_hosts type yes
    15. Now type exit as you are currently ssh'd to the remote node
    16. Now type in mount -ur /
    17. Repeat the above steps on the remote node
12. With this all done you can ssh back and forth without a password yay! Now if you went to try and setup replication via the GUI, it will most likely still fail, and this is because it is trying to send the snapshots you have already seeded because it does not know what you have and what you don't. We need to fix this.
13. To see what your "latest" snapshot for replication is type the following: zfs list -Ht snapshot -o name,freenas:state
14. This will give you a list of all your snapshots and the column called freenas:state, this column actually tells you what is latest and what is new. If you see all of them with - then it doesn't know what your remote system has. To correct this we need to issue a command.
15. In putty on your remote node, type in the following, replace snapshotname with the latest snapshot you had on your seed device: zfs set freenas:state=LATEST snapshotname
16. Now you can issue the list command again to see if the setting took.
17. Once this is in place, you can setup your scheduled replication task in your GUI and all should work. If you already had your replication task setup in the GUI, but it failed because of this freenas:state option, you can manually start the replication task with the following command: python /usr/local/www/freenasUI/tools/autorepl.py
18. This will kick off our replication again based on your GUI settings.

This was a lot of time and trial/error on my side to create this document. I really hope it helps out this great community of FreeNAS and that things continue to progress. Feel free to hit me up with any questions you have, can't guarantee I'll be as speedy as CyberJock, but I'll try my best. Good luck to you all!

Thanks,
Zach