Replication

[FireBIade]

Annoyed... I've been trying to get two freenas servers to replicate but I can't seem to get it to work. I've done all the public key swapping and even rebooted both machines but all I get is a nice email containing:

Hello,
The system was unable to replicate snapshot production to 172.16.64.35 ====================== cannot receive: specified fs (/mnt/backup) does not exist
warning: cannot send 'production@auto-20120106.1107-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1137-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1207-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1237-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1307-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1337-4h': Broken pipe
warning: cannot send 'production@auto-20120106.1407-4h': Broken pipe internal error: Broken pipe Abort trap

I notice it says /mnt/backup does not exist which it does, I can see it in the active volumes tab. SSHing into the remote from the local always asks for a password so it's definately not working as it should. Does anyone have a step by step way of troubleshooting this? The documentation touches on things but doesn't go in-depth.

 

[FireBIade]

OK SSHing into the box has been solved but still no replications are happening, it's something to do with the volume on the destination device. That said the command line example in the docs worked....

 

[Markus Busche]

Hi FireBIade,

how did you solve the problem with your ssh-keying? I successfully set up several FreeNAS boxes with working replication, but this time I tried five times to copy the keys from one machine to the other but ssh-ing from LOCAL to REMOTE using the public key of REMOTE doesn´t work - so replication doesnt´t work, too.

I´m using 8.0.3. Is there any issue concerning replication?

Thanks for help,
Markus

 

[FireBIade]

I'm still looking into this, currently working on other things though but as soon as I come back to it and if I solve it I will let you know.

 

[William Grzybowski]

This is worth a reading: http://doc.freenas.org/index.php/Replication_Tasks

 

[FireBIade]

Ok after a fair bit of head scratching I now have a procedure that at least works for us and is replicatable. NOTE this works with root rather than another user account and so may not be as secure as possible! There also may be steps which are not needed.

*** I'm calling the Source machine A and the destination machine B. ***

Install your 2 machines with 8.03 p1, for simplicity A & B: Configure the machines with the basic settings e.g. change the admin password and root from the web gui, hostnames and IP addresses.
A & B: Configure your storage, e.g create your volumes from the web gui

A & B: Goto Services and modify the SSH service config to allow Login with root on both machines. Save the settings and start the SSH service on both machines.

Using Putty (or similar SSH program)
A & B: Connect Putty into the machine
A & B: Login as root

By default the filing system is in read only mode so you won't be able to modify files until you type the following line, do this on both machines
A & B: mount -uw /

Create some keys using ssh-keygen e.g. (/root/.ssh/id_rsa), accepting all the defaults should be fine and make sure you DO NOT enter a password when asked, just press enter. Do this on both machines.
A & B: ssh-keygen

Now we need to copy the keys (2 sets) to the correct places from A to B. The first line creates the authorized_keys file on B but for the second part I used nano to get the other key in the same file.
A: scp /data/ssh/replication.pub root@<B's IP Address>:/data/ssh/authorized_keys
A: cat /root/.ssh/id_rsa.pub
B: nano /data/ssh/authorized_keys
A: copy and past the result of the cat into B on a new line, making sure it's as one long line (so you may need to move bits up together)
B: save the file.

At this point you will have an authorised_keys file on B which contains A's keys and your about half way through, we now need to repeat this but from the other direction.

B: scp /data/ssh/replication.pub root@<A's IP Address>:/data/ssh/authorized_keys
B: cat /root/.ssh/id_rsa.pub
A: nano /data/ssh/authorized_keys
B: copy and past the result of the cat into B on a new line, making sure it's as one long line (so you may need to move bits up together)
A: save the file.

Ok so now we have /data/ssh/authorized_keys on both A & B machines. We now need to make a copy of this file into the /root/.ssh folders
A & B: cp /data/ssh/authorized_keys /root/.ssh/authorized_keys

The keys are now in the correct places but we need to restart the ssh services to male sure the changes are seen by both A & B
A & B: In the web gui go to the services screen and restart the ssh service

Finally we can setup the periodic snapshot task and the replication task on A.
A: When setting up the replication task in the remote hostkey section paste the contents of B's /root/.ssh/id_rsa.pub key into this section

 

[monarchdodra]

Thank you so much for the tutorial. You have no idea how hard it was to find a simple step by step guide to do this.

I can confirm this works flawlessly.

 

[MARKKIERZKOWSKI]

i have problem configuring replication. I did all the steps but now i get error

cannot unmount '/mnt/Storage01': Device busy
warning: cannot send 'Storage01@auto-20120216.1701-2w': Broken pipe

 

[FireBIade]

I'm not sure why it would say your device is busy, but the fact it says broken pipe suggests to me that the replication is not setup correctly.

Check you have /root/.ssh/authorized_keys file on both A & B machines at that it contains different keys e.g. the files are not identical. Restart the SSH service on both A & B.

 

[winanjaya]

Hi, FireBIade..
I followed the instructions, from A: I able to run ssh -i /data/ssh/replication to B without password prompt
but neither from B .. it keeps ask me the password ..
I also never give passpharse when ssh-keygen

could you please advise? .. what I missed?

many thanks in advance

Regards
Winanjaya

 

[FireBIade]

I'm not 100% sure what you mean but I would go through the "Now we need to copy the keys (2 sets) " section again.

 

[winanjaya]

ok, I have rechecked it.. something missed there!..

now,
on A: I run:

[root@nas01] ~# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
vol2tb01@auto-20120322.0056-2w 422K - 3.29G -

then I try to do replication testing as below, but it fails

[root@nas01] ~# zfs send vol2tb01@auto-20120322.0056-2w | ssh -i /data/ssh/replication 172.16.0.33 zfs receive iscsihd@auto-20120322.0056-2w
cannot receive new filesystem stream: destination 'iscsihd' exists
must specify -F to overwrite it
warning: cannot send 'vol2tb01@auto-20120322.0056-2w': Broken pipe
[root@nas01] ~#


I also tried to ssh -i to 172.16.0.33 as below and the result is Ok..

[root@nas01] ~# ssh -i /data/ssh/replication 172.16.0.33
Last login: Fri Mar 23 01:31:49 2012 from 172.16.0.40
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.2-RELEASE-p6 (FREENAS.i386) #0: Tue Feb 28 18:15:56 PST 2012

FreeNAS nanobsd (c) 2009-2010, The FreeNAS Development Team
All rights reserved.
FreeNAS is under the modified BSD license.

For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
[root@freenas] ~#

what I missed again?

could you or anybody please help?

many thanks in advance

Regards
Winanjaya