SOLVED Replication Help

[Derek Humes]

Hi, newbie here to FreeNAS and FreeBSD. I have inherited a 9.3 FreeNAS, and am attempting to create new 9.10 FreeNAS. I have created the 9.10 machine, and setup my datasets, approximately 40 or so, setup for different departments, as well as user home directories. I have setup a daily snapshot of the top-level parent dataset, and checked the option to recursively include all datasets underneath. This is working fine and creating daily snapshots. I then have enabled replication to my 9.3 FreeNAS which is at a different physical location than where I am at. I have confirmed that SSH is setup correctly to allow me to connect. I have created and enabled a single replication task that replicates the top-level dataset, and all children datasets, to my 9.3 machine on a dataset named 'replicationstore',

This replication starts, but isn't running correctly it seems. On the 9.3 machine, under the Storage/Volumes tab, I can see that underneath 'replicationstore' 3 new datasets have been created, my top-level dataset from my 9.10 box, and a child dataset for "user home directories", and then another child dataset for a specific user. However, this is all that is created, no other datasets. Also, under the Snapshots tab of the 9.3 device, I do not see any snapshots listed, besides a manual snapshot that was taken of the entire local 9.3 datasets taken about 8 months ago.

Can someone assist with this? Is it possible the difference in software version is affecting the replication of the snapshots?

Thank you so much!

 

[nojohnny101]

What version of 9.10 are you running on your main box and what version of 9.3 on your replication target? I read about fairly widespread replication issues when 9.10.2 (a476f16) was released. I ended up rolling back to FreeNAS-9.10.1-U4 (ec9a7d3) on my main box and the issues have gone away. You might be able to search and find the threads.

Others have been able to destroy their replication tasks and older snapshots and it started working again but that wasn't an option for me in my current setup.

 

[Derek Humes]

FreeNAS-9.10.2-U1 (86c7ef5)

FreeNAS-9.3-STABLE-201509022158

I can try again with re-creating the replication tasks. I hadn't tried deleting the old manual snapshot in the remote system either.

 

[nojohnny101]

If you don't need that old snapshot then go ahead and wipe that out. Create the snapshot task from scratch and then see if it runs. How much data are you moving?

Do you plan on eventually updating the 9.3 box?

 

[Derek Humes]

I will have approximately 1TB of data on the 9.10 box, before any compression. I do intend to update the 9.3 box to matching version eventually, but not at this moment, unless I find a strong need to.

The 9.10 box will be at my main site, and the 9.3 at our remote office. I plan on having all users connect to the 9.10 box, as long as our bandwidth permits it effectively. Then want to send my snapshots to the 9.3 remote location for an offsite backup. I have a 20Mb fiber WAN connection between the sites. I am hoping this will suffice, and that the snapshots will not require too much data replication daily that it bogs me down. I have read some about this, but figured running it in test would be my best...test.

 

[nojohnny101]

Sounds like a good plan. Snapshots are extremely efficient (only transmitting changes) so I think you'll be fine there.

 

[Derek Humes]

Well, I recreated everything but no change. However, I did just happen to add SMTP email alert settings, and upon recreation of the job got the following error email:

"Hello,
The replication failed for the local ZFS MyVolume/User Home Folders/johndoe while attempting to
send snapshot auto-20170112.1411-1m to {IP Address of my remote FreeNAS}"

 

[nojohnny101]

Do you have any messages in your logs to indicate what type of problem it is? I know you said you setup SSH properly but are you sure you have entered the correct keys (as per the logs). Also since you are doing this to a remote server (correct?) then you sure your DNS forwarding service is setup correctly? What are you using for that?

 

[Derek Humes]

On the remote server it is creating a new dataset, so I figured the SSH connection would be good if it is getting that far?

I am connecting to the remote server via LAN IP address. Our sites are connected together with an IPSEC VPN tunnel.

I am very unfamiliar with FreeBSD still - I will research what log files to browse and see what they can tell me.

 

[nojohnny101]

Sorry I missed that you were doing it locally and didn't know they had a VPN tunnel between them.

Logs can be found here: /var/log/messages

 

[Glorious1]

FYI, 9.10.2-U1 fixed the auto snapshot/replication bug affecting some users in 9.10.2. But that was a complete all or none deal, doesn't sound like your situation.

 

[Derek Humes]

I have reviewed /var/logs/messages but am not seeing anything standing out as as an error with replication. I see a number of these errors, but they appear to be upon a reboot.


Jan 16 13:45:10 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1

Jan 16 13:45:10 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1

Jan 16 13:45:10 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1

Jan 16 13:45:11 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2

Jan 16 13:45:11 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2

Jan 16 13:45:11 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2

Jan 16 13:45:12 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

Jan 16 13:45:12 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

Jan 16 13:45:12 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

Jan 16 13:45:13 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:6 Err:-1 Errno:61 Connection refused

Jan 16 13:45:13 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:4 Err:-1 Errno:61 Connection refused

Jan 16 13:45:13 KCANPFNAS001 python: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:7 Err:-1 Errno:61 Connection refused

 

[Derek Humes]

I only recently updated to 9.10.2 - U1 - it is possible my snapshots were created on 9.10.2. Not sure if the bug could affect me in that case, but I have no production data yet on my 9.10 box, so I should be able to easily blow out my snapshots and let new create tonight for testing tomorrow.

 

[Derek Humes]

No luck this morning with fresh snapshots. Same error email received. I think I am going to try upgrading my 9.3 box to matching latest release tonight, and see if that resolves this.

 

[nojohnny101]

Hmm, that's strange. You shouldn't have to upgrade for it to work but that would be worth trying. Let us know.

 

[Derek Humes]

Would /var/logs/messages be the only applicable log file, or is there another that I could look at?

 

[wblock]

Can you ssh to that machine? OpenSSH recently deprecated DSA keys.

 

[Derek Humes]

I can successfully SSH from the 9.3 box to the 9.10 box without password using SSH pub key authentication.

 

[Derek Humes]

Actually I meant the opposite, I can SSH from the 9.10 box to the 9.3 box.

 

[wblock]

Can you try the replication manually, as shown in the Troubleshooting section of http://doc.freenas.org/9.10/storage.html#troubleshooting-replication?