Replication process was reset to 0 after server reboot

[shnurov]

I'm trying to understand something - I've been replicating 4.4TB via WAN for the past two days to test out the connection, etc. Now I had to perform a reboot on the PULL server and upon completion the replication picked up again. The Volume manager on the PULL server now shows ~1.5GB replicated (and going up) instead of 150GB right before I processed the reboot.

Am I correct to assume that if there's an issue with the replication ssh tunnel link it gets all dropped and goes back to 0? Instead of picking up where it stopped.

 

[SweetAndLow]

Replication can't pick up where it left off. It's an all or nothing deal.

 

[shnurov]

Alright!

Now it seems like I have another issue where it just get stuck on 'waiting' - I had done a lot of work around trying different settings and such, but no success so far :(

 

[dlavigne]

Please post a screenshot of what you mean by waiting. Also, which build (from System -> Information)?

 

[shnurov]

Here it is.
SSH and all is working - at this point it's a bit weird as it did work when I was home this morning - came into the office to force the 4.4TB copy over GBit LAN vs 10Mbit WAN and changed the 'remote' hostname and nothing else.

When I do the SSH test I get to this error screen that just keeps rolling:

Code:

debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 
debug2: tcpwinsz: 66608 for connection: 4

When I try to manually send a snapshot I get:

Code:

zfs send set4tb/Photo@auto-20160113.1200-3w | ssh -i /data/ssh/replication 192.168.0.198 zfs receive remote/Photo@auto-20160113.1200-3w

cannot receive new filesystem stream: destination 'remote/Photo' exists must specify -F to overwrite it
warning: cannot send 'set4tb/Photo@auto-20160113.1200-3w': Broken pipe

Both systems are on

FreeNAS-9.3-STABLE-201602020212

 

[dlavigne]

cannot receive new filesystem stream: destination 'remote/Photo' exists must specify -F to overwrite it

It looks like it is stuck there. If you check the box to "Delete stale snapshots on remote system" in the replication task, that should unstick it.

 

[shnurov]

I had tried both checked and unchecked with no success.
Oddly enough this is not an issue I had when the Pull server was at home and everything was going over WAN.

 

[jamiejunk]

I've had weird problems with the Freenas replication for a while. Lots of times transfers wouldn't happen, but I wouldn't have much info to troubleshoot with. So I started using zxfer instead and everything has been great. Running it for a few months now.

 

[shnurov]

Small update - this morning at 2am EST the recent update was pushed a second time with the same fixes - FreeNAS-9.3-STABLE-201602031011.

Updated both systems and immediately started getting e-mails saying 'replication failure, remote system unavailable' - once the replication system got up it started! Woohoo!

Now my next step is to replicate between an archive QNAP and the Freenas - hopefully I can get them to talk directly to each other without going through a Windows system.