Hi Guys,
I have setup FreeNAS 8.3.0 RELEASE p1 x64 and have bumped up ZFS to the latest version.
I have replication setup and have been doing so for sometime. Last night, replication seemed to take longer and it ate into business hours the following morning (the replication schedule is set to allow replication 24 hours a day at the moment). To stop it, I amended our firewall to drop packets to the destination FreeNAS at the gateway.
This worked, but then when I lifted the rules, the replication came back with the following errors..
I figured since the backup it was trying to replicate which I stopped was "20130403", I would delete all these snapshots on the remote end. BUT, I accidentally deleted "20130402". So I continued on and deleted "20130403" also..
Now when I replicate I get this error..
I tried also unticking the "recursively replicate" option but this has not helped.
What are your thoughts? Shouldn't the FreeNAS be able to pick itself up when the datastream is interrupted during a replication?
Thanks, Richard
I have setup FreeNAS 8.3.0 RELEASE p1 x64 and have bumped up ZFS to the latest version.
I have replication setup and have been doing so for sometime. Last night, replication seemed to take longer and it ate into business hours the following morning (the replication schedule is set to allow replication 24 hours a day at the moment). To stop it, I amended our firewall to drop packets to the destination FreeNAS at the gateway.
This worked, but then when I lifted the rules, the replication came back with the following errors..
Apr 4 19:01:10 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Hr -o name -S creation -t snapshot -d 1 backup | head -n 1 | cut -d@ -f2"
Apr 4 19:01:11 FreeNAS-Push autorepl.py: [tools.autorepl:307] Remote and local mismatch after replication: data@auto-20130403.1938-6m vs data@auto-20130402.1938-6m
Apr 4 19:01:11 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Ho name -t snapshot backup | head -n 1 | cut -d@ -f2"
Apr 4 19:01:12 FreeNAS-Push autorepl.py: [tools.autorepl:323] Replication of data@auto-20130402.1938-6m failed with cannot receive new filesystem stream: destination has snapshots (eg. backup@auto-20130330.1938-6m) must destroy them to overwrite it warning: cannot send 'data@auto-20130213.1746-6m': Broken pipe warning: cannot send 'data@auto-20130214.1746-6m': Broken pipe warning: cannot send 'data@auto-20130215.1746-6m': Broken pipe warning: cannot send 'data@auto-20130216.1746-6m': Broken pipe warning: cannot send 'data@auto-20130217.1746-6m': Broken pipe warning: cannot send 'data@auto-20130218.1746-6m': Broken pipe warning: cannot send 'data@auto-20130219.1746-6m': Broken pipe warning: cannot send 'data@auto-20130220.1746-6m': Broken pipe warning: cannot send 'data@auto-20130221.1746-6m': Broken pipe warning: cannot send 'data@auto-20130222.1746-6m': Broken pipe warning: cannot send 'data@auto-20130223.1746-6m': Broken pipe warning: cannot send 'data@auto-20130224.1746-6m': Broken pipe warning: cannot send 'data@
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [tools.autorepl:264] Creating backup on remote system
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org /sbin/zfs create -o readonly=on -p backup
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R data@auto-20130402.1938-6m | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "/sbin/zfs receive -F -d backup && echo Succeeded.") > /tmp/repl-13932 2>&1
Apr 4 19:01:11 FreeNAS-Push autorepl.py: [tools.autorepl:307] Remote and local mismatch after replication: data@auto-20130403.1938-6m vs data@auto-20130402.1938-6m
Apr 4 19:01:11 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Ho name -t snapshot backup | head -n 1 | cut -d@ -f2"
Apr 4 19:01:12 FreeNAS-Push autorepl.py: [tools.autorepl:323] Replication of data@auto-20130402.1938-6m failed with cannot receive new filesystem stream: destination has snapshots (eg. backup@auto-20130330.1938-6m) must destroy them to overwrite it warning: cannot send 'data@auto-20130213.1746-6m': Broken pipe warning: cannot send 'data@auto-20130214.1746-6m': Broken pipe warning: cannot send 'data@auto-20130215.1746-6m': Broken pipe warning: cannot send 'data@auto-20130216.1746-6m': Broken pipe warning: cannot send 'data@auto-20130217.1746-6m': Broken pipe warning: cannot send 'data@auto-20130218.1746-6m': Broken pipe warning: cannot send 'data@auto-20130219.1746-6m': Broken pipe warning: cannot send 'data@auto-20130220.1746-6m': Broken pipe warning: cannot send 'data@auto-20130221.1746-6m': Broken pipe warning: cannot send 'data@auto-20130222.1746-6m': Broken pipe warning: cannot send 'data@auto-20130223.1746-6m': Broken pipe warning: cannot send 'data@auto-20130224.1746-6m': Broken pipe warning: cannot send 'data@
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [tools.autorepl:264] Creating backup on remote system
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org /sbin/zfs create -o readonly=on -p backup
Apr 4 19:01:21 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send -R data@auto-20130402.1938-6m | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "/sbin/zfs receive -F -d backup && echo Succeeded.") > /tmp/repl-13932 2>&1
I figured since the backup it was trying to replicate which I stopped was "20130403", I would delete all these snapshots on the remote end. BUT, I accidentally deleted "20130402". So I continued on and deleted "20130403" also..
Now when I replicate I get this error..
Apr 4 19:22:10 FreeNAS-Push autorepl.py: [tools.autorepl:264] Creating backup on remote system
Apr 4 19:22:10 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org /sbin/zfs create -o readonly=on -p backup
Apr 4 19:22:10 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send data@auto-20130402.1938-6m | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "/sbin/zfs receive -F -d backup && echo Succeeded.") > /tmp/repl-15639 2>&1
Apr 4 19:22:11 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Hr -o name -S creation -t snapshot -d 1 backup | head -n 1 | cut -d@ -f2"
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [tools.autorepl:307] Remote and local mismatch after replication: data@auto-20130401.1938-6m vs data@auto-20130402.1938-6m
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Ho name -t snapshot backup | head -n 1 | cut -d@ -f2"
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [tools.autorepl:323] Replication of data@auto-20130402.1938-6m failed with cannot receive new filesystem stream: destination has snapshots (eg. backup@auto-20130330.1938-6m) must destroy them to overwrite it warning: cannot send 'data@auto-20130402.1938-6m': Broken pipe
Apr 4 19:22:10 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org /sbin/zfs create -o readonly=on -p backup
Apr 4 19:22:10 FreeNAS-Push autorepl.py: [common.pipesubr:49] Executing: (/sbin/zfs send data@auto-20130402.1938-6m | /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "/sbin/zfs receive -F -d backup && echo Succeeded.") > /tmp/repl-15639 2>&1
Apr 4 19:22:11 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Hr -o name -S creation -t snapshot -d 1 backup | head -n 1 | cut -d@ -f2"
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [tools.autorepl:307] Remote and local mismatch after replication: data@auto-20130401.1938-6m vs data@auto-20130402.1938-6m
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [common.pipesubr:42] Popen()ing: /usr/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -q -p 22 mybox.no-ip.org "zfs list -Ho name -t snapshot backup | head -n 1 | cut -d@ -f2"
Apr 4 19:22:12 FreeNAS-Push autorepl.py: [tools.autorepl:323] Replication of data@auto-20130402.1938-6m failed with cannot receive new filesystem stream: destination has snapshots (eg. backup@auto-20130330.1938-6m) must destroy them to overwrite it warning: cannot send 'data@auto-20130402.1938-6m': Broken pipe
I tried also unticking the "recursively replicate" option but this has not helped.
What are your thoughts? Shouldn't the FreeNAS be able to pick itself up when the datastream is interrupted during a replication?
Thanks, Richard