Right, so I thought I'd set-up my replication from scratch, as I was still having issues taking a recursive snapshot at the volume level and replicating this to the 2nd machine, as it still appears to interfere with the system settings (I'm replicating at the volume level on the receiving machine)
I destroyed all the snapshots on the receiving machine, detached the volume and recreated the volume (APEpool2) so I was stating from a blank canvas.
I deleted the periodic snapshot and replication task on the sending machine, and destroyed all the previous snapshops. I look a manual snapshot at the volume level, just so I had something to roll back to.
I then created periodic snapshots for each of the main datasets on the volume (APEpool1) and created a replication tasks for one of the datasets (media). My main machine then reported an error [CRITICAL: Replication APEpool1/media -> 192.168.168.65:APEpool2 failed: Failed: APEpool1/media (APEpool1-manual-20150827)] relating to the manual snapshot and not the automatic one that had just run from the periodic snapshot. I thought replication could only be done against periodic snapshots?
I destroyed the manual snapshot and then received the same error relating to the auto snapshot [CRITICAL: Replication APEpool1/media -> 192.168.168.65:APEpool2 failed: Failed: APEpool1/media (auto-20150827.1309-1w)]
So to me, it looks like replication is broken in 9.3.1 :( Here's the output from the console during this:
Code:
Aug 27 13:09:01 freenas1 autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "APEpool1/media@auto-20150827.1309-1w"
Aug 27 13:11:30 freenas1 notifier: Performing sanity check on sshd configuration.
Aug 27 13:12:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:13:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:13:25 freenas1 notifier: Performing sanity check on sshd configuration.
Aug 27 13:13:33 freenas1 manage.py: [py.warnings:206] /usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py:206: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
else unicode(excp.message)
Aug 27 13:13:38 freenas1 notifier: Error: near line 1: database is locked
Aug 27 13:14:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:15:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:15:33 freenas1 notifier: Stopping collectd.
Aug 27 13:15:35 freenas1 notifier: Waiting for PIDS: 47388.
Aug 27 13:15:35 freenas1 notifier: Starting collectd.
Aug 27 13:16:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:17:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:18:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:19:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:20:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:21:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:22:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:23:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:24:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:25:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:26:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:27:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:28:02 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:29:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
Aug 27 13:30:01 freenas1 autorepl.py: [common.pipesubr:71] Popen()ing: /usr/bin/ssh -ononeenabled=yes -ononeswitch=yes -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 192.168.168.65 "zfs list -H -t snapshot -p -o name,creation -r 'APEpool2/media'"
UPDATE
As I've set this up slightly differently, I thought I'd try adding the dataset on APEpool2 (so APEpool2/media) and then adding this into the Remote ZFS Volume/Dataset field in the Replication Task. It now appears to be replicating, although on APEpool2 it's created another dataset (so APEpool2/media/media) which isn't really what I'm looking for!
Here's the Replication Task settings: