Apologies... you're quite right. Here are the details.
I setup a new replication between two 9.2.1.3 boxes. PULL was upgraded from 9.2. PUSH was upgraded from 9.2.1.2.
Snapshots are setup on pool "alpha" as recursive, every 2 hours.
Replication is setup to replicate from "alpha" on PUSH to "beta" on PULL.
Both pools are encrypted. After upgrading to 9.2.1.3 on PUSH, I unlocked the pool "alpha". The PULL box was already booted and unlocked. The replication immediately failed though the log implies replication was attempted prior to finishing the unlock.
Log:
Code:
Mar 21 10:35:35 ironthrone kernel: GEOM_ELI: Device gptid/98aaaec1-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:35:35 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:35:35 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:35:44 ironthrone kernel: GEOM_ELI: Device gptid/994bae4c-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:35:44 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:35:44 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:35:54 ironthrone kernel: GEOM_ELI: Device gptid/9a917149-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:35:54 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:35:54 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:36:01 ironthrone autosnap.py: [tools.autosnap:58] Popen()ing: /sbin/zfs snapshot -r -o freenas:state=NEW alpha@auto-20140321.1036-2m
Mar 21 10:36:01 ironthrone autosnap.py: [tools.autosnap:234] Failed to create snapshot 'alpha@auto-20140321.1036-2m': cannot open 'alpha': dataset does not exist usage: snapshot|snap [-r] [-o property=value] ... <filesystem|volume>@<snap> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
Mar 21 10:36:02 ironthrone kernel: GEOM_ELI: Device gptid/9c39455f-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:36:02 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:36:02 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:36:02 ironthrone autorepl.py: [tools.autorepl:195] Could not determine last available snapshot for dataset alpha: cannot open 'alpha': dataset does not exist
Mar 21 10:36:11 ironthrone kernel: GEOM_ELI: Device gptid/9dc8ee6f-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:36:11 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:36:11 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:36:20 ironthrone kernel: GEOM_ELI: Device gptid/9f5da50b-ae02-11e3-8d8f-005056b01bec.eli created.
Mar 21 10:36:20 ironthrone kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 21 10:36:20 ironthrone kernel: GEOM_ELI: Crypto: hardware
Mar 21 10:36:34 ironthrone notifier: Stopping collectd.
Mar 21 10:36:35 ironthrone notifier: Waiting for PIDS: 3059.
Mar 21 10:36:35 ironthrone notifier: Starting collectd.
I've had snapshots run since the unlock and no replication attempts logged. I disabled replication then re-enabled it to try and get replication to attempt again but it does not appear in the log as attempting replication.
The reports does show PUSH is TXing small amounts of data over the network and PULL is RXing small amounts of data. But nothing explains what this is. It has to be related to replication as if I turn off replication, this network activity dies. The TX and RX on the graphs happen once a minute or so then stop. It is picket fencing across the graph.
Same setup I had when both were on 9.2.0 and replication worked.
This is the web UI status:
Code:
CRITICAL: Replication alpha -> 172.21.14.7 failed: None
OK: The volume lab01 (ZFS) status is HEALTHY
OK: The volume alpha (ZFS) status is HEALTHY