NAS Replication Complications

[w14219]

I have tried to setup the replication according to the documentation and am having issues.

• Build: FreeNAS-9.10.1-U4 (ec9a7d3)
• NAS-01: Push
• NAS-02: Pull
• Directions: http://doc.freenas.org/9.3/freenas_storage.html section 8.2

Issue:
The replication from NAS-01 to NAS-02 is failing. I am getting the following error.
Status: Failed: CoreVolume (auto-20161123.1506-2w)

Troubleshooting Efforts:

NAS-01: ssh -vv -i /data/ssh/replication NAS-02
XXX@freenas:~ % ssh -vv -i /data/ssh/replication NAS-02
OpenSSH_7.2p2, OpenSSL 1.0.1t-freebsd 3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 3: Deprecated option "useroaming"
debug2: resolving "NAS-02" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to NAS-02 [IP_Address] port 22.
debug1: Connection established.
debug1: identity file /data/ssh/replication type 1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /data/ssh/replication-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2 FreeBSD-20160310
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2 FreeBSD-20160310
debug1: match: OpenSSH_7.2 FreeBSD-20160310 pat OpenSSH* compat 0x04000000
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to NAS-02:22 as 'XXX'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss-cert-v01@openssh.com,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:KJHAkajshdKJHKJHKDJS
DNS lookup error: general failure
debug1: Host 'NAS-02' is known and matches the ECDSA host key.
debug1: Found key in /mnt/CoreVolume/home/XXX/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /data/ssh/replication (0x8038160c0), explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /data/ssh/replication
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:dfghdhdfhgdASDADSfr
Load key "/data/ssh/replication": Permission denied
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
XXX@NAS-02's password:



NAS-01: sudo zfs send CoreVolume@auto-20161123.1506-2w | ssh -i /data/ssh/replication NAS-02 zfs receive /mnt/RAID/Backups@auto-20161123.1506-2w

Password:Load key "/data/ssh/replication": Permission denied
XXX@NAS-02's password:
Permission denied, please try again.
XXX@NAS-02's password:
Permission denied, please try again.
XXX@NAS-02's password:
Sorry, try again.
Password:
Permission denied (publickey,password).
sudo: 1 incorrect password attempt

NAS-02: sudo vi auth.log
Nov 25 06:37:21 NAS-02 sshd[7072]: Failed password for XXX from <IP ADDRESS> port 26512 ssh2: RSA SHA256:KJHAkajshdKJHKJHKDJS
Nov 25 06:37:30 NAS-02 sshd[7072]: Failed password for XXX from <IP ADDRESS> port 26512 ssh2
Nov 25 06:37:34 NAS-02 sshd[7072]: Accepted password for XXX from <IP ADDRESS> port 26512 ssh2
Nov 25 06:37:34 NAS-02 sshd[7081]: Received disconnect from <IP ADDRESS> port 26512:11: disconnected by user
Nov 25 06:37:34 NAS-02 sshd[7081]: Disconnected from <IP ADDRESS> port 26512
Nov 25 06:37:49 NAS-02 sshd[7083]: Failed password for XXX from <IP ADDRESS> port 61621 ssh2: RSA SHA256:KJHAkajshdKJHKJHKDJS
Nov 25 06:37:51 NAS-02 sshd[7083]: Accepted password for XXX from <IP ADDRESS> port 61621 ssh2
Nov 25 06:37:52 NAS-02 sshd[7100]: Received disconnect from <IP ADDRESS> port 61621:11: disconnected by user
Nov 25 06:37:52 NAS-02 sshd[7100]: Disconnected from <IP ADDRESS> port 61621

NAS-01: /var/log/messages
Nov 24 00:00:00 freenas newsyslog[10171]: logfile turned over due to size>100K
Nov 24 00:00:00 freenas syslog-ng[1278]: Configuration reload request received, reloading configuration;
Nov 24 09:00:08 freenas autosnap.py: [tools.autosnap:66] Popen()ing: /sbin/zfs snapshot "CoreVolume@auto-20161124.0900-2w"
Nov 25 00:00:00 freenas syslog-ng[1278]: Configuration reload request received, reloading configuration;
Nov 25 04:45:55 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc'
Nov 25 04:45:55 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: mount
Nov 25 04:45:55 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: mount
Nov 25 04:45:55 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: mount
Nov 25 04:45:55 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: /usr/local/bin/net -d 0 getlocalsid
Nov 25 04:45:57 freenas notifier: Performing sanity check on Samba configuration: OK
Nov 25 04:45:57 freenas notifier: Reloading nmbd.
Nov 25 04:45:57 freenas notifier: Reloading smbd.
Nov 25 04:45:57 freenas notifier: Reloading winbindd.
Nov 25 04:45:57 freenas notifier: Stopping mdnsd.
Nov 25 04:45:57 freenas smbd: dnssd_clientstub DNSServiceProcessResult called with DNSServiceRef with no ProcessReply function
Nov 25 04:45:57 freenas notifier: Waiting for PIDS: 2578.
Nov 25 04:45:57 freenas notifier: Starting mdnsd.
Nov 25 04:46:31 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 04:53:33 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 04:54:54 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 04:56:13 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 04:56:50 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 05:25:32 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 05:26:54 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 05:34:15 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 05:39:43 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 06:00:12 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 06:04:33 freenas notifier: Performing sanity check on sshd configuration.
Nov 25 06:05:25 freenas notifier: Performing sanity check on sshd configuration.


Any Ideas as to why this is not working?

Thanks

 

[Sakuru]

First of all, I highly recommend using the 9.10 documentation, not 9.3.
Second of all, it doesn't look like you have the keys set up correctly. Read through the 9.10 documentation I linked and try again.

 

[w14219]

Thank you for your feedback. The documentation between the two versions were very similar, if not identical. However, I did find the issue. I did not configure the "Remote ZFS Volume/Dataset" properly on the push machine. once I entered the correct information from the pull machine, I was golden. The certificates and accounts were correct to begin with. The error I received in the logs were misleading to me.

Thank you for your support.