SOLVED Can't get replication to work in 11.3-RC1 ([EFAULT] SSHException("Server '[remotehost.com]:3333' not found in known_hosts"))

[spitfire]

Hi,

I'm trying to get replication to work between 2 FreeNAS boxes, both run-in 11.3-RC1.
1st host is behind NAT, 2nd one is not - so I figured I'd let it initiate sync.
I followed 7.11 SSH Connections to set it up using the manual method. I've redirected ssh port in my firewall on box 2, and was able to SSH from box 1->2 (I've added ssh public key from box 1 to root on box 2) with keys.
I've tried to set up replication using following steps:

• Went to Tasks->Replication Tasks->Add
• Selected "On a Different System" in "Source Datasets" dropdown
• Used pre-created SSH connection in "SSH connections" dropdown

When trying to expand "Source datasets" (below "SSH Connections") I'm getting alert titled [EFAULT] SSHException("Server '[remotehost.com]:3333' not found in known_hosts") (hostname and port replaced on purpose). When I expand the dialog I'm getting:

Code:

File "/usr/local/lib/python3.7/site-packages/zettarepl/dataset/list.py", line 19, in list_datasets
    return list(filter(None, shell.exec(args).split("\n")))
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/interface.py", line 83, in exec
    return self.exec_async(args, encoding, stdout).wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/interface.py", line 87, in exec_async
    async_exec.run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/base_ssh.py", line 26, in run
    client = self.shell.get_client()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/base_ssh.py", line 88, in get_client
    auth_timeout=self.transport.connect_timeout,
  File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 416, in connect
    self, server_hostkey_name, server_key
  File "/usr/local/lib/python3.7/site-packages/paramiko/client.py", line 824, in missing_host_key
    "Server {!r} not found in known_hosts".format(hostname)
paramiko.ssh_exception.SSHException: Server '[remotehost.com]:3333' not found in known_hosts

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 128, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1113, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1061, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 949, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/replication.py", line 619, in list_datasets
    return await self.middleware.call("zettarepl.list_datasets", transport, ssh_credentials)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1122, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1061, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 354, in list_datasets
    raise CallError(repr(e))
middlewared.service_exception.CallError: [EFAULT] SSHException("Server '[remotehost.com]:3333' not found in known_hosts")

/root/.ssh/known_hosts contains entry for my 2nd server.
When I go to edit my SSH connection in System->SSH Connections and use the "Discover remote host key" I'm getting 2 different ones - 1st time it's an ssh-ed25519 key (matching /usr/local/etc/ssh/ssh_host_ed25519_key.pub on my 2nd box), the other time it's a ecdsa-sha2-nistp256 key (matching /usr/local/etc/ssh/ssh_host_ecdsa_key.pub, so I added both to /root/.ssh/known_hosts. I also tried symlinking /root/.ssh/known_hosts to both /etc/ssh/known_hosts and /usr/local/etc/ssh/known_hosts as I was not sure which one is being used.

 

[spitfire]

Fixed in https://jira.ixsystems.com/browse/NAS-104649