Replication Timeout

[Lukey1]

Wondering if anyone can help me with this.

I have 2 Truenas systems in 2 difference sites and i am trying to get my main system to replicate to my offsite system. When i select the destination system on my main my offsite shows up and i can see everything just fine but once i hit the run button for the replication task i works for about a second and then errors. Then i go back and edit the destination and im unable to see my offsite system for about 10 minutes i just get an error (EACCESS) Timeout.



Error: Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zettarepl.py", line 410, in list_datasets
datasets = await self.middleware.run_in_thread(list_datasets, shell)
File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/zettarepl/dataset/list.py", line 13, in list_datasets
return [dataset["name"] for dataset in list_datasets_with_properties(shell, dataset, recursive)]
File "/usr/local/lib/python3.8/site-packages/zettarepl/dataset/list.py", line 30, in list_datasets_with_properties
output = shell.exec(args)
File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/interface.py", line 84, in exec
return self.exec_async(args, encoding, stdout).wait()
File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/interface.py", line 88, in exec_async
async_exec.run()
File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/base_ssh.py", line 26, in run
client = self.shell.get_client()
File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/base_ssh.py", line 99, in get_client
client.connect(
File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.8/site-packages/paramiko/util.py", line 283, in retry_on_signal
return function()
File "/usr/local/lib/python3.8/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/replication.py", line 650, in list_datasets
return await self.middleware.call("zettarepl.list_datasets", transport, ssh_credentials)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
return await self._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zettarepl.py", line 412, in list_datasets
raise CallError(repr(e).replace("[Errno None] ", ""), errno=errno.EACCES)
middlewared.service_exception.CallError: [EACCES] timeout('timed out')








Also the two systems are connected via a VPN i have a Ubiquiti UDM-Pro where my main system is and a Ubiquiti USG 3P on the offsite end. They are connected using Manual IPsec.



I also ran a ping test while running the replication task on my laptop over the VPN to my Truenas Backup, the ping worked the whole time but replication task failed.


The throughput between the two sites should be around 5MBs 50Mbs on one end 50Mbs on the other






Before running the task

2 Seconds after running the task

and then this after about 10 seconds, then i will have to wait about 10 - 15 minutes for it to look like the first screenshot

 

[artlessknave]

er, isn't 5 MB/s.... really slow? I think ping would work fine but I'm not sure anything else would.
it seems to me that it is simply doing exactly what it says, it's timing out because the connection is too slow.