FreeNAS to O3X Replication: SSH+Netcat errors

[Zandr Milewski]

For a several years now I've been running a Thunderbolt 2 RAID0 (4x4TB) connected to my Mac (SoftRAID), backed up to a FreeNAS box with a RAIDZ2 (6x4TB). A different project here opened my eyes to how performant ZFS replication is compared to filesystem backups, so I'm now experimenting with OpenZFS on OS X (O3X). After making a second backup on local storage, I've rebuilt the Thunderbolt array as a 4x4TB stripe in O3X, and am currently replicating the FreeNAS backup back to the TB array.

[EDIT: I'm driving this replication as a "PUSH" from the FreeNAS box.]

This FreeNAS box is old and slow, however (Turion N40L Microserver), so I'm only seeing about 45MB/s. CPU usage is in the 60% range (dual core), which makes me think that I'm CPU bound by a single ssh thread doing encryption. Enter SSH+Netcat, right? That earlier project included an identical box running happily at wire speed with encryption turned off.

Unfortunately, that doesn't appear to work. I've verified that nc is in the path when connecting with remote commands over ssh (zfs wasn't, initially), so I'm not sure how to troubleshoot further. Here's the full log:

Code:

[2020/03/14 10:37:51] INFO     [Thread-7] [zettarepl.paramiko.replication_task__task_3] Connected (version 2.0, client OpenSSH_8.1)
[2020/03/14 10:37:51] INFO     [Thread-7] [zettarepl.paramiko.replication_task__task_3] Authentication (publickey) successful!
[2020/03/14 10:37:51] INFO     [replication_task__task_3] [zettarepl.replication.run] For replication task 'task_3': doing push from 'Deckard/Photographs' to 'ThunderBay' of snapshot='auto-2020-03-14_10-37' incremental_base=None receive_resume_token=None
[2020/03/14 10:37:51] INFO     [replication_task__task_3] [zettarepl.paramiko.replication_task__task_3.sftp] [chan 4] Opened sftp connection (server version 3)
[2020/03/14 10:37:51] ERROR    [replication_task__task_3] [zettarepl.replication.run] For task 'task_3' unhandled replication error JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 139, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 200, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 372, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 427, in replicate_snapshots
    run_replication_step(step_template.instantiate(incremental_base=incremental_base, snapshot=snapshot), observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 481, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 22, in run
    self.replication_process.run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh_netcat.py", line 128, in run
    listen = self.listen_exec.head(self._parse_listen_exec, 10)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/async_exec_tee.py", line 56, in head
    result = callback(event.data)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh_netcat.py", line 199, in _parse_listen_exec
    return json.loads(data)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

 

[Rchtr]

@Zandr Milewski
Do you have any solution four your error "JSONDecodeError"?

I want to backup my TrueNas System to Ubuntu Server using a replication Task via SSH on the TrueNas side.

Both systems use the same zfs version and the ssh connection works fine.

But I've got the same error...

 

[s851]

Same error here. Any news on this?

 

[Rchtr]

I've created the replication task via the wizard. There I set the option "encrypted ssh transportation".
When I've checked the options manually the transport option was set to SSH+NETCAT. I've changed it to SSH.

Now it works.

Be sure you add the push and the pull host in the "known hosts" file on the correspondending side.

 

[s851]

I can confirm that using the wizard I was able to work around the error. I've created my replication task using the wizard and unencrypted transport, which resulted in ssh+netcat. As soon as the task got created, I tweaked the settings (e.g. replicate full filesystem) using advanced mode.
For my first pool this worked fine. For my second pool, I had to run the task once before tweaking settings in advanced mode or the error would appear again. Unfortunately very hard to trust such a system with something as important as a system backup.