[2023/01/02 21:25:49] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27833] Running ['ps', '-o', 'command', '-p', '69643']
[2023/01/02 21:25:49] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.paramiko.replication_task__task_1] [chan 133] Max packet in: 32768 bytes
[2023/01/02 21:25:49] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 133] Max packet out: 32768 bytes
[2023/01/02 21:25:49] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] Secsh channel 133 opened.
[2023/01/02 21:25:49] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 133] Sesch channel 133 request ok
[2023/01/02 21:25:49] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27833] Reading stdout
[2023/01/02 21:25:49] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 133] EOF received (133)
[2023/01/02 21:25:49] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27833] Waiting for exit status
[2023/01/02 21:25:49] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 133] EOF sent (133)
[2023/01/02 21:25:49] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27833] Success: 'COMMAND\nzfs: sending volume1/some-dataset@daily-2023-01-02_21-00 (84%: 3759644688/4471797000) (zfs)\n'
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27834] Running ['ps', '-o', 'command', '-p', '69643']
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.paramiko.replication_task__task_1] [chan 134] Max packet in: 32768 bytes
[2023/01/02 21:25:59] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 134] Max packet out: 32768 bytes
[2023/01/02 21:25:59] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] Secsh channel 134 opened.
[2023/01/02 21:25:59] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 134] Sesch channel 134 request ok
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27834] Reading stdout
[2023/01/02 21:25:59] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 134] EOF received (134)
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27834] Waiting for exit status
[2023/01/02 21:25:59] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] [chan 134] EOF sent (134)
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.base_ssh] [ssh:root@nasa.example.org] [shell:102] [async_exec:27834] Error 1: 'COMMAND\n'
[2023/01/02 21:25:59] DEBUG    [replication_task__task_1.progress_observer] [zettarepl.transport.progress_report_mixin] zfs send with PID 69643 is gone
[2023/01/02 21:26:00] DEBUG    [replication_task__task_1.async_exec_tee.wait] [zettarepl.transport.local] [shell:1] [async_exec:27727] Error 255: None
[2023/01/02 21:26:00] DEBUG    [replication_task__task_1.process] [zettarepl.transport.local] [shell:1] [async_exec:27726] Error 255: 'ssh_dispatch_run_fatal: Connection to 10.xx.x.Y port 22: message authentication code incorrect\n  (stdin): Unexpected EOF in worker 3\ncannot receive incremental stream: checksum mismatch or incomplete stream.\nPartially received snapshot is saved.\nA resumi....5f94959a9c9250c0cdbe2218661c8a7a515a796806412e0f26c48f2499525a9c540bae2f0ba7bd8f497e4435ce1bca825ae3f6ca28503923c27583e2f31379581a12c3fa73437d5503f29be38b5a82cb5c821253133a752d7c8c0c858d7c050d7c028de08481a80cde76640f83b393fb7a028b5b8383f9b010e00c3fc2cf2\n'
[2023/01/02 21:26:00] DEBUG    [replication_task__task_1.monitor] [zettarepl.transport.local] [shell:1] [async_exec:27727] Stopping
[2023/01/02 21:26:00] ERROR    [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(255, 'ssh_dispatch_run_fatal: Connection to 10.xx.x.Y port 22: message authentication code incorrect\n  (stdin): Unexpected EOF in worker 3\ncannot receive incremental stream: checksum mismatch or incomplete stream.\nPartially received snapshot is saved.\nA resuming stream can be generated on the sending system by running:\n    zfs send -t 1-1476614e32-118-789c636064000310a501c49c50360710a715e5e7a69766a6304081f9d58ac95a6f9c6a15806c762475f94959a9c9250c0cdbe2218661c8a7a515a796806412e0f26c48f2499525a9c540bae2f0ba7bd8f497e4435ce1bca825ae3f6ca28503923c27583e2f31379581a12c3fa73437d5503f29be38b5a82cb5c821253133a752d7c8c0c858d7c050d7c028de08481a80cde76640f83b393fb7a028b5b8383f9b010e00c3fc2cf2\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 181, in run_replication_tasks
    retry_stuck_replication(
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/stuck.py", line 18, in retry_stuck_replication
    return func()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 182, in <lambda>
    lambda: run_replication_task_part(replication_task, source_dataset, src_context, dst_context,
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 279, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 603, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, encryption, observer)
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 686, in replicate_snapshots
    run_replication_step(step, observer)
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 763, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/ssh.py", line 153, in wait
    stdout = self.async_exec.wait()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/async_exec_tee.py", line 104, in wait
    raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: ssh_dispatch_run_fatal: Connection to 10.xx.x.Y port 22: message authentication code incorrect
  (stdin): Unexpected EOF in worker 3
cannot receive incremental stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1476614e32-118-789c636064000310a501c49c50360710a715e5e7a69766a6304081f9d58ac95a6f9c6a15806c762475f94959a9c9250c0cdbe2218661c8a7a515a796806412e0f26c48f2499525a9c540bae2f0ba7bd8f497e4435ce1bca825ae3f6ca28503923c27583e2f31379581a12c3fa73437d5503f29be38b5a82cb5c821253133a752d7c8c0c858d7c050d7c028de08481a80cde76640f83b393fb7a028b5b8383f9b010e00c3fc2cf2
[2023/01/02 21:26:00] DEBUG    [Thread-200] [zettarepl.paramiko.replication_task__task_1] EOF in transport thread