Replicating dataset randomly gets error - dataset does not exist

[azayakabaka]

System: TrueNAS Mini XL
TrueNAS version: TrueNAS-SCALE-22.02.1

I set up a couple replication tasks, both remote to rsync.net and local to another dataset, using this guide by TrueCharts. My replication was randomly, and is now constantly, giving an error similar to the one below (the long hash changes). This happens for both local and remote.

Code:

 [EFAULT] cannot open 'data/ix-applications/docker/7910c763bf26f95094800f8cb277b6568b1b2a50c2eb810f7ed6578827bcf3f0': dataset does not exist.

Here's what I got out of the log:

Code:

[2022/07/11 09:55:30] ERROR    [replication_task__task_2] [zettarepl.replication.run] For task 'task_2' unhandled replication error DatasetDoesNotExistException(1, "cannot open 'data/ix-applications/docker/024190454f7c7bc675ba126f3865aa933fbe2c2a201501712f05ff39e5d5c734': dataset does not exist\n")
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 167, in run_replication_tasks
    retry_contains_partially_complete_state(
  File "/usr/lib/python3/dist-packages/zettarepl/replication/partially_complete_state.py", line 16, in retry_contains_partially_complete_state
    return func()
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 168, in <lambda>
    lambda: run_replication_task_part(replication_task, source_dataset, src_context, dst_context,
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 243, in run_replication_task_part
    step_templates = calculate_replication_step_templates(replication_task, source_dataset,
  File "/usr/lib/python3/dist-packages/zettarepl/replication/run.py", line 363, in calculate_replication_step_templates
    for property, (value, source) in get_properties(
  File "/usr/lib/python3/dist-packages/zettarepl/transport/zfscli/__init__.py", line 110, in get_properties
    return get_properties_recursive(shell, [dataset], properties, include_source)[dataset]
  File "/usr/lib/python3/dist-packages/zettarepl/transport/zfscli/__init__.py", line 96, in get_properties_recursive
    output = shell.exec(cmd)
  File "/usr/lib/python3/dist-packages/zettarepl/transport/zfscli/exception.py", line 29, in __exit__
    raise DatasetDoesNotExistException(exc_val.returncode, exc_val.stdout) from None
zettarepl.transport.zfscli.exception.DatasetDoesNotExistException: cannot open 'data/ix-applications/docker/024190454f7c7bc675ba126f3865aa933fbe2c2a201501712f05ff39e5d5c734': dataset does not exist

Here's the configuration for the remote rsync.net job:

I've been struggling with this for months, and even replaced the SATA cables since I was told that could be a potential cause of weirdness, but to no avail. Any help is much appreciated.

Thank you!

 

[ggiinnoo]

Did you ever find an solution to tis problem?

 

[ggiinnoo]

I found out the problem btw.
The issue is that it goes from non encrypted -> encrypted and that the key is not loaded.