TrueNAS 12-U8.1 Replication - unsupported feature flag 3420004 ?!?

[Richard Durso]

Replication task to an Ubuntu 20.04.4 LTS server. Replication started to fail with the following message last month.​

I've not been able to find any information on this feature flag number. I assume this means a ZFS feature flag.

Code:

Replication "main/vm_storage -> rpool/backups/vm_storage" failed: stream has unsupported feature, feature flags = 3420004
No error..

So clearly something was enabled on TrueNAS that Ubuntu doesn't like. My issue is that feature flags are not documented in numbers like that. I have no idea what "3420004" feature flag represents. And not sure what the "No error" is about since the replication fails. Whatever this is, seems specific to this dataset as all other replication tasks to the server continue to work fine.

Logs from Task:

Code:

[2022/07/18 23:00:00] INFO     [Thread-10891] [zettarepl.paramiko.replication_task__task_12] Connected (version 2.0, client OpenSSH_8.2p1)
[2022/07/18 23:00:00] INFO     [Thread-10891] [zettarepl.paramiko.replication_task__task_12] Authentication (publickey) successful!
[2022/07/18 23:00:02] INFO     [replication_task__task_12] [zettarepl.replication.run] For replication task 'task_12': doing push from 'main/vm_storage' to 'rpool/backups/vm_storage' of snapshot='auto-20220618.0500-3m' incremental_base='auto-20220611.0500-3m' receive_resume_token=None encryption=False
[2022/07/18 23:00:02] INFO     [replication_task__task_12] [zettarepl.paramiko.replication_task__task_12.sftp] [chan 9] Opened sftp connection (server version 3)
[2022/07/18 23:00:02] INFO     [replication_task__task_12] [zettarepl.transport.ssh_netcat] Automatically chose connect address '192.168.10.102'
[2022/07/18 23:00:08] ERROR    [replication_task__task_12] [zettarepl.replication.run] For task 'task_12' unhandled replication error SshNetcatExecException(ExecException(1, 'stream has unsupported feature, feature flags = 3420004\n'), None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 164, in run_replication_tasks
    retry_stuck_replication(
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/stuck.py", line 18, in retry_stuck_replication
... 9 more lines ...
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 764, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/ssh_netcat.py", line 198, in wait
    raise SshNetcatExecException(connect_exec_error, self.listen_exec_error) from None
zettarepl.transport.ssh_netcat.SshNetcatExecException: stream has unsupported feature, feature flags = 3420004
No error

 

[Bonnie Follweiler]

Did anything change in the TrueNAS, or the receiving system, between the time it was working and the time it stopped working?
I would encourage you to create a support ticket in from your TrueNAS UI.

Please note: If you don't have a JIRA account you will need to create one. There is a link in the UI to create an account in the Contact Support area to assist you.

To report your issue:
Log into your TrueNAS, navigate to the System menu, and then select Support. Fill out the Contact Support information and click the Attach Debug checkbox. Attach any screenshots you have. If possible, in the ticket's Description area, provide the previous steps that lead to the error message (to assist in diagnosing/reproduction). Click Submit.

 

[Samuel Tai]

This is because Ubuntu 20.04 runs ZFS 0.8.3, which can't handle features from an OpenZFS 2.0 ZFS stream from TrueNAS 12.0-U8.1. You'll need to install the Jonathon F PPA to install ZFS on Linux 2.1, which does support the 2.0 feature flags.

Note, the PPA requires DKMS to install, and you'll also need to delete the bundled 0.8.3 ZFS modules in /lib/modules/<running kernel version>/kernel/zfs to have the DKMS modules load on boot.

Alternatively, you could upgrade to Ubuntu 22.04, which runs OpenZFS 2.1.

 

[Richard Durso]

This is because Ubuntu 20.04 runs ZFS 0.8.3, which can't handle features from an OpenZFS 2.0 ZFS stream from TrueNAS 12.0-U8.1. You'll need to install the Jonathon F PPA to install ZFS on Linux 2.1, which does support the 2.0 feature flags.

Note, the PPA requires DKMS to install, and you'll also need to delete the bundled 0.8.3 ZFS modules in /lib/modules/<running kernel version>/kernel/zfs to have the DKMS modules load on boot.

Alternatively, you could upgrade to Ubuntu 22.04, which runs OpenZFS 2.1.

I suspect this is correct, but without knowing what that code means, I'm not sure how to confirm something was enabled on that dataset.
For now, it might be simpler to create a replacement dataset and copy the data vs switching to DKMS or doing an OS upgrade on the target.

 

[Samuel Tai]

I don't know what that code means either. I wasn't able to find it in the source code for zettarepl, but if it's related to ZFS features, you should check out @Ericloewe's resource on ZFS features. If you do decide to recreate the dataset, you'll need to restrict it to the features for 11.3 to work with Ubuntu 20.04.