Veeam copy jobs failing after upgrade to TrueNAS 13 - Failed to decompress LZ4 block

[Cruiseader]

We are using a TrueNAS array as a backup repository for Veeam. After upgrading from TrueNAS 12.0-U8.1 to 13.0-U2 and 13.0-U3.1, the backups start failing. I upgraded to version 13.0-U2 a while ago and had to revert. I just tried upgrading to version 13.0-U3.1 and get the same issue. We get the following message in Veeam:

12/14/2022 2:30:26 PM :: Processing <<servername>> Error: Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-178321', expected length: '1048576').
Failed to download disk '1f7c25a0-d2ae-4642-ba7b-5454a5561221'.
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [1f7c25a0-d2ae-4642-ba7b-5454a5561221];
Agent failed to process method {DataTransfer.SyncDisk}.

I have tried troubleshooting this with Veeam, but have exhausted the options there. We've totally rebuilt the pool in TrueNAS and created brand new backup jobs, repositories and everything in Veeam but still get the same error. Reverting the system back to TrueNAS 12.0-U8.1 resolves the issue. Any ideas why we'd be getting a decompression error after upgrading to version 13.0? The only thing I didn't upgrade was the ZFS version since we needed to revert back to version 12.0. I'm at a loss as to what to try next...

Thank you!

 

[alex992]

Is your file over 16 TB ?

 

[Cruiseader]

Is your file over 16 TB ?

It is not. The largest file is 5.5TB

 

[Cruiseader]

I should clarify the issue. The backups in Veeam run fine. We are copying the files to a secondary repository via Veeam which is backed by TrueNAS 13.0-U3.1. It's this copy that is failing. Trying to read the files from the primary repository causes the error. We are connected to the TrueNAS devices via iSCSI and the volumes are formatted as ReFS. Reverting the primary repository back to TrueNAS 12.0-U8.1 resolves the issue.

 

[morganL]

I should clarify the issue. The backups in Veeam run fine. We are copying the files to a secondary repository via Veeam which is backed by TrueNAS 13.0-U3.1. It's this copy that is failing. Trying to read the files from the primary repository causes the error. We are connected to the TrueNAS devices via iSCSI and the volumes are formatted as ReFS. Reverting the primary repository back to TrueNAS 12.0-U8.1 resolves the issue.

There are two TrueNAS systems... lets isolate the problem.

Can you confirm whether the files in the primary repository are OK when they are used for restoration? both 12.0-U8.1 and 13.0-U3.1?

If you copy the files from a primary running 12.0-U8.1 to a secondary running 13.0-U3.1 is there an issue ?

 

[Cruiseader]

There are two TrueNAS systems... lets isolate the problem.

Can you confirm whether the files in the primary repository are OK when they are used for restoration? both 12.0-U8.1 and 13.0-U3.1?

If you copy the files from a primary running 12.0-U8.1 to a secondary running 13.0-U3.1 is there an issue ?

I was able to restore from the primary repository running on 12.0-U8.1 and on 13.0-U2. I didn't try the restore when it was running 13.0-U3.1 but would expect the restore would have worked fine. I am able to copy files between the primary repository running 12.0-U8.1 and our secondary repository running 13.0-U3.1 without issue. It's only when we upgrade the primary repository to 13.0 that we start having issues with the copy.

 

[morganL]

Its a bit difficult to understand how these two things could be true:
1. Primary running 13.0-U3.1 can be primary backup and restore successfully
2. Primary running 13.0-U3.1 can be the cause of an unsuccessful copy using the same protocol - knowing that secondary previously worked,

The Veeam message about LZ4 would be due to their compression.... not TrueNAS compression
I'd suggest confirming that the 13.0-U3.1 can restore or not.
If it can restore, there is no data integrity issue.
If it can't restore...there is a data integrity issue (eg broken compression algorithm somewhere). Lets report this as a bug for our engineerring team to review.

 

[ljt3705]

I also meet this Failed to decompress LZ4 block problem. My environment is TrueNAS-13.0-U2 VM running on a vmware esxi host, with a 30TB vmdk disk as backup repository, using SMB and NFS protocol to connect to a Veeam backup replication 11 server (which also a VM on the same esxi host).

After many times of trying, I have found the following:
1. I use Veeam to backup several VMs regularly with its default backup parameters (enable lz4 compression), in total size about 6TB+, using SMB protocol. When Veeam create first full backup I found no problem. the backup process complete successfully and then I use Veeam's built in health check function to verify the backup file.

2. After first full backup, next time Veeam will use incremental mode to perform backup. At this time the problem could have 50% chance to happen. the backup process complete successfully but health check failed.

3. I have tried to switch to NFS protocol, the chance down to 30%, only better than using SMB.