Cloud Sync SFTP corrupted files

[Crekate]

Hello,
Recently I've setup a cloud sync task to backup my cloud storage.
I've connected to Egnyte via SFTP and I run the sync every day.

Unfortunately some files got corrupted but the sync reports success and no issues found in the logs. I just cannot open some files (error: 0xc00d36e5).
For example:
I have an .mp4 files.
I can open it in Egnyte.
I can download it from Egnyte and open it.
I cannot open it from TrueNas.
I can download it from TrueNas but I cannot open it.
When I manually copy the file from Egnyte to TrueNas the file is ok and I can open it.

I've also tried deleting the files and running the sync again. Corrupted files again. Tried it multiple times. The files have same size but not all metadata are present on TrueNas side.

Any ideas what might cause such a corruption every time?

 

[Crekate]

Anyone?

 

[Ericloewe]

The files have same size but not all metadata are present on TrueNas side.

Do you mean metadata that is part of the files themselves, or metadata at the filesystem level? Can/cannot open is a bit unspecific, I take it that the hashes do not match between the file as downloaded directly from the cloud service and via SFTP? What happens if you just use a normal SFTP client and get them manually?

 

[Crekate]

Thank you for the replay.
Here are the metadata the file has on Egnyte and when download from Egnyte:

Here are the metadata the file has in TN after sync:

When I copy the file from Egnyte to PC I can open the file - no problem.
When I copy the file manually from Egnyte to TN, I can open the file - no problem.


When I try to open the file from trueNas after sync automaticly I have following error:

The file cannot be opened.

All files (on TN, on PC and Egnyte) have the same size.

 

[Ericloewe]

When I copy the file from Egnyte to PC I can open the file - no problem.
When I copy the file manually from Egnyte to TN, I can open the file - no problem.

Using SFTP?

 

[Crekate]

Yes using SFTP.
On PC I use WinSCP

 

[Crekate]

Anyone?

 

[HoneyBadger]

What is the SHA256 sum (hash) of the file when it's transferred via SFTP/cloud sync, vs when you manually download it from Egnyte to your PC and again manually from Egnyte to TN?

It would seem unusual that the use of the cloud-sync task would result in failure.

 

[Crekate]

Here is has for the file copied with cloud sync:

Here is from Egnyte to PC:

Here is from Egnyte to TN (manually):

The first one differs.

 

[HoneyBadger]

If you delete and re-copy the file again via cloud sync, does it result in the same hash, or a different one? If it's the same hash, it means that something is generating consistently wrong data in transit.

 

[Crekate]

I will test the hash, but from what I've seen, if I delete the file, sync it again via cloud sync, it will be corrupted. I've tried it multiple times, hoping that just deleting and resyncing will help, but it doesn't. I will check the hash and post the result.

 

[Crekate]

I finally was able to test the delete -> cloudsync hash. Here it is:

The file is still corrupted and the hash is the same as before (as the corrupted file before delete).

Any1 has any idea what to do?

 

[Ericloewe]

Is this really SFTP or some stupid variant of FTP? I ask because some (all?) of them can be coaxed into doing stupid things like converting between character sets (which, in 2024, is somewhere between "active irritant" and "supremely useless"), which might begin to explain the problem.

Also, does the file size match?

 

[Crekate]

Here is the size on Egnyte:

Here is the size on TN after sync:

Here is the setup I have in TN:

The same host, port, username and password I use in WinSCP and with WinSCP the file is copied corretly.

 

[Ericloewe]

I'm nearly out of ideas here. This is truly bizarre.

I think you need to narrow this down further and identify who's doing something they shouldn't be. Try using more, different clients and see which ones reproduce the issue. And see if manually transferring the data using the sftp client on TrueNAS produces the same results.

 

[Crekate]

Do you mean manually transfer the data using sftp client on TrueNas? I have no ideaa what Reue As is.

 

[Ericloewe]

Yeah, TrueNAS. Sorry about that, either I need more sleep or autocorrect on my phone is seriously acting up today.