Cloud Sync Failing

[NAS0_DATA1]

So I got Cloud Sync setup on my TN Scale test VM. Credentials verification and dry run are both successful but when I try to actually run the task, it fails. It’s setup to sync to an iDrive e2 bucket. Configuration is the same as AWS S3 setup.

I created a simple dataset/share to test. I can access the share from other clients on the network. Which tells me permissions are fine. Or at least they look fine to me. The task is setup to SYNC. Not sure what I’m missing.

Hopefully someone has experienced this issue and can help. Thanks.

 

[NAS0_DATA1]

This is the log output when the task fails:

<3>ERROR : Proxmox/9-Step Calm and Easy Proxmox VE Boot Drive Failure Recovery - Page 2 of 2.pdf: corrupted on transfer: md5 hash differ "ea468d5d7d07786f9345cb2c68ec6f25" vs "38be00c28afce57e5d6629f512bd3763"

<6>INFO : Proxmox/9-Step Calm and Easy Proxmox VE Boot Drive Failure Recovery - Page 2 of 2.pdf: Removing failed copy

<3>ERROR : Proxmox/9-Step Calm and Easy Proxmox VE Boot Drive Failure Recovery.pdf: corrupted on transfer: md5 hash differ "4704e1e42420f137de792a980149d024" vs "a27dcfc3edd5ca42f53d4b3af9cde9f3"

<6>INFO : Proxmox/9-Step Calm and Easy Proxmox VE Boot Drive Failure Recovery.pdf: Removing failed copy

<3>ERROR : Proxmox/Add a new physical hard drive to Proxmox VE 4x 5x – HostFav Blog.pdf: corrupted on transfer: md5 hash differ "e7e0ee39eaed3420ee7cc13f3fc29a09" vs "1fa010f1a53452dd39fc996d27fede69"

<6>INFO : Proxmox/Add a new physical hard drive to Proxmox VE 4x 5x – HostFav Blog.pdf: Removing failed copy

<3>ERROR : Proxmox/How to Add Storage Drive to Proxmox 7.pdf: corrupted on transfer: md5 hash differ "32508ab529f83485ada9b178c3de9e98" vs "44000f328192cef20e81751910fd77f1"

<6>INFO : Proxmox/How to Add Storage Drive to Proxmox 7.pdf: Removing failed copy

<3>ERROR : Proxmox/Robocopy_ Copy Contents of one Drive to another_.pdf: corrupted on transfer: md5 hash differ "d0760a488df0d3e7d84aa093f59fe0b2" vs "70171710a411fd4ee3ca3e3934a06734"

<6>INFO : Proxmox/Robocopy_ Copy Contents of one Drive to another_.pdf: Removing failed copy
... 52 more lines ...

<3>ERROR : Proxmox/The Ultimate Guide to Robocopy.pdf: corrupted on transfer: md5 hash differ "d0d94289cb690b1f8d210df6a3eaa4ba" vs "68f6831da7652dc0c9741208a3c4746a"

<6>INFO : Proxmox/The Ultimate Guide to Robocopy.pdf: Removing failed copy

<3>ERROR : Proxmox/Proxmox VE Administration Guide.pdf: corrupted on transfer: md5 hash differ "4c702598069ae1fc969c91ddb6300cf1" vs "6b1a12f513c1684c80b03e1f70dc0f78"

<6>INFO : Proxmox/Proxmox VE Administration Guide.pdf: Removing failed copy

<3>ERROR : Proxmox/[TUTORIAL] - Proxmox Beginner Tutorial_ How to set up your first virtual machine on a secondary hard disk. _ Proxmox Support Forum.pdf: corrupted on transfer: md5 hash differ "862f7c9a30d0ec28f833e8994a2d4af5" vs "ab2705770d667cc0d5b69b88ec744f46"

<6>INFO : Proxmox/[TUTORIAL] - Proxmox Beginner Tutorial_ How to set up your first virtual machine on a secondary hard disk. _ Proxmox Support Forum.pdf: Removing failed copy

<3>ERROR : S3 bucket truenas-test path TrueNAS_Scale: not deleting files as there were IO errors

<3>ERROR : S3 bucket truenas-test path TrueNAS_Scale: not deleting directories as there were IO errors

<3>ERROR : Attempt 3/3 failed with 9 errors and: corrupted on transfer: md5 hash differ "862f7c9a30d0ec28f833e8994a2d4af5" vs "ab2705770d667cc0d5b69b88ec744f46"
Failed to sync with 9 errors: last error was: corrupted on transfer: md5 hash differ "862f7c9a30d0ec28f833e8994a2d4af5" vs "ab2705770d667cc0d5b69b88ec744f46"

 

[efortin]

Hi,

I have exactly the same setup and problem. As I activated versioning on IDrive e2, I can see the files were transferred. I've downloaded all versions which have been deleted to my local system and did a md5 on all of those and they are identical to the original file that is on the TrueNas Scale server. So, the problem doesn't seem to be while the Data Protection Sync is uploading the file as it would not have the same md5 hash when I download the erased version. But up to now, I can't figure why a perfectly valid file would be recognized as corrupted by the Data Replication Sync md5 hashing.

Any idea? I can provide log but they are exactly like what was provided above except with my own files name.

So I got Cloud Sync setup on my TN Scale test VM. Credentials verification and dry run are both successful but when I try to actually run the task, it fails. It’s setup to sync to an iDrive e2 bucket. Configuration is the same as AWS S3 setup.

I created a simple dataset/share to test. I can access the share from other clients on the network. Which tells me permissions are fine. Or at least they look fine to me. The task is setup to SYNC. Not sure what I’m missing.

Hopefully someone has experienced this issue and can help. Thanks.

Thank you.

efortin

 

[spry.key8595]

If you have SSE enabled on your S3 bucket, the Cloud Sync tasks will fail due to an rclone bug where it isn't checking SSE status and thus comparing the plaintext source against the destination encrypted file incorrectly. The following thus has to be true for this to be related to your problem:

1. Your "remote" is configured with S3 related credentials
2. You have a S3 bucket policy which uses server side encryption as your default encryption

Here is the rclone bug issue from 2017: https://github.com/rclone/rclone/issues/1824

 

[NAS0_DATA1]

Hi,

I have exactly the same setup and problem. As I activated versioning on IDrive e2, I can see the files were transferred. I've downloaded all versions which have been deleted to my local system and did a md5 on all of those and they are identical to the original file that is on the TrueNas Scale server. So, the problem doesn't seem to be while the Data Protection Sync is uploading the file as it would not have the same md5 hash when I download the erased version. But up to now, I can't figure why a perfectly valid file would be recognized as corrupted by the Data Replication Sync md5 hashing.

Any idea? I can provide log but they are exactly like what was provided above except with my own files name.



Thank you.

efortin

After my last post with the log output, I decided to create a Backblaze account to test. Created a new cloud sync task for Backblaze and it ran without any issues. Files were uploaded to my Backblaze bucket and I am able to view them and restore them as well. I’m wondering if it’s something to do with iDrive since technically TrueNAS doesn’t have it listed as a provider from the drop-down like it does for AWS, Backblaze and others.

 

[NAS0_DATA1]

If you have SSE enabled on your S3 bucket, the Cloud Sync tasks will fail due to an rclone bug where it isn't checking SSE status and thus comparing the plaintext source against the destination encrypted file incorrectly. The following thus has to be true for this to be related to your problem:

1. Your "remote" is configured with S3 related credentials
2. You have a S3 bucket policy which uses server side encryption as your default encryption

Here is the rclone bug issue from 2017: https://github.com/rclone/rclone/issues/1824

@spry.key8595 Thanks for the info. I went into my iDrive dashboard and created a new test bucket. I left "Default Encryption" unchecked/disabled. I then created a new access key for this bucket. On the TrueNAS server I setup a new backup credential and Cloud Sync task. Ran the task and it was successful. I confirmed the test files were synced to the iDrive bucket. I am able to view them and restore them as well. @efortin Enabling default encryption when creating a new bucket was the issue all along.