Cloud Sync Task To B2 Error

[StoreMore]

I have a Cloud Sync Task backup to B2 configured that has worked as expected the last several times. Everything was working as expected but now is failing with an error, "ValueError: rclone failed with exit code 3".

The only things that have changed since my last succesfull run were the update of my server to 11.3 U2 from U1. I also installed a Plex Media Server & Jail System. I am NOT backing up the jail or Plex to to this B2 location.

Here's the error messages/logs.

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
await self.future
File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py", line 1007, in sync
await rclone(self.middleware, job, cloud_sync)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py", line 229, in rclone
raise ValueError(message)
ValueError: rclone failed with exit code 3

--- log ---

Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 0s

2020/04/10 23:49:46 ERROR : : error reading source directory: directory not found
2020/04/10 23:49:46 INFO : Encrypted drive 'encrypted:/': Waiting for checks to finish
2020/04/10 23:49:46 INFO : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2020/04/10 23:49:46 ERROR : Encrypted drive 'encrypted:/': not deleting files as there were IO errors
2020/04/10 23:49:46 ERROR : Encrypted drive 'encrypted:/': not deleting directories as there were IO errors
2020/04/10 23:49:46 ERROR : Attempt 1/3 failed with 2 errors and: directory not found
2020/04/10 23:50:08 ERROR : : error reading source directory: directory not found
2020/04/10 23:50:08 INFO : Encrypted drive 'encrypted:/': Waiting for checks to finish
2020/04/10 23:50:08 INFO : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2020/04/10 23:50:08 ERROR : Encrypted drive 'encrypted:/': not deleting files as there were IO errors
2020/04/10 23:50:08 ERROR : Encrypted drive 'encrypted:/': not deleting directories as there were IO errors
2020/04/10 23:50:08 ERROR : Attempt 2/3 failed with 2 errors and: directory not found
2020/04/10 23:50:31 ERROR : : error reading source directory: directory not found
2020/04/10 23:50:31 INFO : Encrypted drive 'encrypted:/': Waiting for checks to finish
2020/04/10 23:50:31 INFO : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2020/04/10 23:50:31 ERROR : Encrypted drive 'encrypted:/': not deleting files as there were IO errors
2020/04/10 23:50:31 ERROR : Encrypted drive 'encrypted:/': not deleting directories as there were IO errors
2020/04/10 23:50:31 ERROR : Attempt 3/3 failed with 2 errors and: directory not found
2020/04/10 23:50:31 Failed to sync with 2 errors: last error was: directory not found

 

[Eric Collyer]

I'm having the same issue. Everything works fine on 11.3 U1, but on 11.3 U2 I get the same error you are experiencing when I try and cloud sync (yes, I'm using that as a verb) to B2.

 

[StoreMore]

Interesting, glad I am not the only one seeing this but not sure how to go about fixing it.

 

[blooper98]

I'm also having this issue after updating from 11.3 U1 to U2. My error logs are nearly identical to StoreMore's (at least, I couldn't find a difference besides the time stamps).

 

[ig12]

I'm facing the same issue.
Did any one of you find a Workaround or is it a confirmed bug?

 

[stualden]

Virtually the same issue here - traceback is identical, word for word, until the end, where I get

ValueError: rclone failed with exit code 1

rather than code 3. The original cloud sync (about 550G of data, just a day or two ago) was successful, but subsequent syncs have been giving me this error. I jumped from 9.10 all the way to 11.3 U2 last week so this was my first attempt at Cloud Sync.

 

[StoreMore]

I have resolved my issue. I unchecked the "Take Snapshot" option from the task.

 

[stualden]

FWIW, sadly, that option for me was not checked. If I check it, I get failure of the cloud sync within seconds. Unchecked, it runs the full course (a few hours), then fails.

 

[StoreMore]

Sorry to hear this didn't work for you. Here's all of my settings from the cloud sync task just as a reference for anyone that wants to compare.

Description: BackBlaze B2 Backup
Direction: PUSH
Credential: BackBlaze (B2)
Bucket: MYBUCKETNAME
Folder: /myfoldername
Upload Chunk Size: 96
Use-fast-list: CHECKED
Directory/Files: /mnt/Storage/MyFolder
Transfer Mode: SYNC
Take Snapshot: UNCHECKED (THIS RESOLVED MY ISSUE)
Prescript: BLANK
Post-script: BLANK
Remote Encryption: CHECKED
Filename Encryption: UNCHECKED
Encryption Password: MYPASSWORD
Encryption Salt: MYSALT
Transfers: 4
Follow Symlinks: UNCHECKED
Enabled: CHECKED
Exclude (This is how they exist in the Directory/Files path):
Folder/
Folder2/
Folder3/Folder4/

In my investigation for an answer to my issue the other things that I read that may cause an issue is having path names too long, symbolic links, and backing up jails.

 

[blooper98]

I have resolved my issue. I unchecked the "Take Snapshot" option from the task.

Thank you for posting this, it resolved my issue as well.

 

[stualden]

Thanks for the clue about long filenames - that turned out to be my issue. One file (which had been deleted, but was still in the .recycle folder, so it was a bit hard to find) had a ridiculously long name.

I'm still curious about the 'take snapshot" option - is there any documentation on that? What exactly does it take a snapshot of, and how long is that snapshot kept?

 

[StoreMore]

Thanks for the clue about long filenames - that turned out to be my issue. One file (which had been deleted, but was still in the .recycle folder, so it was a bit hard to find) had a ridiculously long name.

I'm still curious about the 'take snapshot" option - is there any documentation on that? What exactly does it take a snapshot of, and how long is that snapshot kept?

Glad you found the problem.

The official documentation I can find only states, "Take a snapshot of the dataset before a PUSH. (source)" It doesn't mention any other details. I am fairly new to FreeNAS myself and don't have any other knowledge of it other than the fact it seemed to work fine for me being checked in earlier versions of 11.3". My original thought of the way it worked was it would take the snapshop of the way the files were exactly at that time and then start the transfer to your cloud of choice based on this snapshot such that if you had changed a bunch files after this snapshot started it would only sync it based on this snapshot and not any subsequent files and finally remove the snapshot when done but I don't have any real evidence of this.

 

[katticus]

After hours and hours of troubleshooting, In order to get rid of that exact error I had to exclude my Desktop folder from the sync, everything sync'd successfully after that. I believe it has something to do with either the shortcuts, recycle bin, symlinks or file name length, I could be wrong.

Cloud Tasks > OneDrive > Exclude > /Desktop

 

[KazuyaDarklight]

Had the same problem, so far, it seems to have been caused by old 11.2/3 records in our IOCAGE. Put the IOCAGE on the exceptions list and its no longer almost instantly failing. We've now also deleted those records. If the current upload is successful, we'll run a second with the exception removed.