Cloudsync [rclone] errors on remote server side with Microsoft Onedrive

[soko]

Hi,

I suspect this is an rclone bug with Onedrive, or possibly a Onedrive limitation.

Setup:

TrueNAS version = TrueNAS-12.0-U4
rclone version =
rclone v1.53.1-DEV
- os/arch: freebsd/amd64
- go version: go1.16.4

Using Onedrive personal 1TB account.

Cloudsync task is setup with parameters as follows:
- Remote encrytion on
- Filename encryption on
Encryption password and salt setup accordingly.

Job is set to "COPY" as opposed to "SYNC"

Runs daily on cron.

Symptoms:

- Cloudsync task runs first time as expected shows "SUCCESS", rclone authentication details work as expected.
- Upon subsequent updates, the job fails, some files are transferred.
I have attached log snippet with redacted portions:

Code:

2021/06/30 05:00:03 INFO  :
Transferred:                0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.1s

2021/06/30 05:01:00 NOTICE: Scheduled bandwidth change. Bandwidth limits disabled
2021/06/30 05:01:03 INFO  : X/IMG_20210628_181137.jpg: Copied (new)
2021/06/30 05:01:04 INFO  : X/IMG_20210628_183501.jpg: Copied (new)
2021/06/30 05:01:04 INFO  : X/IMG_20210628_182912.jpg: Copied (new)
2021/06/30 05:01:04 INFO  : X/IMG_20210628_180443.jpg: Copied (new)
2021/06/30 05:01:05 INFO  : X/IMG_20210628_183834.jpg: Copied (new)
2021/06/30 05:01:07 INFO  : X/IMG_20210628_184522.jpg: Copied (new)
2021/06/30 05:01:07 INFO  : X/IMG_20210628_184738.jpg: Copied (new)
2021/06/30 05:01:07 INFO  : X/IMG_20210628_184910.jpg: Copied (new)
2021/06/30 05:01:31 ERROR : X/2019_02_04: error reading destination directory: couldn't list files: Get "https://graph.microsoft.com/v1.0/drives/xxxxxxxxxxxx/items/1EB39BB4D7A18FEC!123542/children?$top=1000": dial tcp: lookup graph.microsoft.com on A.B.C.D:53: server misbehaving

This last error repeats hundreds of times, also impacting other backup jobs determining that final cloudsync status always "FAILED"

My theory, there could be too many files in the directory, causing a timeout. Is there a way to extend rclone wait time?

Also found this regarding rclone/onedrive
https://github.com/rclone/rclone/issues/2011
https://forum.rclone.org/t/error-reading-destination-directory/7316
These issues are all dated 2018 thereabouts. It seems the next step is raising this in the rclone forums.

Wanted to check if any other person seeing this with CloudSync/rclone->Onedrive or other cloud storages.

Question: I cannot figure out how to enable additional debug via the GUI for rclone, ie. rclone -vv option. The other approach is to run from command line.

Thanks
C

 

[originalprime]

I'm seeing the same error. Took me a while to find the logs. They're located in /tmp/middlewared/jobs. Running TrueNAS core 12.0-U4. I'm following the links you provided now. Hopefully we can get to the bottom of this!

 

[ChrisRJ]

My theory, there could be too many files in the directory, causing a timeout.

How many files are we talking about?

 

[originalprime]

@ChrisRJ I'm doing a sync with over 100k files, but they're broken up into dozens of subdirectories. Over 99k files transferred successfully, then the sync failed. And it keeps failing on the same files. Weird.

 

[originalprime]

A little more searching turned up some more info, but it doesn't seem to be exactly what we're looking for. This page indicates that there could be a number-of-files limitation with OneDrive. It references this page from Microsoft. They seem to be conflicting.

The Rclone documentation thinks that 100k files is an issue, but the Microsoft link says "...If you have more than 100,000 files, sync can take a long time." It does not say that 100k files is a hard limit. Either I've hit another limitation like file name length (which is something I've looked at on my end. It doesn't seem to be the issue) or some other limitation I haven't figured out yet.