Cloud Sync task - add more rclone flags or manual input box to GUI

[Solway]

Can we get more flags for the rclone sync added to the GUI please....?
or a text box where flags can be added manually.
A hint of rclone version might also be helpful to make sure flag works with it.

For example:

I have come across a issue with syncing to sharepoint/onedrive, where microsoft for some reason adds meta data to the file causing the checks to fail.
Rclone solution (although not ideal) is to use a --ignore-checksum --ignore-size flag. There are also throttling issues, so a flag of --user-agent "ISV|rclone.org|rclone/v1.55.1" is helpful.

link to doc here https://rclone.org/onedrive/#unexpected-file-size-hash-differences-on-sharepoint

link to microsoft adding meta data "bug" https://github.com/OneDrive/onedrive-api-docs/issues/935#issuecomment-441741631

Currently syncing to sharepoint cant be used on truenas.

 

[jose-pr]

I would love this and setting environment variables.

 

[mouuro]

I was having some problems that could be easily solved by setting some rclone parameters and I found that this is possible by using the API to update args field of cloudsync tasks.
It doesn't show up in the GUI after setting it but it seems to stay as was set even when changing other settings from the GUI later on.

An example using curl (replace all example fields in <>):
1. Find the relevant <CLOUD_SYNC_TASK_ID>:

Code:

curl -X GET -H "Authorization: Bearer <API_KEY>" http://<TRUENAS_ADDR>/api/v2.0/cloudsync

2. Update the args field:

Code:

curl -X PUT -H "Authorization: Bearer <API_KEY>" -H "Content-Type: application/json" -d '{"args":"<RCLONE_FLAGS>"}' http://<TRUENAS_ADDR>/api/v2.0/cloudsync/id/<CLOUD_SYNC_TASK_ID>

<RCLONE_FLAGS> have to be properly JSON encoded but for simple cases it's just normal text, example of the whole JSON:

Code:

'{"args":"--b2-hard-delete --track-renames --track-renames-strategy=leaf"}'

Edit: Removed misleading details, and made clearer what exactly works

 

[Paulson]

No luck with the API method. The update is happening, but command not recognized.

Code:

 "encryption_salt": "",
 "args": "RCLONE_DELETE_EMPTY_SRC_DIRS=true",
 "post_script": "",
 "pre_script": "",

result:

Code:

 "error": "[EFAULT] Error: unknown command \"RCLONE_DELETE_EMPTY_SRC_DIRS=true\" for \"rclone\"\nRun 'rclone --help' for usage.\nYou could use 'rclone selfupdate' to get latest features.\n\n2024/02/14 12:00:30 Fatal error: unknown command \"RCLONE_DELETE_EMPTY_SRC_DIRS=true\" for \"rclone\"\n"

 

[Paulson]

Adding RCLONE_DELETE_EMPTY_SRC_DIRS=true
to Cloud Sync Task Edit: Pre-script did not work, empty dirs remain. MOVE context.

 

[mouuro]

Sorry for the confusion the args are just normal flags and it has nothing to do with previously mentioned environment variables, I'm going to edit my previous post again to try and make it more clear. So in your case you should just add the normal --delete-empty-src-dirs flag in the args field

 

[Paulson]

Using the API and args -

verification it landed:

Code:

 "encryption_salt": "",
 "args": "--delete-empty-src-dirs",
 "post_script": "",
 "pre_script": "",

but doesn't work live:

Code:

[EFAULT] Error: unknown command "remote:/Archive2Nas" for "rclone"
Run 'rclone --help' for usage.

Looks like the command string is getting mangled. Archive2Nas is the src dir for PULL/MOVE.

 

[mouuro]

I see in your footer that you run CORE where I'm doing this on SCALE, if that's the case it might just be that those arguments are handled differently. Unfortunately I don't have a CORE deployment at hand to test.

But here are some relevant details that helped me to track this down, In the file /usr/lib/python3/dist-packages/middlewared/plugins/cloud_sync.py from line 204 inside rclone function there is:

Code:

args += shlex.split(cloud_sync["args"])

args += [cloud_sync["transfer_mode"].lower()]

which adds those additional arguments right before the transfer mode of sync, copy or move. I've also tested with just one flag in case there was some edge case as I was using multiple but it still works for me.

My current system is TrueNAS-SCALE-22.12.4.2 hope that helps track it down.