TrueNAS-SCALE-23.10.0.1 - cloudSync with Microsoft Onedrive.

[ragametal]

I upgraded from Bluefin to Cobia and I noticed that "Microsoft Onedrive" is missing from the list of providers in CLOUD CREDENTIALS.
Because of this, the CloudSync task i had (to backup my Onedrive to my local TrueNAS) fails after the upgrade to Cobia.

According to this post https://www.truenas.com/community/threads/return-to-truenas-scale-bluefin.112019/#post-775688
"OneDrive had to be removed because driver was not supported with current python".

OK, that explains what happened but not necessarily how to get the service we once had back.

My understanding is that CloudSync uses Rclone internaly.
I'm more than willing to try an App to replace this functionality but i was unable to find an rclone app in either the official TrueNAS catalog or the TrueCharts catalog.

The user @Kiva suggested a solution to bring it back https://www.truenas.com/community/threads/onedrive-personal-cloudsync.112559/page-2#post-788917
but i don't know how safe it would be to implement something like this. It almost feels like hacking into the trueNAS middleware, which i Honestly would prefer not to do.

He also seems to have proposed his solution at https://github.com/truenas/middleware/pull/11143 but he was shutdown.

Based on this, have any of you know how sync from onedrive to trueNAS Cobia?

 

[skadlec72]

I have been using this software. At the moment I run it on a windows VM that I have running on Truenas. He has release a version for TrueNas but I have not tried it yet.

Click the link below and check it out. It can sync to a ton of cloud services. I even use it to connect to my 3d printer and Ubiquiti/Unifi Cameras to pull video.

Syncovery

 

[Eldeng20]

Hi Ragametal,
Did you end up finding a good solution to this issue?
Amazing how there are no disclaimers for this kind of thing when people discuss whether you should upgrade or not.

 

[null-sparkle]

What about running https://github.com/abraunegg/onedrive via apps and just mounting the directory to sync or something?
I'm actually using a similar approach to sync a folder into a PVC in my kubernetes cluster.

Not quite as straightforward as configuring cloud sync though.

 

[ragametal]

Hi Ragametal,
Did you end up finding a good solution to this issue?
Amazing how there are no disclaimers for this kind of thing when people discuss whether you should upgrade or not.

Short answer:
Yes, i ended up using Rclone via the CLI.

Long answer:
I though about using the Onedrive sync client as suggested by @null-sparkle and some others but i decided to try Rclone because if i learned how to use it properly, i may be able to use it on other systems besides TrueNAS.

So, I tried installing an Rclone container but i hit a wall trying to enable the web GUI for it.
Then somebody suggested to download Rclone directly to TrueNAS and use that. I honestly didn't like that approach as i don't like to modify the base system.

That's when i realized that i could just use the Rclone instance already installed in TrueNAS (as it is used for the "Cloud Sync" functions). So, i used the CLI to create an Rclone Config file for my remote Onedrive/Sharepoint and made made sure to save that file into a dedicated dataset. https://rclone.org/onedrive/. Then, i used a daily cron job to perform the Rclone sync specifying that config file.

It works like a charm. I'm very happy that i decided to go this route as Rclone has many more providers and options than what TrueNAS makes available trhought their GUI.

I would still prefer to use the GUI if possible as I'm missing the visual feedback if the sync failed, but aparently that will not be possible with onedrive.

What i still don't understand is that, suposedly the Rclone onedrive driver is not supported by the python version in TrueNAS and that is the reason why it was eliminated from the "Cloud sync". However, the Rclone instance installed in TrueNAS works flawlessly on that same system if used via the CLI and using the same Python version as the GUI.

I have asked multiple times for input on this but nobody seems to know, or care about the market sector affected by this.

 

[Eldeng20]

Amazing, thank you. I'll give this a shot!

 

[theonexx]

Short answer:
Yes, i ended up using Rclone via the CLI.

Long answer:
I though about using the Onedrive sync client as suggested by @null-sparkle and some others but i decided to try Rclone because if i learned how to use it properly, i may be able to use it on other systems besides TrueNAS.

So, I tried installing an Rclone container but i hit a wall trying to enable the web GUI for it.
Then somebody suggested to download Rclone directly to TrueNAS and use that. I honestly didn't like that approach as i don't like to modify the base system.

That's when i realized that i could just use the Rclone instance already installed in TrueNAS (as it is used for the "Cloud Sync" functions). So, i used the CLI to create an Rclone Config file for my remote Onedrive/Sharepoint and made made sure to save that file into a dedicated dataset. https://rclone.org/onedrive/. Then, i used a daily cron job to perform the Rclone sync specifying that config file.

It works like a charm. I'm very happy that i decided to go this route as Rclone has many more providers and options than what TrueNAS makes available trhought their GUI.

I would still prefer to use the GUI if possible as I'm missing the visual feedback if the sync failed, but aparently that will not be possible with onedrive.

What i still don't understand is that, suposedly the Rclone onedrive driver is not supported by the python version in TrueNAS and that is the reason why it was eliminated from the "Cloud sync". However, the Rclone instance installed in TrueNAS works flawlessly on that same system if used via the CLI and using the same Python version as the GUI.

I have asked multiple times for input on this but nobody seems to know, or care about the market sector affected by this.

Hey. I am still new to TrueNAS and I am currently facing the same challenge. Can you please elaborate more on how you have done Rclone instance? if you could share screenshots or video I would be grateful for that.

 

[ragametal]

Hey. I am still new to TrueNAS and I am currently facing the same challenge. Can you please elaborate more on how you have done Rclone instance? if you could share screenshots or video I would be grateful for that.

I will not be able to provide a "guide" about this for some time (I'm swamped at work).
But, if you can access the terminal of your truenas system you can just follow the instructions provided by RCLONE https://rclone.org/onedrive/

That is mainly what i used.

Good luck and sorry i cannot be of further help at the moment.

 

[theonexx]

I will not be able to provide a "guide" about this for some time (I'm swamped at work).
But, if you can access the terminal of your truenas system you can just follow the instructions provided by RCLONE https://rclone.org/onedrive/

That is mainly what i used.

Good luck and sorry i cannot be of further help at the moment.

Thank you for the feedback.

Actually since my last post, I tried to figure it out on my own but unfortunately I have hit a road-block.

so what I have done so far is as follows

1. Created "ONEDRIVE_CACHE" dataset and I have made the share type to be SMB. //the reason I have change the share type to be SMB is allow admin user to have the permissions to create data under this dataset
2. Download, installed and authenticated Rclone app with my Personal OneDrive through the Shell
3. I have done a Rclone VFS cache mount with mount point path to "ONEDRIVE_CACHE" dataset
4. have verified that the data has been transferred to the mount point directory by ls command. Now I am stuck at this step, where TrueNAS can't access this dataset (as shown in the screenshots attached). Hence I can't see any information/meta data related to this dataset. Even TrueNAS does not recognize that the storage is full.

I feel that I have messed up the chmod permissions on the dataset, hence the TrueNAS is not able to access it.

 

[ragametal]

@theonexx i would agree that it seems you have a permissions problem.
In my case, the local dataset is owned by the user "admin" which is also the user running the rclone script.
That dataset is also owned by the group to which the end user belongs to. That way the end user can always access the content of the folder.

Now, i don't want to assume anything but the wording on your post seems to imply that you created an SMB share so the end user can create files in the "Target" dataset.

If that is your intent i believe this will not work. Rclone typically makes a one-way sync, it is kind of a replication from source to target. Either from the cloud to your local dataset or from the dataset to the cloud.

I believe there is another mode that kinds of does what you want by "mounting" the cloud storage as a local dataset. But if you do that the data will only be on the cloud. Therefore, you will not have a local backup for it.

 

[theonexx]

@theonexx i would agree that it seems you have a permissions problem.
In my case, the local dataset is owned by the user "admin" which is also the user running the rclone script.
That dataset is also owned by the group to which the end user belongs to. That way the end user can always access the content of the folder.

Now, i don't want to assume anything but the wording on your post seems to imply that you created an SMB share so the end user can create files in the "Target" dataset.

If that is your intent i believe this will not work. Rclone typically makes a one-way sync, it is kind of a replication from source to target. Either from the cloud to your local dataset or from the dataset to the cloud.

I believe there is another mode that kinds of does what you want by "mounting" the cloud storage as a local dataset. But if you do that the data will only be on the cloud. Therefore, you will not have a local backup for it.

@ragametal first of thank you for taking the time to respond to my post despite your busy schedule.

My objective is to only PULL data from my personal OneDrive and to keep a local copy on my NAS. Also every time the data is changed on the cloud I want the local copy to be synced automatically to be up to date with the cloud copy.

With regards to the SMB share, I have created it in order to give the default "admin" user the privilege to mount data though rclone. since when I used the following command

Code:

rclone mount OneDrive: /mnt/DP1/ONEDRIVE_CACHE --vfs-cache-mode writes --allow-non-empty

it give error that I don't have the rights to do that. When I looked at the dataset when I have initially created it, the owner was the root users.

To be honest I feel there could be an easier way of giving the default "admin" user the permission to mount, but given I am new to TrueNAS and this whole ACL/Rights thing I might not be aware of it.

 

[ragametal]

@theonexx If what you want to do is to create local backups from the cloud (aka "source") to a local dataset (aka "destination") then i think you are using the wrong command. You should use rclone sync source:path dest:path [flags].

If you do it that way the "admin" user should be able to access the dataset directly without the need for a SMB share. At least that is how i do it at home. I'm not in front of my system but i believe my script uses a command similar to:
rclone sync source:path dest:path --config=PATH-TO-CONFIG-FILE --delete-after --create-empty-src-dirs --ignore-checksum --ignore-size --metadata --stats 0 --log-level NOTICE --log-file PATH-TO-LOG-FILE

The downside to this "Sync" is that is not realtime. You will have to invoke this command every-time you want to update your destination folder with the contents of your source.

This approach is great for me as i do a backup once a day, but if what you want is realtime sync then you will need a diferent solution. I think you should use the onedrive client for linux suggested on this thread.

Something like that would do two-way sync and should be real-time as well.

 

[theonexx]

@ragametal thank you for your feedback. I have finally done based on your guidance. I really appreciate your support on this.

If I want to update the local copy, shall I use the same command ? Also was there any automatic (scheduled) way for running this update.

 

[ragametal]

@theonexx, 'm glad to hear you got it running. To update the local copy (destination), you just need to run the command again. Rclone will then compare the contents on both sides and transfer only the files that has changed since the last sync.

To automate the process you would need to setup a cronjob in truenas to run this command at the frequency of your choosing. In my case it is once a day (at about 2 am), but you may need to more often than that or more relaxed (weekly or monthly maybe?).

 

[stk]

Simplest solution is to download a private rclone to a dataset and using the GUI to create a cron job to run it.

Details here.