Onedrive Personal CloudSync?

[xkpx]

How you bypass

Code:

admin@Aurora2-truenas[~/onedrive-sdk-python]$ sudo cp -ar src/onedrivesdk /usr/lib/python3/dist-packages
cp: cannot create directory '/usr/lib/python3/dist-packages/onedrivesdk': Read-only file system

sudo zfs set readonly=on boot-pool/ROOT/24.04-BETA.1/usr

But other error occurs on latest beta TrueNAS-SCALE-24.04-BETA.1, you can view in video here

Watch 2024-02-29 09-26-32 | Streamable

Watch "2024-02-29 09-26-32" on Streamable.

streamable.com

 

[Aipoc76]

How you bypass

Code:

admin@Aurora2-truenas[~/onedrive-sdk-python]$ sudo cp -ar src/onedrivesdk /usr/lib/python3/dist-packages
cp: cannot create directory '/usr/lib/python3/dist-packages/onedrivesdk': Read-only file system

couldnt you just chmod 777 /usr/lib/python3/dist-packages/onedrivesdk
rm -r /usr/lib/python3/dist-packages/onedrivesdk

the do the setup

 

[solit]

I'm not too big on modifying stuff on the OS and the previous posts did not give me confidence in making an exception now :)
(running TrueNAS-SCALE-24.04-BETA.1 here)
So I went and installed this as a custom app: https://hub.docker.com/r/driveone/onedrive
It seems to work well enough for my purposes, although I haven't tested it for long. I did however test how well it would play with nextcloud sync on the same folder and that worked surprisingly flawlessly.

Some things to note:

• It's a bit tricky to get it to authenticate with OneDrive, but after that's done, at least there are no secrets on the server which is nice.
  1. Start the container
  2. Go to logs, copy auth link (this failed a couple of times for me, b/c the container had crashed before I could see those)
  3. Go to browser and authenticate OneDrive use
  4. Set an ENV variable and put the auth response there
• It will (perhaps only on the first run?) try to chown all things that are mounted to be the OneDrive folder. For this the image expects an ENV variable (GUI, UID) of the target user. Not the greatest behaviour, but I made it work for me.
• If you don't have any other options set, it will immediately start syncing (in both directions) once setup is successful, so plan accordingly or prepare settings in advance.

Definitely have a read over the docs, it explains all things quite well imo: https://github.com/abraunegg/onedrive/blob/master/docs/Docker.md

Here's an example of my ENV variables to get OneDrive only as push-style backup location for my office files:

• ONEDRIVE_UID=33 (this is www-data)
• ONEDRIVE_GID=33
• ONEDRIVE_UPLOADONLY=1
• ONEDRIVE_NOREMOTEDELETE=1

And this I removed again after auth was successful:

• ONEDRIVE_AUTHRESPONSE=[...]

 

[Black_Duck]

I've tried a different approach with limited success. I installed Rclone as a container and followed the Rclone documentation to enable the GUI front end. After a lot of messing around with the application settings I finally managed to get the container up and logged in via the gui.
Unfortunately, this led to a new set of problems:

• When invoking a new 'config' through the container the authorisation fails as it invokes a browser at "127.0.0.1:53682", which fails as there is no browser. Rclone documentation around running rclone in a headless machine has some suggestions around this. The simplest is to run rclone on another machine and copy the Access token over. Can't see a way around this.... You really need Rclone to do this through HTML back to your browser....
• Once you have the config, "exploring" and "Mounting" the drive is pretty straight forward. The downs side is:
  • I can't see how to make the mount visible outside the container. Mounting to a Host Path volume doesn't work. This post suggests there is a way around this, but I haven't figured it out yet.
  • Mounts do not persist over a restart of the app. Effectively you have to go back into the interface and remount Onedrive.

I was hoping to have a persistent mount so that I could possibly use Truenas cloudsync to do my copying....

 

[theonexx]

It should be possible to download rclone to a dataset specifically set up for this (to preserve the whole thing during updates) and script the backup itself via a cron task.

Code:

/mnt/Tank/rclone/rclone sync /mnt/Tank/dataset_to_backup remote:/path/to/remote/folder --create-empty-src-dirs --metadata

You'd have to set up the OneDrive endpoint manually, but this is so easy that even I have managed to do it... cd to the rclone dataset, type

Code:

./rclone config

and follow the instructions; full docs here: https://rclone.org/onedrive/

P.S.: I believe rclone will save its config file to /root/.config/rclone/rclone.conf - it won't survive a reboot or update; you'd have to keep an extra copy of this file and have it copied back to the above location during boot.

P.S.2: Even better, set an environment variable:

Code:

RCLONE_CONFIG=/mnt/Tank/rclone/rclone.conf

Hey,

I am a noob who's migrating from QNAP/QTS system to TrueNAS in general. That is when I was hit by the fact that Cobia does not support OneDrive Sync.

Syncing with OneDrive is a critical feature in my NAS storage where I do PULL sync from one drive to keep a read-only local copy of my Personal OneDrive data.

I was going through the forms to find a workaround, that I came across this thread and couple other threads suggesting the rclone instance workaround.

so what I have done so far is as follows

1. Download, installed and authenticated Rclone app with my Personal OneDrive through the Shell \\This was a nightmare for a linux systems noob like me but I managed to get done.
2. Created "ONEDRIVE_CACHE" dataset and I have made the share type to be SMB. //the reason I have change the share type to be SMB is allow admin user to have the permissions to create data under this dataset
3. I have done a Rclone VFS cache mount with mount point path to "ONEDRIVE_CACHE" dataset
4. have verified that the data has been transferred to the mount point directory by ls command. Now I am stuck at this step, where TrueNAS can't access this dataset (as shown in the screenshots attached). Hence I can't see any information/meta data related to this dataset. Even TrueNAS does not recognize that the storage is full.

I feel that I have messed up the chmod permissions on the dataset, hence the TrueNAS is not able to access it.

I would be grateful for any help to get this up and running.

 

[M3PH]

So I just tried this on SCALE 23.10.2 and it completely screwed up the middleware. I had to reinstall truenas to fix it. I was lucky that i still had shell access over IPMI and was able to copy /data to a usb stick (my config backup was on share I couldn't access. It's now on a usb stick). I think my issue was becasue most of the commands required me to use sudo to get them to execute. My advice would be to not use this solution if you are running 23.10.2 or newer and instead look into rclone (which is what i'm going to do after i get over the panic attack i'm having)

 

[Aipoc76]

So I just tried this on SCALE 23.10.2 and it completely screwed up the middleware. I had to reinstall truenas to fix it. I was lucky that i still had shell access over IPMI and was able to copy /data to a usb stick (my config backup was on share I couldn't access. It's now on a usb stick). I think my issue was becasue most of the commands required me to use sudo to get them to execute. My advice would be to not use this solution if you are running 23.10.2 or newer and instead look into rclone (which is what i'm going to do after i get over the panic attack i'm having)

See my Post SCALE 23.10.2 works fine.

Onedrive Personal CloudSync?

yes

www.truenas.com

 

[M3PH]

See my Post SCALE 23.10.2 works fine.

Onedrive Personal CloudSync?

yes

www.truenas.com

i followed the advice in your post and it didn't help at all.

 

[Aipoc76]

Sorry it didnt work for you.

 

[Cellobita]

So I just tried this on SCALE 23.10.2 and it completely screwed up the middleware. I had to reinstall truenas to fix it. I was lucky that i still had shell access over IPMI and was able to copy /data to a usb stick (my config backup was on share I couldn't access. It's now on a usb stick). I think my issue was becasue most of the commands required me to use sudo to get them to execute. My advice would be to not use this solution if you are running 23.10.2 or newer and instead look into rclone (which is what i'm going to do after i get over the panic attack i'm having)

I have been using a bash script with a standalone copy of rclone on its own dataset - it has been working perfectly over the past few months and is untouched when upgrading SCALE.

Create the dataset (Tank/Rclone, in my case), then from the shell, as root (use sudo -i) switch to it, and download and extract the latest version of rclone with

Code:

wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip -j rclone-current-linux-amd64.zip

Create the config file rclone.conf on your own workstation, copy it to the dataset, then invoke rclone with a script like this:

Code:

#!/bin/sh
/mnt/Tank/Rclone/rclone sync /mnt/Tank/[Share] [OneDriveEndpoint]: \
 --config=/mnt/Tank/Rclone/rclone.conf \
 --create-empty-src-dirs \
 --ignore-checksum \
 --ignore-size \
 --metadata \
 --stats 0 \
 --delete-excluded \
 --human-readable \
 --log-level NOTICE \
 --log-file /mnt/Tank/Rclone/report.txt

 

[M3PH]

I have been using a bash script with a standalone copy of rclone on its own dataset - it has been working perfectly over the past few months and is untouched when upgrading SCALE.

Create the dataset (Tank/Rclone, in my case), then from the shell, as root (use sudo -i) switch to it, and download and extract the latest version of rclone with

Code:

wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip -j rclone-current-linux-amd64.zip

Create the config file rclone.conf on your own workstation, copy it to the dataset, then invoke rclone with a script like this:

Code:

#!/bin/sh
/mnt/Tank/Rclone/rclone sync /mnt/Tank/[Share] [OneDriveEndpoint]: \
 --config=/mnt/Tank/Rclone/rclone.conf \
 --create-empty-src-dirs \
 --ignore-checksum \
 --ignore-size \
 --metadata \
 --stats 0 \
 --delete-excluded \
 --human-readable \
 --log-level NOTICE \
 --log-file /mnt/Tank/Rclone/report.txt

thanks. I got rclone installed and cron task setup before I went to bed last night but the cron task doesn't work, which is to be expected as it was 5am, I was exhausted and I had been up for 20 hours. I'm having a day off today but will look at it again in the week. Your post will be very useful

 

[sfatula]

I'm not too big on modifying stuff on the OS and the previous posts did not give me confidence in making an exception now :)
(running TrueNAS-SCALE-24.04-BETA.1 here)
So I went and installed this as a custom app: https://hub.docker.com/r/driveone/onedrive
It seems to work well enough for my purposes, although I haven't tested it for long. I did however test how well it would play with nextcloud sync on the same folder and that worked surprisingly flawlessly.

Some things to note:

• It's a bit tricky to get it to authenticate with OneDrive, but after that's done, at least there are no secrets on the server which is nice.
  1. Start the container
  2. Go to logs, copy auth link (this failed a couple of times for me, b/c the container had crashed before I could see those)
  3. Go to browser and authenticate OneDrive use
  4. Set an ENV variable and put the auth response there
• It will (perhaps only on the first run?) try to chown all things that are mounted to be the OneDrive folder. For this the image expects an ENV variable (GUI, UID) of the target user. Not the greatest behaviour, but I made it work for me.
• If you don't have any other options set, it will immediately start syncing (in both directions) once setup is successful, so plan accordingly or prepare settings in advance.

Definitely have a read over the docs, it explains all things quite well imo: https://github.com/abraunegg/onedrive/blob/master/docs/Docker.md

Here's an example of my ENV variables to get OneDrive only as push-style backup location for my office files:

• ONEDRIVE_UID=33 (this is www-data)
• ONEDRIVE_GID=33
• ONEDRIVE_UPLOADONLY=1
• ONEDRIVE_NOREMOTEDELETE=1

And this I removed again after auth was successful:

• ONEDRIVE_AUTHRESPONSE=[...]

This is the solution IMHO. No mods to the OS, and, it should simply work once setup correctly, as long as they maintain that container which likely they will.

 

[M3PH]

I have been using a bash script with a standalone copy of rclone on its own dataset - it has been working perfectly over the past few months and is untouched when upgrading SCALE.

Create the dataset (Tank/Rclone, in my case), then from the shell, as root (use sudo -i) switch to it, and download and extract the latest version of rclone with

Code:

wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip -j rclone-current-linux-amd64.zip

Create the config file rclone.conf on your own workstation, copy it to the dataset, then invoke rclone with a script like this:

Code:

#!/bin/sh
/mnt/Tank/Rclone/rclone sync /mnt/Tank/[Share] [OneDriveEndpoint]: \
 --config=/mnt/Tank/Rclone/rclone.conf \
 --create-empty-src-dirs \
 --ignore-checksum \
 --ignore-size \
 --metadata \
 --stats 0 \
 --delete-excluded \
 --human-readable \
 --log-level NOTICE \
 --log-file /mnt/Tank/Rclone/report.txt

I'm just looking at this over some beer and I want to check that it's one entry like that for each share I want to transfer, correct?

and where the hell are the log files in scale!!!!!! i can not find them

 

[gt2416]

After a update mine failed also. I then Deleted

Code:

/usr/lib/python3/dist-packages/onedrivesdk

and re-ran the following steps provided by Kiva.. Although my browser or this forum did truncate his link. Hopefully this stay fully normal.

Code:

git clone https://github.com/imkiva/onedrive-sdk-python
cd onedrive-sdk-python/
cp -ar src/onedrivesdk /usr/lib/python3/dist-packages
cp -ar src/python3/request /usr/lib/python3/dist-packages/onedrivesdk

cd /usr/lib/python3/dist-packages/middlewared/rclone/remote/
wget https://raw.githubusercontent.com/truenas/middleware/cd9acacac17f9a4e98883c7d8baefacd2085eea7/src/middlewared/middlewared/rclone/remote/onedrive.py
sed -i '1d;$d' onedrive.py

Rebooted and all is back to normal.

EDIT: I did have to recreate my tasks..

View attachment 76036

View attachment 76035

I think its crap that iXsystems wont just fix this for the community. Its not like Microsoft is some sort of small potato's...

Do you happen to know where the config files for cloud sync tasks are saved?
Perhaps before upgrading to the new version I can save my config files and restore them.
I have over 80 cloudsync tasks for Sharepoint. Re creating all of them would be a nightmare.
Sadly a lot of my apps are broken and cant update, I need to upgrade Truenas Scale version to fix them, really don't want to recreate all my cloud tasks.

 

[andyjay777]

Any thoughts from iXSystems on this? We've got people:
- modifying the middleware
- installing rclone and using scripts
- running docker containers
Lots of command line activity for something that worked simply from the GUI.
Will this functionality be restored in Scale 24?

 

[Cellobita]

I'm just looking at this over some beer and I want to check that it's one entry like that for each share I want to transfer, correct?

and where the hell are the log files in scale!!!!!! i can not find them

Depends. I have some customers with a single pool, using a single rclone command for all their datasets/shares - in this case what I do is exclude iXsystem's and my own utility datasets from the copy, with the --filter-from option in rclone. My filter-from.txt is:

Code:

- /Config/**
- /Rclone/**
- /ix-applications/**
+ *

This will copy everything under /mnt/Tank with a single rclone command,

Code:

/mnt/Tank/Rclone/rclone sync /mnt/Tank [OneDriveEndpoint]: \
 --config=/mnt/Tank/Rclone/rclone.conf \
 --create-empty-src-dirs \
 --ignore-checksum \
 --ignore-size \
 --metadata \
 --stats 0 \
 --filter-from /mnt/Tank/Rclone/filter-from.txt
 --delete-excluded \
 --human-readable \
 --log-level NOTICE \
 --log-file /mnt/Tank/Rclone/report.txt

except the Config, Rclone and ix-applications datasets.

 

[ragametal]

Any thoughts from iXSystems on this? We've got people:
- modifying the middleware
- installing rclone and using scripts
- running docker containers
Lots of command line activity for something that worked simply from the GUI.
Will this functionality be restored in Scale 24?

I have been trying to get a response from them for months.

I do hope a fix for this gets implemented in Scale 24 but their silence implies this is not on their list of priorities at the moment.

 

[Bainnor]

I have been trying to get a response from them for months.

I do hope a fix for this gets implemented in Scale 24 but their silence implies this is not on their list of priorities at the moment.

Considering it's an 8 year old module someone else wrote, I'm not surprised it's pretty low on their priorities. Perhaps if Microsoft would maintain their repository and merge updates, it would be easier to justify including it.

 

[Stephan4711]

@eturgeon can you confirm delopers are aware of this situation?
THe Workaround on page 2 works perfectly, it would be much easier, it would be integrated back to the official code, that there is no fiddling around the OS. As you see many people want this OneDrive Sync back

 

[Bainnor]

The link in post 3 of this thread goes to a post by morganL, where he links this, where they acknowledge the contribution, but point out that the last pull request Microsoft merged was 7 years ago, and that Python 3.12 has already been released, which may break compatibility again. They also comment on desiring to keep up to date with future Python versions, which would only increase the likelihood that it will break again since no one upstream is maintaining the repo.