Cloud Credentials/Cloud Sync - Google Drive

[LostFate]

Hey all,

I was wondering if we could get more information on how to configure Cloud Credentials for Google Drive? I know it's new functionality so the documentation is sparse. I'm sure I'm missing something really silly or making everything overly complicated but I've tried the following to get this to work:

• https://rclone.org/drive/ - This actually works but isn't configurable through the GUI so isn't a great long term solution for what we are looking for. Also, I'm not entirely sure if the Cloud Sync Tasks are just cron jobs or something more. I'm hesitant to poke around too much without any guidance, lest I nuke the clients data.
• https://help.talend.com/reader/Ovc10QFckCdvYbzxTECexA/EoAKa_oFqZFXH0aE0wNbHQ - We tried this to generate the access token because there doesn't appear to be any documentation on this elsewhere. These tokens expire and the GUI doesn't ask for a refresh_token. That's moot anyway because we get "undefined" errors when trying to use these access tokens.
• We also tried copying the entire token line from the config generated by the rclone directions and pasted it into the GUI to see if it'd take but unfortunately that didn't work either.

We recently bought a FreeNAS Mini for a client to store their bulk files on and they want to be able to sync their files from Google Drive down to the NAS to decrease the amount of bandwidth required to use the dataset for all staff at the client site. The dataset is like 3.5TB so avoiding local sync to all users workstations is a priority, it'd be nice to have one copy on the NAS.

Additional Information (sorry):
FreeNAS Mini
32GB RAM
12TB usable disk space (4x 6TB drives in RAID10)
L2ARC
ZIL
FreeNAS 11.2-Beta2

 

[LostFate]

Bump... If possible, we'd like to start the sync today so it has the weekend to pull down the bulk of the data. If you can even point me in the direction of something I haven't tried yet, I'd appreciate it.

 

[dlavigne]

Were you able to figure this out? If not, does this comment help?

 

[LostFate]

It did! Thanks a ton! I tried copying the line but I must have left out the brackets. We now have it syncing down for the client.

 

[hervon]

Were you able to figure this out? If not, does this comment help?

I'm trying to configure Cloud Credentials for Google Drive as well. Sadly, i'm not authorized to see that comment ... that may help.

 

[dlavigne]

You should be able to see it now.

 

[hervon]

Works!

 

[hervon]

Sadly, it only works a short while. This morning I had to manually renew the token. Any idea how to make it more permanent ? Maybe I do something wrong. Here is the structure of my token FYI:
{
"access_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"scope": "https://www.googleapis.com/auth/drive",
"token_type": "Bearer",
"expires_in": 3600,
"refresh_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

I'm on beta 3 BTW. ;-)

 

[hervon]

The connection works for a short while then the authorization expires. As suggested by VladimirI modified "expires_in": to 0 and 1 but the issue still remains. Any idea ?

 

[hervon]

My bad. I was not using rclone at all. I was working on this site : https://developers.google.com/oauthplayground/
Now I retried using rclone integrated in FreeNAS and it seems to work. Now the token looks exactly like Vladimir described. Thanks!

 

[Antebios]

I have a question on this: Does this work with Google's One storage solution? Or this works with Google Drive storage sync only?

 

[jbrown705]

Do you have rclone syncing data to gdrive using the gui or the cloud credentials somehow? I just got rclone setup and am making my initial upload, but it keeps stopping every time shell window closes. I could use some help getting it setup to keep a constant sync and/or run as a service so it starts with a system restart. Never used cloud credentials and this is my first use of rclone so its new to me as well.

 

[hervon]

Followed this tutorial: https://rclone.org/drive/

Through freenas cli

Works perfectly.

 

[jbrown705]

Followed this tutorial: https://rclone.org/drive/

Through freenas cli

Works perfectly.

I did follow that to get it up and running, but as soon as the shell closes, the upload stops. My initial upload is about 5TB's so i am currently running from the CLI from my mac mini but will likely take about 2 weeks to fully complete. Is there a way to have this run as a service in the background or set up to check and sync regularly?

 

[hervon]

Strange. Mine runs independently from the shell. It runs as scheduled in the GUI.

 

[jbrown705]

Strange. Mine runs independently from the shell. It runs as scheduled in the GUI.

How did you schedule in the GUI? Everything I did from that guide was CLI. Once I was done I ran rclone copy /mnt/media gdrive:/media/ to start my initial sync. Did you set something up in the freenas gui? Maybe I missed something?

 

[hervon]

GUI / Tasks / Cloud Sync Tasks / Schedule the Cloud sync task

 

[jbrown705]

GUI / Tasks / Cloud Sync Tasks / Schedule the Cloud sync task

Oh ok, I didn’t do anything with that. I’m not familiar with it. I just tried to setup the credentials and I’m not sure what to put in the access token. Is that that alphanumeric code that came back when I authorized rclone to use gdrive? Some value they show in the CLI if I run config again and hit edit the remote?

 

[hervon]

oh. The token goes here :

GUI / System / Cloud Credentials /

Mine looks like this : {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2018-09-19T08:04:17.130942669-04:00"}

 

[jbrown705]

oh. The token goes here :

GUI / System / Cloud Credentials /

Mine looks like this : {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2018-09-19T08:04:17.130942669-04:00"}

Thats what it was! I was only taking after the semicolon for the access_token and putting it in. I tested and single files and it works. So you just set yours up to run daily and it will check for differences and push them out to gdrive? Did you have any issues doing like this for the initial upload? If i set to daily, it will not be finished before the next run tries to initiate.

Possible to set up any notices to tell you it completed or failed or something?

It the cloud sync supposed to be insanely slow? Like 2Mbps slow?