Bidirectional Cloud Sync Task

Chiaki

Explorer
Joined
Apr 4, 2016
Messages
51
Hello everyone, so I'm a bit confused about the Cloud Sync Task types.

I understand how COPY and MOVE work, but I'm confused on how or rather why SYNC works as it works.

What I want to achieve is that I have a dataset on TrueNAS Scale that is mapped to a WebDAV folder in Nextcloud and I want these two locations to be exactly identical (if possible even in real-time).

So what I did was create a Cloud Sync Task of the SYNC and PULL type. What this did though is, that whenever someone put a new file in the TrueNAS dataset it got deleted next time the Cloud Sync Task ran, because the file does not exist on Nextcloud.

I guess if I changed the direction to PUSH I would be able to put a new file to Nextcloud but then again if I created a new file on Nextcloud it would just disappear.

How can I create a Cloud Sync Task that deletes a file on Nextcloud if I delete it locally and vice-versa? I want a full bidirectional sync as it happens on a Synology NAS when you use Cloud Sync Tasks.

If I create one PUSH COPY and one PULL COPY task, will I at least have the same state on both locations and be able to add and change files on both sides or would I run into problems then? The caveat to this would be that deletions wouldn't sync :/.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Bi-directional sync is fraught with danger and disappointment.

Will you be able to ensure nobody is editing the same file on one side as somebody else edits it on the other?... last change wins (destroying the changes of the previous one) or need to handle that with some kind of change review system whereby conflicts are saved separately and somebody must review and manually merge changes from both sides (which you must develop and implement yourself as no software deals with it on its own for fileshares (since file locks are supposed to prevent that and they won't be there on both sides).

What about the creation of a file with the same name before a sync can complete... complete loss of original content.

What you're asking for doesn't exist outside of paid services like Office365.

Maybe you can get closer to it by working out how to use collabora properly (https://www.youtube.com/watch?v=dpZurFVxh4c)
 

Chiaki

Explorer
Joined
Apr 4, 2016
Messages
51
Bi-directional sync is fraught with danger and disappointment.

Will you be able to ensure nobody is editing the same file on one side as somebody else edits it on the other?... last change wins (destroying the changes of the previous one) or need to handle that with some kind of change review system whereby conflicts are saved separately and somebody must review and manually merge changes from both sides (which you must develop and implement yourself as no software deals with it on its own for fileshares (since file locks are supposed to prevent that and they won't be there on both sides).

What about the creation of a file with the same name before a sync can complete... complete loss of original content.

What you're asking for doesn't exist outside of paid services like Office365.

Maybe you can get closer to it by working out how to use collabora properly (https://www.youtube.com/watch?v=dpZurFVxh4c)
Creating conflict copies is exactly how Nextcloud deals with it. I can be sure that when a Nextcloud Client tries to overwrite a file that got changed since the last time it looked at it, it will create a conflict copy.

Creating a file before the sync was complete is a tough one but I'm wondering if this is even an issue. I don't have deep knowledge of how WebDAV or Nextcloud handles that but I believe files are locked until a sync for that specific file is completed.

Nevertheless, the use case here is that it's 99,99 % sure one file won't be touched by two different users at the same time, even if that happens a conflict copy would be alright. Nextcloud Client is able to do that. Cloud Backup Sync Tasks aren't?

Everything put aside, is there even a way for truenas scale for a bidirectional Cloud sync or not?
 

Chiaki

Explorer
Joined
Apr 4, 2016
Messages
51
I've found multiple YouTube videos of people that said a PUSH task would download files from the target if they were never instead of overwriting them with anything that was on the source. This is proof to how the Cloud Sync Task seem to not be documented well enough with examples and diverge from how other products that call themselves like that work.

It looks like I'll need to run a Linux Desktop VM, mount the datasets as shares in there and let a Nextcloud Client do the trick.
Docker containers that do the same seem to exist but they weren't updated in a long time so I'd rather not use them. I'm still surprised that two-way sync is not an option. It is an option for Synology products and can be used with a lot of different cloud storages; it even has the same name.


If anyone has a better suggestion for me on how to achieve a bidirectional sync on datasets in TrueNAS Scale (instead of using a Linux VM) I'd be glad to hear it.
 

mgoulet65

Explorer
Joined
Jun 15, 2021
Messages
94
Why don't you just have NextCloud mount the dataset in question as an external share? Then there's nothing to sync.
 

Chiaki

Explorer
Joined
Apr 4, 2016
Messages
51
Why don't you just have NextCloud mount the dataset in question as an external share? Then there's nothing to sync.
To put it frankly, the TrueNAS server side does not provide a proper internet connection for simply mounting it externally on the Nextcloud server. If I just mount it as a WebDAV share there, working in Nextcloud will be horrendously slow because the files aren't sitting there as a local copy. Every time someone would download files from Nextcloud to sync it with their client, they would actually download it from the TrueNAS server (with Nextcloud as the middle-man) which has horrible upload speed.

This would only work somehow if the Nextcloud mount of the storage would be completely cached locally and would check for changes on the TrueNAS dataset to synchronize them on-demand.

This is exactly what a Nextcloud Client would do that I sadly can't run directly from TrueNAS SCALE; I guess I'll have to use a VM for that. (I don't want to install a DE to properly operate the Nextcloud Client GUI.)
 

mgoulet65

Explorer
Joined
Jun 15, 2021
Messages
94
To put it frankly, the TrueNAS server side does not provide a proper internet connection for simply mounting it externally on the Nextcloud server. If I just mount it as a WebDAV share there, working in Nextcloud will be horrendously slow because the files aren't sitting there as a local copy. Every time someone would download files from Nextcloud to sync it with their client, they would actually download it from the TrueNAS server (with Nextcloud as the middle-man) which has horrible upload speed.

This would only work somehow if the Nextcloud mount of the storage would be completely cached locally and would check for changes on the TrueNAS dataset to synchronize them on-demand.

This is exactly what a Nextcloud Client would do that I sadly can't run directly from TrueNAS SCALE; I guess I'll have to use a VM for that. (I don't want to install a DE to properly operate the Nextcloud Client GUI.)
Where is your NextCloud instance with respect to your TN instance (physically)?
 

FabianX2

Dabbler
Joined
Dec 26, 2020
Messages
12
Is there any development on that issue? I would really love to have bidirectional sync. We did use a Synology solution with a google dirve backup for the last two years and where really happy with it. As we now need 200TB+ Storage we did build a TrueNas Scale mashine but are unhappy with the sync methods.

Whats the best workarround if there is no integrated solution?
 

Broffrey

Cadet
Joined
Jul 19, 2023
Messages
4
Crap!
I was trying TrueNAS Scale on my own recent DIY NAS with cool components hoping to get get rid of my old Synology boxes, but if there is no viable equivalent solution to Synology Drive Server and Drive ShareSync, that won't make it :(
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
if there is no viable equivalent solution to Synology Drive Server and Drive ShareSync, that won't make it :(
If you're obsessed with Synology features, but want to bring your own hardware, just run Xpenology on it, not TrueNAS.

If you want similar features to what you mention there, maybe Nextcloud would be it for TrueNAS, but it won't be exactly the same, so if that disappoints you, see the first sentence.
 

Broffrey

Cadet
Joined
Jul 19, 2023
Messages
4
I tried Xpenology but incompatible with my hardware (or to much "crafts" to make if work) and it means hacking the DSM that is forbidden, so I want to stuck with a more standard OS and apps if possible. And ZFS seems pretty cool against BTRFS.
I will take a look at "Syncthing", "Nextcloud" if they can do the job and work together without interference if one completes the other solution.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
it means hacking the DSM that is forbidden
It's open-source... not sure about being "forbidden".

In fact, according to this: https://kb.synology.com/en-my/DSM/help/DSM/Home/license?version=7 it seems to be under the GNU license (making sense if they employ linux under the hood), which implies that they are forced to re-share any development they did on top of other GNU licensed works.

Anyway, if it's not what you want, you'll need to look at the alternatives as it seems you already are...
 

Broffrey

Cadet
Joined
Jul 19, 2023
Messages
4
I wasn't aware of this license aspect, and what it implies. Good to know.

What boring me the most is: If Xpenlogy devs are enough skilled to make DSM believes their loaders is a "real" Synology motherboard, who knows if they let a secret backdoor. Plus it seems on very active dev phase, so the stability is tricky. I'm not comfortable with these 2 main points, so I won't go on this option.


Either tests on TrueNAS Scale enriched with Syncthing/Nextcloud are concluant, otherwise I'll stay on Synology boxes.
 
Top