TrueNAS unusable when Syncthing plugin is enable

[Fred974]

Hi all,

I have a huge performance issue where I can barely navigate the TrueNAS web GUI or the CLI after installing the Syncthing plugin.
I found a workaround from this very old post (2018):
https://www.truenas.com/community/t...ui-ssh-terminal-when-running-syncthing.64044/
We are now in 2021 and 3 years later the problem is still present. Could anyone please tell me if this is a know bug or if a fix is in the works?
I tried on 2 different server with different specs with the same outcome. I also installed Syncthing in a jail (not plugin) and still had the same outcome.
Could the problem be that Syncthing is jails is not working properly?

Thank you

 

[Fred974]

Hi, is anyone able to help please?

 

[speedtriple]

Installed Syncthing few days ago, using youtube-Lawrence video for install. There where some issues/dropouts with Syncthing, and finally TrueNAS chrashed.
Rebooted. Since then all seems normal, no sluggishness with TrueNAS GUI and Syncthing working perfectly.
I know Syncthing earlier have had issues with cpu/excessive ram use. Maybe it is about how much ram your TrueNAS have?

The plugin is v1.15 and not the latest Syncthing, so maybe an update to latest v1.18 will solve problems.
I do not know how to manually do this update.

 

[Fred974]

Installed Syncthing few days ago, using youtube-Lawrence video for install. There where some issues/dropouts with Syncthing, and finally TrueNAS chrashed.
Rebooted. Since then all seems normal, no sluggishness with TrueNAS GUI and Syncthing working perfectly.
I know Syncthing earlier have had issues with cpu/excessive ram use. Maybe it is about how much ram your TrueNAS have?

The plugin is v1.15 and not the latest Syncthing, so maybe an update to latest v1.18 will solve problems.
I do not know how to manually do this update.

Hi,

My system work perfectly as long as the do not set the watch for changes to files.

I have 15 folder in my Syncthing, and the folder's size vary from 2GB to 25GB. How does your setup compare if you enable 'Watch for Change' on multiple large folders ?

 

[speedtriple]

I only had 2 folders with 500mb or so. Default, so "Watch for change" is enabled with 3600s.
Tested my Android camera folder now, 23gb. Started syncing to TrueNAS, and in the Syncthing GUI on TrueNAS I shared the folder further to my laptop - with TrueNAS as a hub.

While syncing was going on, tried to logon to TrueNAS GUI - unable to log in, got error on password, 2FA etc.
Tried to access the samba folders on windows and mapped network drives/folders on TrueNAS from another windows pc, unable..error.
The Syncthing GUI on my laptop and phone now shows TrueNAS as disconnected - and sync error to my laptop.

So clearly something is wrong with the Syncthing plugin on TrueNAS, this situation is useless.
I have not rebooted TrueNAS, yet. Will wait and see if it recovers, but so far the whole TrueNAS seems crashed - again.

Assume I will remove Syncthing from TrueNAS, and just use it between my 2 pc's and phone in stead.
I can not really see the need for having it on TrueNAS either.

 

[Fred974]

Assume I will remove Syncthing from TrueNAS, and just use it between my 2 pc's and phone in stead.

...OR just do what I do. If you unchecked the box I mentioned in previous entry..Everything work fine. TrueNAS is the recipient so I dont need to watch the folder.
In the Syncthing on TrueNAS, I set the folder to receive only and disabled 'Watch file'

Anyway.. I point question still not answered tough.
It is clearly a 3 years + old problem that is still ongoing. Like I mentioned in first post, I have the same result when Installing Syncthing inside a fresh jail manually. I don't know if the issue is with iocage or if there is some tunable to adjust.

Any advise is greatly appreciated.

 

[Patrick M. Hausen]

I'd file a bug with Syncthing referring to a problem when installing on FreeBSD. One single thing that I can think of: possibly it's relying in /dev/fd being available. Try: iocage set mount_fdescfs=1 <jailname> and restart the jail. Possibly procfs too: iocage set mount_procfs=1 <jailname>

 

[speedtriple]

...OR just do what I do. If you unchecked the box I mentioned in previous entry..Everything work fine. TrueNAS is the recipient so I dont need to watch the folder.
In the Syncthing on TrueNAS, I set the folder to receive only and disabled 'Watch file'
View attachment 48250

Anyway.. I point question still not answered tough.
It is clearly a 3 years + old problem that is still ongoing. Like I mentioned in first post, I have the same result when Installing Syncthing inside a fresh jail manually. I don't know if the issue is with iocage or if there is some tunable to adjust.

Any advise is greatly appreciated.

Maybe the Syncthing v1.18.0 would work better on TrueNAS. All my other devices have 1.18.0, while TrueNAS 1.15.1.
Have to wait until they update det plugin repo.

Have rebooted TrueNAS, and disabled "watch for changes" in TrueNAS Syncthing.
Initial testing ok. Able to accsess TrueNAS GUI, Syncthing GUI, and windows samba-folders and mapped drives same time as syncing the 23gb camera-folder. No sluggishness at all, and no crash so far. Just have to wait and see how this works now over time.

Since you use "receive only" on TrueNAS, it is for backup purpose only? If recovery needed how to get to the files, just change to "send only"?

 

[speedtriple]

update: TrueNAS still not crashed, but samba stopped working again, this never happened before Syncthing install.
When I log in I see the ram-use is exessive, and slowly increasing towards max. And no syncing going on, all is up-to-date according to Syncthing-v1.15.1. Maybe when it fills up is when TrueNAS crashes again.
This ram-use never occured before I installed Syncthing.

 

[morganL]

My non-technical understanding is that syncthing relies on either inotify to detect file system changes or file system scans.

Inotify in freeBSD is not as performant as in Linux. TrueNAS SCALE may be better.
File system scans will cause metadata reads and file reads which will fill up ARC and RAM. They will have an impact on system performance.

ZFS replication is the more efficient way to replicate data. Syncthing doesn't provide a free lunch.

 

[ChrisRJ]

I just checked on my system (details in signature) and Syncthing 1.9 (installed manually in jail following Tom Lawrence's video) works well with "watch for changes" enabled.

Have you checked the system load during the issue? Is there anything in the log files? That might give a clue about the underlying reason, in addition to the symptoms.

 

[Fred974]

Have you checked the system load during the issue? Is there anything in the log files

I cannot access either the web or cli when the problem occure. htop didn't show me anything but I guess that's because it crashed during the load.

(installed manually in jail following Tom Lawrence's video) works well with "watch for changes" enabled.

I tried it as well and still had the issue. How many folder do you have and what the size?

I think @morganL hit the nail on the head here. What he/she says make sense in terms of metadata reads and zfs performence.

 

[FrankNAS]

As mentioned, libinotify is a shim/api(?) that takes inotify calls and coverts them into something kqueue can use. https://github.com/libinotify-kqueue/libinotify-kqueue
It will quickly run into problems with extremely large number of files. There are other issues too, mostly related on how trying to monitor files under kqueue works. Check the link above if you want to dig into it :)

 

[speedtriple]

As mentioned, libinotify is a shim/api(?) that takes inotify calls and coverts them into something kqueue can use. https://github.com/libinotify-kqueue/libinotify-kqueue
It will quickly run into problems with extremely large number of files. There are other issues too, mostly related on how trying to monitor files under kqueue works. Check the link above if you want to dig into it :)

So Syncthing on TrueNAS is bound to be just trouble with folders with lots of gb/files then?
Better to install Syncthing on something else then if a 24h hub is needed.

 

[morganL]

So Syncthing on TrueNAS is bound to be just trouble with folders with lots of gb/files then?
Better to install Syncthing on something else then if a 24h hub is needed.

My guess is the issue is rate of change of the files and data.. - 100 files a day or 10,000 files per day?
The users running successfully can indicate their rate of change and their CPU/RAM.
The users with issues may have a rate of change that is too high for the platform.
The expectation is that TrueNAS SCALE will outperform, but the upper threshold for TrueNAS CORE is not documented.

Size will matter if scanning the data is needed rather than "watch for changes".... in that case, larger datasets should have less frequent scans. That may not change much between CORE and SCALE.

 

[speedtriple]

Removed Syncthing-plugin from TrueNAS.
TrueNAS now always allow login/2FA-logins.
Seems like Syncthing with lots of files is not the best for a stable TrueNAS.
Maybe on TrueNAS SCALE later.

 

[gt2416]

I have syncthing running in a business setting on TrueNAS Core. It's about 800gb of data and 100's - 1000's of file changes everyday. I have turned off watching changes for all shares and set my folder to receive only. The system seems well managed with average 3-5% cpu usage.

 

[speedtriple]

I might try again.
If receive only, it is only for backup-purpose then?
May I ask if there is an easy way to get the data out again if needed from the backup?

 

[gt2416]

Yes, simply share your syncthing folder/ dataset via smb. You may have to adjust permissions first but you can do it easily from the GUI.
To make this process easier first create and mount a separate dataset for syncthing to your jail (can be done in gui).

 

[speedtriple]

Excellent, thanks for inspiration.