Syncthing Jail with Folder Watcher Unusably Slow, Crashes Host

alexmarkley

Dabbler
Joined
Jul 27, 2021
Messages
40
Good evening! I've identified a serious issue with Syncthing running in a jail on TrueNAS 12. If I enable folder watching within Syncthing, it slows down the entire system, and it eventually crashes the host completely.

I wrote my situation in more detail in this Syncthing forum post: https://forum.syncthing.net/t/truenas-freenas-folder-watcher-unusably-slow-crashes-host/17161

However, I'm not convinced it's a Syncthing bug! It honestly seems like it could be a bug anywhere in the system.

Apparently this issue has been written about in a few different places:

- https://www.truenas.com/community/threads/freebsd-directory-watching-causing-issues.63578/
- https://forum.syncthing.net/t/directory-watcher-on-freebsd-issue/11561
- https://github.com/syncthing/syncthing/issues/6531

Seems like the conversation ended somewhat unproductively last time, so I would like to kick it back off here.
 

alexmarkley

Dabbler
Joined
Jul 27, 2021
Messages
40
I got a very helpful response from the Syncthing maintainers. See:
I'd like to highlight the comment from the Syncthing forum post: "It’s indeed a known limitation of the BSD file watching mechanism (kqueue). There’s no solution other than disabling filesystem watcher."

Now obviously I'm treading on thin ice here. I understand that FreeBSD has a lot of history behind it, and it is generally very well-respected. So as a Linux-head myself, I am hoping to avoid starting a "why not do it the way Linux does?" flamewar.

Instead, I'll ask it this way. What would it take for TrueNAS (presumably by extending FreeBSD) to offer an optimized file system watching interface that can scale to hundreds of thousands of files without crippling the system?
 

speedtriple

Explorer
Joined
May 8, 2020
Messages
75
I got a very helpful response from the Syncthing maintainers. See:
I'd like to highlight the comment from the Syncthing forum post: "It’s indeed a known limitation of the BSD file watching mechanism (kqueue). There’s no solution other than disabling filesystem watcher."

Now obviously I'm treading on thin ice here. I understand that FreeBSD has a lot of history behind it, and it is generally very well-respected. So as a Linux-head myself, I am hoping to avoid starting a "why not do it the way Linux does?" flamewar.

Instead, I'll ask it this way. What would it take for TrueNAS (presumably by extending FreeBSD) to offer an optimized file system watching interface that can scale to hundreds of thousands of files without crippling the system?
I am about to try Syncthing again on TrueNAS.

Will use it as a "hub" to sync Lightroom catalog between 2 PC's (so that both the pc's does not have to be powered on at same time), and keep photos updated on TrueNAS. As I understand, Nextcloud is not good to keep so many small files constantly in instant sync.

If I disable the filesystem watcher on Syncthing in TrueNAS, but not on the 2 client PC's, what will the result be?
Too late sync between PC's and TrueNAS for rather instant sync?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I missed this at the time of posting because I was out of town. It warrants a response. I do not have any particular comment for @speedtriple 's post, but it at least brought the previous post to my attention.

What would it take for TrueNAS (presumably by extending FreeBSD) to offer an optimized file system watching interface that can scale to hundreds of thousands of files without crippling the system?

As with any free software project:

Feel free to write and contribute the code. You can do this directly yourself, or by finding and funding an interested and capable developer.

Now obviously I'm treading on thin ice here.

Not really. We all like to have nice things. Understanding how to get the nice things should not be thin ice.

o as a Linux-head myself, I am hoping to avoid starting a "why not do it the way Linux does?" flamewar.

Because the way Linux does stuff is often "Linux's way" and the code is undoubtedly GPL-tainted. Additionally, there may be covering patents or other issues that prevent the use of a given code or design strategy.

So, first, kqueue is not a "file system watching interface". It was written as an enhancement/replacement for poll/select to monitor activity on file descriptors, which is a related-but-not-identical issue. It solves a different problem. kqueue is analogous to Linux epoll, which attempts to solve similar select issues. select breaks on a large number of sockets, and with the advent of the Web, seeing a server handle ten or twenty thousand simultaneous connections became a thing, and select broke horribly. Both kqueue and epoll are ways to address this, and software like nginx makes good use of these kernel interfaces to do awesome socket trickery.

I'm not really all that familiar with what's happened in kqueue since the FreeBSD 4 days, incidentally, so someone may come along and one-up me on this. :smile:

I imagine that Syncthing may be using Linux inotify. BSD has no exactly parallel functionality, though a root process could always arrange for audit to provide a raw data stream and then filter for interesting events, sorta like a pcap-for-filesystem activity.

In the end, these things usually get written because some company needed the feature for some reason, and paid a developer to write it. Sometimes things get written by graduate students as part of a thesis. Occasionally a hobbyist just starts writing code for giggles. The reason that this functionality does not exist for FreeBSD is because none of these things has happened. Sad but true.
 
Top