TrueNAS with Syncthing nonresponsive within a day - have to kill power

[Chris.karst]

I have 3 machines I have installed TrueNAS 12.0 on.
All have updated to U4.1.
All 3 are set up the same, except hardware.
Syncthing plugin installed, data folder setup outside of jail, and mapped as mountpoint to the jail.
Syncthing initial instal was v1.15, now upgraded to current 1.18. (problem happens before and after)

After reboot, all systems run OK, WEB UI ans Syncthing both function.
but within a few hours it get so unresponsive that WEB UI will not load, and syncthing stops making progress.
Only message on screen of machine is an error about timeout getting drive temperature.
At this point even asking for a reboot at the machine does nothing.
Only recovery is to force power cycle the machine.
This happens on all three machines, only the durration changes.

At this point, I know I need help, I just don't know where to diagnose.

TrueNAS-1:
SuperMicro X9SCL
I3-3220
8GB ECC RAM
SSD boot drive
1 HDD 1TB Pool

TrueNAS-T:
Intel DH61WW
Pentium G620
8GB RAM
SSD boot drive
2 x 500GB HDD as a 1TB Data pool

TrueNAS-S:
Asus P8H67-M LX
I5-3330
16GB RAM
HDD boot drive
3 x 1TB HDD as Raid 1.12 TB Data pool

 

[sretalla]

Start with htop and see if it points to anything obvious in terms of memory or CPU use.

 

[Patrick M. Hausen]

Disable "Watch for changes" in SyncThing.

 

[Chris.karst]

unable to use

Code:

htop

when it becomes nonresponsive, i can't get a shell via the machine itself, and unable to even get the dashboard to load remotely.

I turned off

watch for changes

in syncthing on all shared folders on all three machines.

All three machines were rebooted this morning, and all are nonresponsive this evening.
TrueNAS-S showed last activity less than 5 hours after booting, other 2 are showing about 10 hours.

 

[Chris.karst]

When the web interface is active, It always shows minimal cpu usage, and RAM usage looks OK as well.

 

[Chris.karst]

htop is currently running on all 3 machines,
I'll watch to see if it shows anything when they hang.

 

[Chris.karst]

Not sure if I'm interpreting htop wrong, or if it is reporting it wrong:

Each of 4 cores is about 25-30% (which is inline with what the dashboard is reporting)
But the CPU% of the processes is over 100%

Apears as though it is adding the percentages instead of averaging?

 

[ChrisRJ]

But the CPU% of the processes is over 100%

I am not sure I understand this. So can you elaborate on this or post a screenshot?

 

[Patrick M. Hausen]

But the CPU% of the processes is over 100%

FreeBSD top shows percentage per core. So if a process is multi threaded it can easily use more than 100%. i don't know about htop - I use top.

 

[Chris.karst]

I am not sure I understand this. So can you elaborate on this or post a screenshot?

Like Patrick said above:
htop shows a graph at the top for each core, and percentage of load for each core.

For example: my 4 core system can be showing 30% load on each core.
The UI Dashboard correctly shows that CPU load is 30%

But the process list on htop adds these load% numbers.
This caused Syncthing to be reported as using 120% of the CPU, even tho the CPU load was only 30% total.

 

[Chris.karst]

Disable "Watch for changes" in SyncThing.

Changing this setting after syncthing was running didn't seem to help at first.
Altho syncthing did stop watching, the system still hung after a few hours.
I suspect that syncthing didn't stop whatever process was running even tho it was no longer using it.

After rebooting each machine, and making no other changes (other than watching htop all three machines seem to be running well now.

Thank you!

 

[Chris.karst]

Is it possible that having the active shared folder outside of the syncthing jail and having it as a mount point possibly causing the watch for changes system to get overloaded?

Just curious because some low powered NAS boxes actually run better with it turned on.
Obviously it depends greatly on the OS and file system of the NAS.