FreeNAS sluggish & webUI not responding, Samba4Alert failed messages

[j2mc]

Been running great for 2 or so years now, then 2 days ago my shares got really slow(under 1mbit read speed) and my Jails stopped responding. Tried to get into the web UI and it was also not responding. SSH works but is also very sluggish, /var/log/messages is full of:

Code:

Sep  3 13:08:48 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:13:03 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: Failed connection handshake
Sep  3 13:21:36 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: Failed connection handshake
Sep  3 13:26:59 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:30:52 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:34:27 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:36:57 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: Failed connection handshake
Sep  3 13:39:45 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:41:40 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out
Sep  3 13:44:52 homenas /alert.py: [system.alert:393] Alert module '<samba4.Samba4Alert object at 0x814d0c828>' failed: timed out

After a reboot it works great again for a couple hours, then stops responding/gets sluggish again. The same time it stops responding those "samba4" messages start up again.

When searching I found someone recommended to "have selected zfs_space, zfsacl and streams_xattr modules" on all SMB shares, those are all already selected on my shares. Another thread said a Windows 10 update was to blame, however I've avoided accessing any of the SMB shares from my windows or Linux machines and it still started happening again. When the problem started I was on version 11.1U4 and attempted an update to U6, but everytime I tried to update these samba4 alerts would start up and it would stop responding. Finally last night I took a backup of the config and installed 11.1U6 on a new USB drive, reloaded my config and the problem still exists. Not sure what else to try/look for now.

System Specs:
HP Z420 Workstation
Xeon X5679
12GB ECC Ram
6x WD AV 2.0TB

 

[dlavigne]

Someone (possibly you?) reported this the other day: https://redmine.ixsystems.com/issues/44229. The ticket isn't viewable right now as the dev is still reviewing the OP's debug info.

 

[j2mc]

Not my ticket, I thought I would start here in the forum first. Let me know if there's anything I can provide to help!

 

[soleilblanc]

Same problem reported in another thread here https://forums.freenas.org/index.php?threads/samba4-samba4alert.63499/

Let me know as well if you need anything to tshoot this.

Jon

 

[j2mc]

Sorry, can't help anymore, it was impossible for me to continue with this problem so I moved onto another platform. Same hardware working great there.

 

[soleilblanc]

This is solved, at least with my scenario. There was a small bug in freenas (bug 46821) that was causing the error messages, but that was not the root of the problem. I'm running a syncthing jail which use a feature to watch changes on the filesystem. When doing this, i was running out of file descriptors, which was causing all king of weird error messages as well as slowdown. It had over 128000 entries in fstat, which got reduced to 200 after i removed the watcher.

Hope this can help someone else.

Jon

 

[j2mc]

Good find, I was running a syncthing jail as well. Now that you mention it, the problems started just after adding a new device to my syncthing network, I remember that new watch for changes feature getting enabled when I logged in to add that device!