Hello everyone,
I have two different systems (see my signature) in two separate location running FreeNAS (11.2-U6/U7) with different specs. They both serve files via Samba shares and run Syncthing just fine. They remotely backup each other via zfs replication over ZeroTier tunnel. This base configuration is quite stable. One machine has been running great for almost two years, the other one is newer and has been fine for 3 months.
Long story:
The problem
On both systems, Nextcloud and Emby jails, under certain circumstances, "silently" choke the whole system.
The symptoms
The possible causes
With Emby the problem appears to be, as said here this new 1TB library of mp3 files which I mounted one the both the systems. By deleting the mount from the jail and the system seems stable. I want to remark though, that the system is working fine with a 2TB library of movies and 50GB library of mp3.
This is driving me nuts, in both cases, some inner problem makes the whole system to hang, and this is quite serious. Jails should mitigate if not prevent this.
What to do? Where to post debug archives?
After this facts I saved the Debug Archives from Freenas UI and I tried to have a look at logs and stuff but I am not qualified to do that.
Can someone try to have a look at those? Where I could post them? Do they contain sensitive information I need to strip off?
Thank you so much,
Marco
Related Discussions
https://www.ixsystems.com/community...ink-makes-freenas-console-unresponsive.80120/
https://www.ixsystems.com/community...by-plugin-whole-freenas-system-is-busy.80046/
I have two different systems (see my signature) in two separate location running FreeNAS (11.2-U6/U7) with different specs. They both serve files via Samba shares and run Syncthing just fine. They remotely backup each other via zfs replication over ZeroTier tunnel. This base configuration is quite stable. One machine has been running great for almost two years, the other one is newer and has been fine for 3 months.
Long story:
At one point I started experimenting with Nextcloud plugin. I mounted my shares into the jail and used "External storage" feature in Nextcloud. In LAN everything was working fine (apart some minor permissions issues). At the same time I installed Emby to share some movies (2 TB) and music (50GB) locally and, again, everything was ok.
Many things happened. I'll try to put order in my memories and summarize a bit.
One weekend different things happened on System1:
At first I thought was a router fault. The machine was not reachable trough ZeroTier either.
At one point the local users told me that the SMB shares where not reachable. I kept thinking the issue was the router.
At one point I manage to connect to the BMC/IPMI interface (I plan to secure it further with a VPN but atm is just exposed over HTTPS on a high port) and the FreeNAS console itself was not responding (!). At the time seemed that the issue was an intermittent link, but still I was worried from the weird behavior. I mean, the system should stay up and not hang even if the link is intermittent. I described the problem here.
Since I was 300km away from System 1, to distract myself I started to replicate the steps 1. and 2. on my home machine, System 2.
Again, at first everything was working fine, but then after few hours Nextcloud stopped responding, FreeNAS web UI was slow as hell and I could not manage to reach the jail tab, at one point the system stopped responding at all. Since I don't have BMC/IPMI I connected the monitor to the server and I was not getting any VGA feed. I ended up restarting the server and quickly accessing the shell to stop the Nextcloud jail since it was set to autostart. The system was fine again. After few hours the system started slow down again. The Nextcloud jail was not running. The FreeNAS Web UI was accessible but extremely slow. I managed to load the "Display system processes" page and nothing was pinning the CPU. But still the system was slow as hell. Samba shares were barely accessible. Took me 2 minutes to load the jails page and to stop Emby. The system went back to normal.
Needless to say that when I went back where System 1 is, after I replaced the modem/router because you-never-know, I experienced the same behavior of System 2. So, the router is not the problem, even though at the moment I am sticking with the new one in order to reduce the variables.
So I am left with two different systems manifesting the same weird problem: Emby and Nextcloud jail are, in certain circumstances, choking the system "silently".
If you read until here, thank you very much. This whole thing is one of those you need to write down somewhere to feel a bit better!
Many things happened. I'll try to put order in my memories and summarize a bit.
One weekend different things happened on System1:
- I opened Nextcloud to the internet using a Caddy jail with a reverse proxy
- I added a 1TB mp3 music library to Emby
- Few days before we changed ISP and a new modem/router was installed
At first I thought was a router fault. The machine was not reachable trough ZeroTier either.
At one point the local users told me that the SMB shares where not reachable. I kept thinking the issue was the router.
At one point I manage to connect to the BMC/IPMI interface (I plan to secure it further with a VPN but atm is just exposed over HTTPS on a high port) and the FreeNAS console itself was not responding (!). At the time seemed that the issue was an intermittent link, but still I was worried from the weird behavior. I mean, the system should stay up and not hang even if the link is intermittent. I described the problem here.
Since I was 300km away from System 1, to distract myself I started to replicate the steps 1. and 2. on my home machine, System 2.
Again, at first everything was working fine, but then after few hours Nextcloud stopped responding, FreeNAS web UI was slow as hell and I could not manage to reach the jail tab, at one point the system stopped responding at all. Since I don't have BMC/IPMI I connected the monitor to the server and I was not getting any VGA feed. I ended up restarting the server and quickly accessing the shell to stop the Nextcloud jail since it was set to autostart. The system was fine again. After few hours the system started slow down again. The Nextcloud jail was not running. The FreeNAS Web UI was accessible but extremely slow. I managed to load the "Display system processes" page and nothing was pinning the CPU. But still the system was slow as hell. Samba shares were barely accessible. Took me 2 minutes to load the jails page and to stop Emby. The system went back to normal.
Needless to say that when I went back where System 1 is, after I replaced the modem/router because you-never-know, I experienced the same behavior of System 2. So, the router is not the problem, even though at the moment I am sticking with the new one in order to reduce the variables.
So I am left with two different systems manifesting the same weird problem: Emby and Nextcloud jail are, in certain circumstances, choking the system "silently".
If you read until here, thank you very much. This whole thing is one of those you need to write down somewhere to feel a bit better!
The problem
On both systems, Nextcloud and Emby jails, under certain circumstances, "silently" choke the whole system.
The symptoms
- Samba shares extremely slow/not working
- FreeNAS UI extremely slow/not working
- FreeNAS Console (!) extremely slow/not working
- SSH extremely slow/not working
- (No output from VGA port, but this might be an issue with the HPE Microserver)
- "Display System processes" not showing anything weird
- Netdata and FreeNAS UI, when working, are not showing high CPU utilization
The possible causes
With Emby the problem appears to be, as said here this new 1TB library of mp3 files which I mounted one the both the systems. By deleting the mount from the jail and the system seems stable. I want to remark though, that the system is working fine with a 2TB library of movies and 50GB library of mp3.
I could not find a plausible explanation of what happened with Nextcloud. Seems ridiculous, but the triggering factor was putting it behind an HTTPS reverse proxy. I didn't experimented much because it used to completely hang the system causing me to hard reset it. I wrote "used" because in the meantime I also updated the system from 11.2-U6 to U7 and I kinda lost track of the status of it. Now I does not work at all apparently. Of course I am not saying much, I know. I should try to replicate this with in a new jail but this issue is so frustrating that until now I didn't make further experiments.
This is driving me nuts, in both cases, some inner problem makes the whole system to hang, and this is quite serious. Jails should mitigate if not prevent this.
What to do? Where to post debug archives?
Apart from experimenting further with plugins and opening discussion on the specific (sub)forums for Emby and Nextcloud, I would like to know what causes this from the base system perspective, i.e. FreeNAS/FreeBSD. Again, I don't think is desirable that plugins/jails may have such a destructive impact on the base system.
After this facts I saved the Debug Archives from Freenas UI and I tried to have a look at logs and stuff but I am not qualified to do that.
Can someone try to have a look at those? Where I could post them? Do they contain sensitive information I need to strip off?
Thank you so much,
Marco
Related Discussions
https://www.ixsystems.com/community...ink-makes-freenas-console-unresponsive.80120/
https://www.ixsystems.com/community...by-plugin-whole-freenas-system-is-busy.80046/
Last edited: