SCALE becomes unreachable every few weeks

[grigory]

<div class="bbWrapper">FYI, in the ticket with ixsystems it was suggested that one or more pods is overloading the system and I should try to narrow down the problem process/pod.<br /> Since replicating this seems to take on average 2 weeks, I am not able to justify disabling an app and then waiting two weeks without that application.<br /> For now I have set the following script to run every 5m and hopefully next time this happens I will capture something useful.<br /> <br /> <div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>#!/bin/bash # Define the directory where the stats files will be stored output_directory=&quot;/root/dockerstats_output/&quot; # Define the file name format with date output_file=&quot;${output_directory}$(date +&quot;%Y%m%d&quot;).txt&quot; # Get current timestamp timestamp=$(date +&quot;%Y-%m-%d %H:%M:%S&quot;) # Get top processes with headers top_processes=$(top -b -n 1 | head -n 22) # Run docker stats command and filter top 10 containers by memory usage docker stats --format &quot;table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}&quot; --no-stream | { echo &quot;Timestamp: $timestamp&quot; echo -e &quot;\nDocker Stats:&quot; echo -e &quot;CONTAINER ID\tNAME\tCPU %\tMEM USAGE (LIMIT)&quot; tail -n +2 | sort -k4 -h -r | head -n 10 echo -e &quot;\nTop Processes:&quot; echo &quot;$top_processes&quot; } &gt;&gt; &quot;$output_file&quot;</pre></div></div>

 

[Davvo]

<div class="bbWrapper"><blockquote data-attributes="member: 121047" data-quote="grigory" data-source="post: 780544" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780544" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780544">grigory said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I am not able to justify disabling an app and then waiting two weeks without that application. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Imho the justification is the resolution of the issue, but it's not not my system nor my time.</div>

 

[grigory]

<div class="bbWrapper"><blockquote data-attributes="member: 117605" data-quote="Davvo" data-source="post: 780567" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780567" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780567">Davvo said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Imho the justification is the resolution of the issue, but it's not not my system nor my time. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Yes fair, I mean I am not willing to disable each app for two weeks to figure out it isnt that app and I need to go on to the next one. If I could easily pick the bad one and disable it I think I could justify that.</div>

 

[Davvo]

<div class="bbWrapper"><blockquote data-attributes="member: 121047" data-quote="grigory" data-source="post: 780569" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780569" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780569">grigory said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Yes fair, I mean I am not willing to disable each app for two weeks to figure out it isnt that app and I need to go on to the next one. If I could easily pick the bad one and disable it I think I could justify that. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Yup and I completely understand that... in the end someone has to put in some of his own time for the greater (including his) good.</div>

 

[morganL]

<div class="bbWrapper">Looking for similar problem and found this problem and later resolution:<br /> <br /> PROBLEM: I've been troubleshooting my NAS crashing. I'm running the latest edition of TrueNAS Scale 22.12.3.3, but I've has this on previous versions too. This issues has popped up on and off since updating from Angelfish to Bluefin. Occasionally I will notice my NAS has crashed and have watched it crash. I have pinpointed it down to when the Plex app is running (Official or Truecharts). Right before it crashes, the dockerd process (seen in top and in the gui dashboard shows &quot;services&quot; consuming ram) consumes all system memory, causing other processes to be killed by oom-killer (seen in log/messages) until ultimately it needs a reboot. I have even tried backing up my config and reinstalling the OS fresh and restoring config to no avail.<br /> <br /> SOLUTION: unmounted pool in apps setting window, nuked the ix-applications folder, mounted pool and reconfigured apps from scratch the same. This resolved my issue. Suspect leftover garbage code from updating from angelfish to bluefin as that is when this issue started.<br /> <br /> So a question: does the problem occur only on systems that updated from angelfish to bluefin??</div>

 

[sdf786]

<div class="bbWrapper">I am having the same issue on TrueNAS Scale 22.12.3.3 with a fresh install, no upgrade. I am not using any local apps or docker on the Truenas host itself. I only have a Rocky 9 VM going with podman/docker on it with a mounted NFS volume from Truenas. Single network port is in bridged mode. <br /> <br /> After 2 days the network drops to all VMs and the Truenas Scale host.<br /> <br /> Running with: <br /> 32GB ECC Ram<br /> 3900X CPU<br /> ASUS AM4 TUF Gaming X570-Plus<br /> rx 580 GPU<br /> 4x 8 TB WD Red plus<br /> 2 x 1 TB Samsung 870 evos</div>

 

[morganL]

<div class="bbWrapper"><blockquote data-attributes="member: 119280" data-quote="sdf786" data-source="post: 780580" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780580" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780580">sdf786 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I am having the same issue on TrueNAS Scale 22.12.3.3 with a fresh install, no upgrade. I am not using any local apps or docker on the Truenas host itself. I only have a Rocky 9 VM going with podman/docker on it with a mounted NFS volume from Truenas. Single network port is in bridged mode.<br /> <br /> After 2 days the network drops to all VMs and the Truenas Scale host. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Sounds like a different problem.... can you see if the same symptoms of memory shortages are causing the issue?<br /> <br /> How much RAM was allocated to the VM?</div>

 

[grigory]

<div class="bbWrapper"><blockquote data-attributes="member: 84138" data-quote="morganL" data-source="post: 780578" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780578" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780578">morganL said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Looking for similar problem and found this problem and later resolution:<br /> <br /> PROBLEM: I've been troubleshooting my NAS crashing. I'm running the latest edition of TrueNAS Scale 22.12.3.3, but I've has this on previous versions too. This issues has popped up on and off since updating from Angelfish to Bluefin. Occasionally I will notice my NAS has crashed and have watched it crash. I have pinpointed it down to when the Plex app is running (Official or Truecharts). Right before it crashes, the dockerd process (seen in top and in the gui dashboard shows &quot;services&quot; consuming ram) consumes all system memory, causing other processes to be killed by oom-killer (seen in log/messages) until ultimately it needs a reboot. I have even tried backing up my config and reinstalling the OS fresh and restoring config to no avail.<br /> <br /> SOLUTION: unmounted pool in apps setting window, nuked the ix-applications folder, mounted pool and reconfigured apps from scratch the same. This resolved my issue. Suspect leftover garbage code from updating from angelfish to bluefin as that is when this issue started.<br /> <br /> So a question: does the problem occur only on systems that updated from angelfish to bluefin?? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>It would be interesting to compare the nuked ix-apps folder with the new fully configured one. What exactly would have changed there?<br /> I do think this problem occurred for me after this upgrade but I also added probably 5+ apps and make several other changes (changed datasets to unencrypted and other config changes) after upgrading as well.</div>

 

[grigory]

<div class="bbWrapper">I tried to replicate this using the closest thing I could to the crash that happened yesterday (my kids watching a movie). Although this has happened at other times.<br /> I was monitoring htop and jellyfin was using a lot of resources as I expected, although all cores pinned around 95~100% was not expected.<br /> I let the movie play for 30m, and no crash, very high cpu. I noticed ffmpeg was taking a lot of CPU I think it is transcoding when it should not be, my understanding is that jellyfin will only transcode when necessary. I have explicitly disabled this in jellyfin now.<br /> When I stopped the movie in jellyfin waiting about 5~10m saw all resources drop to reasonable levels (30~60% cpu). After this happened the server crashed.<br /> I am hesitant to say this is the issue every time, but at least in my case this appears to maybe the most common cause. As I said it has happened in the middle of the night when no one is actively streaming media.<br /> <br /> attached is some of the script output during this time.</div>

 

[morganL]

<div class="bbWrapper"><blockquote data-attributes="member: 121047" data-quote="grigory" data-source="post: 780649" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780649" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780649">grigory said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> It would be interesting to compare the nuked ix-apps folder with the new fully configured one. What exactly would have changed there?<br /> I do think this problem occurred for me after this upgrade but I also added probably 5+ apps and make several other changes (changed datasets to unencrypted and other config changes) after upgrading as well. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>There was a major change to overlayfs data management. I have no idea whether this could be a cause.<br /> Just trying to see if this is a pattern.</div>

 

[grigory]

<div class="bbWrapper">I just had this happen again while I was uploading a large number of assets to immich.<br /> however my script output is less helpful.</div>

 

[grigory]

<div class="bbWrapper">I have had this happen two more times while trying to finish uploading assets to immich. And one other time while the system was coming back up it just crashed without me trying to do anything at all, just as all the apps were deploying.<br /> I noticed this discord thread about prometheus being a resource hog as well maybe useful for some. <a href="https://discord.com/channels/830763548678291466/1143544215809298522" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://discord.com/channels/830763548678291466/1143544215809298522</a><br /> <br /> I am very tempted to grasp the Cobia RC straw, and install it to see if this gets resolved.</div>

 

[grigory]

<div class="bbWrapper">I disabled prometheus metrics on all apps and shutdown the prometheus-operator and prometheus app (all truecharts).<br /> So far, after doing that, I was able to perform the same action in immich that had crashed things previously.<br /> So can this be chalked up to my system being overloaded? I will wait and see.</div>

 

[morganL]

<div class="bbWrapper"><blockquote data-attributes="member: 121047" data-quote="grigory" data-source="post: 780983" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780983" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780983">grigory said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I disabled prometheus metrics on all apps and shutdown the prometheus-operator and prometheus app (all truecharts).<br /> So far, after doing that, I was able to perform the same action in immich that had crashed things previously.<br /> So can this be chalked up to my system being overloaded? I will wait and see. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> Useful to know..... I don't know if its overloaded or whether the app or dockerd has a memory leak?</div>

 

[sdf786]

<div class="bbWrapper"><blockquote data-attributes="member: 84138" data-quote="morganL" data-source="post: 780620" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=780620" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-780620">morganL said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Sounds like a different problem.... can you see if the same symptoms of memory shortages are causing the issue?<br /> <br /> How much RAM was allocated to the VM? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Should have mentioned I do have 64 GB of ECC Ram on the system with 16 GB allocated to the singular VM</div>

 

[sdf786]

<div class="bbWrapper">Will add too, i moved to 23.10-RC.1 and still the same issue after a few days with the network hanging.</div>

 

[morganL]

<div class="bbWrapper"><blockquote data-attributes="member: 119280" data-quote="sdf786" data-source="post: 781189" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=781189" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-781189">sdf786 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Will add too, i moved to 23.10-RC.1 and still the same issue after a few days with the network hanging. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>If you have any diagnostics on the memory consumption.. eg htop.... could you open a NAS ticket (report a bug).<br /> <br /> Simple configurations that exhibit the issue make diagnosis much easier.<br /> <br /> Any Apps at all?</div>

 

[sdf786]

<div class="bbWrapper">no, no apps. Had a fresh install of 22.12.3.3 from the ISO in which i moved away from using the app directly so never configured that part at all. <br /> <br /> I have a separate VM for docker on RHEL which i upped the mem too. <br /> <br /> Outside of that I have one NFS (mounted to VM) and SMB share and bridged networking (coming from an ethernet pcie card) so the VM and Scale can talk. Z1 array. Nothing crazy going in terms of config.<br /> <br /> I am trying to find a way to get a crash dump/support bundle via cli the next time it nose dives, since I will probably only have console access. If you have the command I will see if i can get it for you guys</div>

 

[morganL]

<div class="bbWrapper"><blockquote data-attributes="member: 119280" data-quote="sdf786" data-source="post: 781246" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=781246" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-781246">sdf786 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> no, no apps. Had a fresh install of 22.12.3.3 from the ISO in which i moved away from using the app directly so never configured that part at all.<br /> <br /> I have a separate VM for docker on RHEL which i upped the mem too.<br /> <br /> Outside of that I have one NFS (mounted to VM) and SMB share and bridged networking (coming from an ethernet pcie card) so the VM and Scale can talk. Z1 array. Nothing crazy going in terms of config.<br /> <br /> I am trying to find a way to get a crash dump/support bundle via cli the next time it nose dives, since I will probably only have console access. If you have the command I will see if i can get it for you guys </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>If you report the issue, the developers may tell you what they need. Thanks.</div>