Memory Issue TrueNAS Freezes

[donschwarz]

Hello,
Needing some help here. I'm running into an issue with my TrueNAS where over the course of the day the majority of the RAM will be taken up by Services. This is a production server and it is getting a lot of use. This issue was introduced in v13 update from v12. I've been having to reboot every morning from the IPMI since TrueNAS is frozen. When I log in to IPMI I'm either greeted with

Code:

collectd 11774 - - plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics

or

Code:

swap_pager : indefinite wait buffer : bufobj: 0, blkno: 3236, size: 36864.

I have no Jails or VMs running. I am running SMB, AFP, SMART and SSH for services. Others with this similar issue that have solutions posted seem to be dealing with VMs or Jails causing the problem.

 

[donschwarz]

So this morning I was up early to do my reboot of the server and this time the web GUI was up and accessible but the Dashboard was blank. Couldn't SSH and AFP and SMB were not working. The IPMI window was filled with collectd 11774 - - plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics. But since the TrueNAS Web GUI was kind of up and running I thought I would try to restart using it. It initiated but hung on:

After waiting for way too long I went ahead and used IPMI to power cycle it.

Just adding this information in hopes someone else has dealt with this.

 

[donschwarz]

So after more days of trying to figure out what is going on and trying may different approaches to see what causes TrueNAS to freeze up and services to stop, the only thing that consistently brings it to its knees is running my backup to a different fileserver. I am using a mac running Chronosync that is connected to the TrueNAS share as well as an old Quantum file server that I use for backups. About 1-2hours after the Chronosync job starts the server becomes unresponsive. I then have to reboot it using the IPMI which always has the "collectd 11774 - - plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics" in the terminal window when I log on.

If I keep my backup off, the server stays up and running but still using up most of the memory in Services.