Strange High Memory Usage by Services

[drinking12many]

Still encountering some sort of memory leak as well I think. I am trying to get more info/narrow it down, but I have maybe 10 jails and 2 VMs and after maybe 8-10 days with 64GB or RAM my dashboard shows services taking most of the RAM and jail services etc will start failing or having issues. A full server reboot fixes it usually for another 8-10 or so days. If I can get something concrete I will submit a bug or comment on one. I am not using AFP of any kind its disabled. Its only been running 4 days since last reboot with half of the RAM showing services. Looks like Zoneminder may be eating a bunch of mine but I haven't really dug into it much, I only have 3 cameras but it could be doing something weird in a jail.

 

[MagnusO]

I am most likely also seing the same behaviour on my system. I have run into this just as I started testing TrueNAS. The system is freshly installed, running nothing other than NFS and SMART. No VMs, no jails, nothing...
In my case, starting more than 1 simultaneous transfer to my NFS share triggers this.
After about 30 minutes of transferring files "services" is hoggin about 95.5% of installed memory and transfer speeds are dropping from around 50MB/s to about 3MB/s.

 

[MagnusO]

Quick update...
I may have found a solution to the problem (for meg at least)
I realised I had done som testing on jumbo frames. I tried resetting the MTU to 1500, both on the TrueNAS and on the NFS client.
Steadily transferring at around 75MB/s.

 

[drinking12many]

One of my issues appears to be the Tautulli Jail, it was using over 8GB of RAM after about 8 days. I restarted the Jail and recovered a good 3rd of my services RAM being reported. Does anyone know of any way to limit RAM usage in a jail? 8GB seems um a little crazy.

EDIT: was using 10GB for Tautulli today (5/7)... I am trying this to see if it causes any problems - iocage set memoryuse=4G:deny tautulli

 

[Dunuin]

Got the same issue and I'm also using jumboframes. But I really need the jumboframes because otherwise the 10Gbit NIC gets slowed down to 3Gbit because the CPU can't keep up with all the packets. IF that might be caused by jumboframes, someone knows why that could be a problem?

 

[Herr_Merlin]

sounds alright with jumboframes. the issue only occures on systems with jumbo frames

 

[AnosVoldigoad]

Hi all, I noticed in the last few days that I have the same problem and also know what it is with me. Since I offer webdav as an access option to pull data via cloud sync with another TrueNAS, the services RAM gets very full.

Image 1:

Image 2:
The other TrueNAS pulled 15 GB of data from 19:00 to 20:50. Exactly during this time, the Servuces RAM increased by about 15.6 GB, which cannot be a coincidence.

In short, I claim that the webdav service also causes this problem. But it is not enough to restart the service, only a reboot of the TrueNAS fixes the problem again.

 

[hansonmi]

Just found this thread researching possible causes for extremely high memory usage for "services". For me I just have the SMB service enabled. RAM usage for services starts out around 2GiB after a reboot but after a few days it just gets to be over 50% of my system memory. This is after a fresh install of 12.0U7. Other than creating a user and group and setting permissions on a single dataset on a pool consiting of just 2x3TB drives, pretty much everything is just left at defaults. The only things been using this for is just getting to know how to use this software and learning. I have the share set up and it is accessed by a Plex server that is on a different machine. Other windows machines just access the dataset via SMB through Windows 11 to mess around moving files, etc just to see how performance is before i commit to building a server box for this. Right now just basically using this as a proof-of-concept for my needs.

 

[anodos]

Just found this thread researching possible causes for extremely high memory usage for "services". For me I just have the SMB service enabled. RAM usage for services starts out around 2GiB after a reboot but after a few days it just gets to be over 50% of my system memory. This is after a fresh install of 12.0U7. Other than creating a user and group and setting permissions on a single dataset on a pool consiting of just 2x3TB drives, pretty much everything is just left at defaults. The only things been using this for is just getting to know how to use this software and learning. I have the share set up and it is accessed by a Plex server that is on a different machine. Other windows machines just access the dataset via SMB through Windows 11 to mess around moving files, etc just to see how performance is before i commit to building a server box for this. Right now just basically using this as a proof-of-concept for my needs.

Does `top` show any processes with unusually high RES?

 

[hansonmi]

Just found this thread researching possible causes for extremely high memory usage for "services". For me I just have the SMB service enabled. RAM usage for services starts out around 2GiB after a reboot but after a few days it just gets to be over 50% of my system memory. This is after a fresh install of 12.0U7. Other than creating a user and group and setting permissions on a single dataset on a pool consiting of just 2x3TB drives, pretty much everything is just left at defaults. The only things been using this for is just getting to know how to use this software and learning. I have the share set up and it is accessed by a Plex server that is on a different machine. Other windows machines just access the dataset via SMB through Windows 11 to mess around moving files, etc just to see how performance is before i commit to building a server box for this. Right now just basically using this as a proof-of-concept for my needs.

This morning i looked at memory usage and it was still low. But i did get it to jump way up by doing the following:
Connected to my SMB share on the trueNAS via mapped network drive and selected a movie files (20GB) that was stored in a folder.
Copied (CTRL-V) the file to add it to my clipboard
Navigated to another folder in the share and paste (CTRL-V). This would put the movie file in 2 locations.

While the copy was in progress, the servioes would jump from around 2.9GiB to 3.9GiB then come back down. However, i cancelld the copy via windows during mid-copy. After the copy was terminated, the Services suddenly jumped up to 6.9GiB and even after 10 mins has not gone back down. What is weird is that the reports (using non-chromioum browser) don't show this happening:

Below are the screenshots. If i were to initiate another copy and cancel it, the memory usage reported thn goes back down a but if it is cancelled via windows during the copy process, it then goes back up and will hover back around the 6.9GiB mark or higher.

I am just really new to this so i am sorry if this is not a lot of info or the same type of thing that was originally being discussed in the thread. I am just trying to figure out why "services" seems to hog memory and not free it.

 

[hansonmi]

Does `top` show any processes with unusually high RES?

not from what i can tell. I responded to my message from last night (above) and looks like our messages crossed..

 

[anodos]

not from what i can tell. I responded to my message from last night (above) and looks like our messages crossed..
View attachment 52489

top -o res

 

[hansonmi]

top -o res

sorry... here is what that shows... And at this time, nothign has changed from my earlier screenshots of the reports or anything either. Still using 6.9GiB for "services" as well.

 

[hansonmi]

So i just did a wipe and clean install to see if this would persist under a different hardwar config and using LSI SAS controllre for the HDDs instead of the built in intel sata controller.
The steps to duplicate the issue of high "services" usage remain the same:

1. Connect to the SMB share via Windows and copy a large file (in my case a 6GB movie) from one location on the pool to another folder on the pool. (yes i get that using putty to get in or using the console is probably the best way to copy/move things).
2. While the file is copying and windows is showing the progress, the "services" usage will fluxuate bit. But during the copy process if the action is cancelled by windows (clicking the "x" to cancel it on the progress window) the memory usage reported for "services" suddenly jumps up quite a bit.

This is where things get weird...
If i were to copy another 10GB file or something, the services usage reports on the dashboard that it actually goes down. It fluctuates a bit but if the copy process actually completes, it reports less usage that when it started out. Another odd thing is that when i deleted that copy of the file from the location i just copied it to (like it someone copied something to a folder and realized it was the wrong location and deleted it) via windows, the memory usage went from 4GiB to over 10GiB. The only way it seems to get this to go back down to a small amount is just to reboot the TrueNAS OS. At this point, i am not sure if it is just a bug in the reporting on the dashboard or what. I am still really new to this whole thing and only hav had it runnign for a few days trying to test this out before putting into a home lab server.
I am not sure and i think i need read more in the manual abotu some features but it seemed like it was storing the file in memory when being deleted almost like a trashcan to restore it or something, but i cannot find any setting for a recycle bin or anything just going through the menus. Windows says it will be permenately deleted, but maybe it is being stored somewhere... i am not sure since i just started using this and need to read more about this software.

Anyways, not sure if any of this helps, but i think since i can reproduce this every time, nothing more for me and i will just wait to see if someone says this is actually a bug or operating as intended.

Below are the screens of the reports and the top -o res in the console.

 

[mistermanko]

collect.d seems to have a little too much hunger for my taste

 

[mistermanko]

The jira ticket from #7 has some interesting information for that matter:

Unfortunately we can't repeat this and can't fix this. There is a number of collectd write plugin memory leak reports open for years. I have to close this issue for now.

A simple workaround is to create a Cron Job that will run /usr/local/etc/rc.d/collectd-daemon restart every midnight.

 

[Constantin]

Suggests that this should be a default installation cron job until they can reproduce and fix the issue?

 

[indy]

Part of this might just be a display error of the middleware.

1) TrueNAS 13 after boot
2) After saving a file
3) After deleting the file

The ARC should immediately discard freed blocks, which maybe confuses the middleware.

 

[NugentS]

err - you only appear to have 4GB of RAM. Thats not enough

 

[indy]

That is just a VM for testing.