Strange High Memory Usage by Services

[farshidhss]

Hi everyone!

I need help to understand what is going on with my freenas memory usage. After a reboot the nas memory is normal but over time Services increases to the point that almost entire Nas is consumed (see image below)

I only have one jail with MiniDLNA and one ubuntu virtual machine with 4GB memory allocated. Can someone help me investigate why Services are consuming so much?

I'm running a system based on a Dell T1700 with 24GB memory.

 

[farshidhss]

By the way, when I do htop I get totally different results:

 

[ilmarmors]

I have observed similar memory problem:

System Information
Platform: Generic
Version: FreeNAS-11.3-U5
Uptime: 1:45PM up 38 days, 7:14, 1 user

I have only SMART, SSH, WebDav services enabled. There are 1TB a day upload via SSH. Services memory gets larger and larger over time. I have restarted SMART, WebDav services, it didn't make any changes. Sudden jump in mid November could be due to installing and running IXsystems plugin, but I stopped it after few days, but service memory didn't decrease.

htop output doesn't explain where the memory shown in graph above went.

 

[farshidhss]

Anyone with any ideas?

 

[SweetAndLow]

If you are going to use htop sort by Virt memory to get better results. Is anyone using auto tune? If you are disable that and all the tunable.

 

[Scoot_Mulner]

Anyone make any progress on this? I'm running into the same issue.

Version: TrueNAS-12.0-RELEASE
Intel(R) Pentium(R) CPU G4620 @ 3.70GHz

I have an Ubuntu VM running with 2 GB of RAM allocated to it as well as the Syncthing plugin.

 

[danb35]

I'll bet it's the same issue I'm seeing:

https://jira.ixsystems.com/browse/NAS-108863

 

[Scoot_Mulner]

I'll bet it's the same issue I'm seeing:

https://jira.ixsystems.com/browse/NAS-108863

Hi danb35,
Thanks for the reply. I had a look but I'm not sure that it is the same issue. My "collectd" process doesn't seem to be consuming nearly as much RAM as shown in your earlier screenshots. In fact, according the "htop", my system has plenty of free RAM.

 

[danb35]

My "collectd" process doesn't seem to be consuming nearly as much RAM as shown in your earlier screenshots.

From what you're showing, I'd agree. Not sure what's going on, then. I wonder if it's just a matter of the RAM on your system and what you're using it for.

 

[Scoot_Mulner]

For my own case, I think I have narrowed it down to the AFP service. I was watching the amount of RAM consumed by "Services" grow as my MacBook was running a TimeMachine backup. I have turned off the AFP service, rebooted and the RAM used by "Services" has stayed around 5.7 GB. I'll monitor it for a few more days and see if anything changes. If it is indeed the AFP service, is there anything I can do to fix it? I would really like to keep my backups running on my MacBook.

 

[anodos]

For my own case, I think I have narrowed it down to the AFP service. I was watching the amount of RAM consumed by "Services" grow as my MacBook was running a TimeMachine backup. I have turned off the AFP service, rebooted and the RAM used by "Services" has stayed around 5.7 GB. I'll monitor it for a few more days and see if anything changes. If it is indeed the AFP service, is there anything I can do to fix it? I would really like to keep my backups running on my MacBook.

View attachment 44455

If you can reproduce the issue with netatalk, I'd be keen to look at it. If there's a memory leak in it, I'll try to fix it.

 

[Herr_Merlin]

Same here on 1 of your systems.. runs fine.
Only nfs share thats it. About 140GB used for services

 

[Scoot_Mulner]

If you can reproduce the issue with netatalk, I'd be keen to look at it. If there's a memory leak in it, I'll try to fix it.

Ok, back again after trying to nail down exactly what is happening. It is definitely the AFP service. I have two MacBook Pro's. With only one MacBook on and running Time Machine, I see higher memory usage from "Services" at about 7 - 7.5 GB. With both MacBooks on, my "Services" memory usage goes up to 13.5 GB which is almost all of my 16 GB of RAM on the machine. I set the log level to 'debug' and left TrueNAS for about 6 hrs today. That was a bit of a mistake... the afp.log file is now 2.6 GB. oops... I'll try again with a lower log level and see if there is anything obvious in it.

Not sure if it matters but one MacBook Pro is a 2018 model while the other is from 2013. Also, the drive I have shared for the Time Machine backups is full. 153.26 GiB free with 2.48 TB consumed. Time Machine says it deletes older backups to make room for the new backup. Maybe something is wrong with that?

 

[anodos]

Ok, back again after trying to nail down exactly what is happening. It is definitely the AFP service. I have two MacBook Pro's. With only one MacBook on and running Time Machine, I see higher memory usage from "Services" at about 7 - 7.5 GB. With both MacBooks on, my "Services" memory usage goes up to 13.5 GB which is almost all of my 16 GB of RAM on the machine. I set the log level to 'debug' and left TrueNAS for about 6 hrs today. That was a bit of a mistake... the afp.log file is now 2.6 GB. oops... I'll try again with a lower log level and see if there is anything obvious in it.

Not sure if it matters but one MacBook Pro is a 2018 model while the other is from 2013. Also, the drive I have shared for the Time Machine backups is full. 153.26 GiB free with 2.48 TB consumed. Time Machine says it deletes older backups to make room for the new backup. Maybe something is wrong with that?

Services usage increase in GUI isn't really what I'm concerned about (that number is relatively unhelpful). It's specifically if the the afpd process memory usage is growing without bounds. top -o res

 

[Scoot_Mulner]

Services usage increase in GUI isn't really what I'm concerned about (that number is relatively unhelpful). It's specifically if the the afpd process memory usage is growing without bounds. top -o res

Oh, top and htop don't show anything strange. For some reason the GUI seems to think all my free RAM is consumed. Even when I try to start my VM (which I have allotted 2 GB of RAM) I get an error warning me I don't have enough memory. So something is up, but I'm not sure what...

 

[Baenwort]

So I'm seeing this also (since the U1.1 update) and I have no Apple devices that access the TrueNAS nor do I have AFP services enabled.

I do not have autotune enabled.

I am seeing performance problems on clients and a reboot of the TrueNAS server restores performance (and proper memory usage) but it seems ~.1GB is being consumed and not released per hour which results in problems after a couple days.

 

[steveroch-rs]

I'd like to add to this.
Ever since I enabled AFP to use a SMB-share to backup a MacBook Air my memory gets constantly and continuously eaten away. At least in the GUI.
I have yet to try rebooting but as you can see in the screenshots 'Laundry' is taking up bigger and bigger chunks of memory.
htop shows only 11.7G/64G used.
I am very confused. Maybe a TrueNAS dev or representative can make a statement on what's going on?

 

[danb35]

Ever since I enabled AFP

Why? Unless you're using a very old version of macOS, it supports (and prefers) Time Machine over SMB.

 

[Constantin]

Part of jumping to TrueNAS meant for me to switch to SMB as the primary workhorse protocol - made sense since AFP is deprecated (though working well thanks to netatalk / efforts by @anodos / iXsystems / etc.) The issues of the past setting up TM backups via SMB, fruits, etc. seem resolved. The process was super solid, no special commands had to be added to share configuration setups, and so on. That said, even in Mojave, AFP works wonderfully and I find unix permissions somewhat easier to navigate than the various ACL options.

 

[lancing]

Also having this issue.