High "Services" memory consumption

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
1620918790936.png


Image says it all. Doing some testing with HCIBench (VMWare ova) to determine what kinds of performance I can expect before taking this box to a temporary production (subplanting a Nimble Array during a migration). System is running 12U3.1. Was running with dedupe vdev (NVME mirror), Meta VDEV (SATA SSD mirror), and 12x 10TB Z2 array. This is after testing it and ditching the pool and building it fresh without dedupe; suspecting it could be related, but that image is *after* the pool nuke and rebuild (NVMe as Meta VDEV, no dedupe, omit SATA SSD); so thought is that if that was in any way related to dedupe that it would have flushed that out as well at that time.

Top:
1620918987432.png


Only "Services" running, according to the GUI at least is iSCSI and SMART; flushed both and still there. Rebooting it seems to resolve, but concerned that putting this into production I'd see this behavior occur and rebooting won't be an option.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You can see it better with htop, but I expect that it's python3.8 and therefore the middlewared that's the culprit.

If you restart it (service middlewared restart) does that improve anything? (at least temporarily?)
 

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
You can see it better with htop, but I expect that it's python3.8 and therefore the middlewared that's the culprit.

If you restart it (service middlewared restart) does that improve anything? (at least temporarily?)

Apologies; box is offline and getting packed up soon. I'll keep these suggestions in mind if I see the issue persisting once deployed again.

Odd thing is that this is a clean install with only iSCSI enabled, so there's not a lot of venues for which middleware would be giving the issue.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
Just to keep the thread going : I have the same issue.
Restarting jails , services and middleware has no effect.
It takes about 10 days to look like what the OP has posted.....
 

ddaenen1

Patron
Joined
Nov 25, 2019
Messages
318
Just to keep the thread going : I have the same issue.
Restarting jails , services and middleware has no effect.
It takes about 10 days to look like what the OP has posted.....

I have the same thing. Services keep creeping up after every reboot. Went all the way up to 17GiB middle of last month. Now already back up to 8GiB after 17 days uptime.

1621277383331.png
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
I just went through my memory reporting and can see that this issue did not occur until 12-U2.1 was install on the 24th February.
This is a home server in a relatively stable environment.
TrueNas Memory.png


The spikes are reboots primarily for firmware upgrades.
I have just rebooted into U1.1 and will monitor !
 
Last edited:

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I just went through my memory reporting and can see that this issue did not occur until 12-U2.1 was install on the 24th February.
This is a home server in a relatively stable environment.
View attachment 47241

The spikes are reboots primarily for firmware upgrades.
I have just rebooted into U1.1 and will monitor !
Are you running any jails?
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
Are you running any jails?
HDHomerun Server
Mariadb 10.4.17 for Kodi
PY-Medusa
RslSync
Transmission 3
URBackup

BaseJails at 12.2-P4
Restarting jails makes no difference.

Services

SMB
NFS
SSH
Smart

Restarting services has no effect.
The last time I tried to restart MiddlewareD the server did not come back and I had to do a hardware reset.
 
Last edited:

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
HDHomerun Server
Mariadb 10.4.17 for Kodi
PY-Medusa
RslSync
Transmission 3
URBackup

BaseJails at 12.2-P4
Restarting jails makes no difference.
I was seeing something similar till I upgraded my jails to the latest (12.2-P6). Reboot and weeks later it hasn't reappeared.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
I was seeing something similar till I upgraded my jails to the latest (12.2-P6). Reboot and weeks later it hasn't reappeared.
I'll let it run for a week or two at 12-U1.1 then I will try the jail upgrade.
 

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
FWIW, on another box, I'm only running SMB/CIFS, same result.

Nuked the pool (pool previously had metadatavdevs and dedupe vdevs, going without this round), built a new pool.

1621602970366.png


All I've done since nuking the pool was build a new one and then create a datastore tree that I like to use for representing shares, and then create the shares. There's 6 shares.
Of note, before nuking hte pool, I did copy it all off to another freenas box via SMB (didn't want to do a zfs send/receive, also looking at changing dataset sizing, from what I understand, that doesn't change the dataset record size when using that method)

Only running service according to the UI is SMART and SMB; restarted SMB, no change.

1621603306443.png


1621603334165.png


Interesting, collectd may be a bit of a culprit.....and nopes. stop/start of collectd didn't make any difference.

Also killed/restarted middlewared and no change.

Worth noting, this is "wired" memory. My googlese mentions this is kernel memory; so it's likely there's a leak somewhere either in the kernel itself or in a module.
 
Last edited:

Sirius

Dabbler
Joined
Mar 1, 2018
Messages
41
I'm also having similar issues. Only running SMB, SSH and SMART. No jails and no VMs.

I tried restarting middlewared and that didn't help.

Screen Shot 2021-05-25 at 4.32.42 pm.png
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I'm also having similar issues. Only running SMB, SSH and SMART. No jails and no VMs.
You're not having issues... SMB does caching of its own and is included in the services count. Your chart looks fine.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
I'll let it run for a week or two at 12-U1.1 then I will try the jail upgrade.
So I have upgraded the jails to 12.2-P6 and gone to 12.0-U4 and the laundry still climbs. It seems to be at specific times but so far I have been unable to correlate that to network activity.
Screenshot (43).png
 
Last edited:

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
So I have upgraded the jails to 12.2-P6 and gone to 12.0-U4 and the laundry still climbs. It seems to be at specific times but so far I have been unable to correlate that to network activity.
View attachment 47680
Mine was doing the same after the 12.0-U4 update. Turns out there is a jail upgrade of 12.2-P8 that came with 12.0-U4. I applied the jail upgrade and it's back to normal.

Wish I could say the jail patch was the silver bullet that'll fix your problem, but I've seen a lot of these posts and they weren't all running jails. The 12.0-U4 was supposed to be the fix, so looks like the devs don't have this nailed down.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
I've gone to 12.2-P8 and will see what happens.
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
So nothing so far is making any difference.
I have traced the rise in "Services" to the replay of recordings from the HDhomerun DVR jail.
Restarting that jail makes no difference and TOP within the Jail looks ok.
It is now clear that the Reporting Graph label of "Laundry" is not Laundry as reported by TOP.

Screenshot (45).png
 

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,175
Joined
Jan 27, 2020
Messages
577
So this seems unfixed, still. I was unaware of your issue and created a duplicate. https://jira.ixsystems.com/browse/NAS-114927
Apologies. However, there are still issues with collectd and or system reporting incorrect memory usage.

I'd like to get some pointers as to where I should look to investigate further which application is the culprit. I suspected grafana, but permanently offlining the jail did nothing.
 
Last edited:

hasore

Cadet
Joined
Jul 23, 2019
Messages
9
A common issue on TrueNAS 12, probably related to collectd 5.11 or some middlewared issue, seems to not affect arc or other services, but that wrong reporting can lead to other issues.
Tried to apply the patch related to NAS-111093 but no effect.

There's some memory issues on collectd repository, some seems to be fixed on next major release (collectd 6.0), i don't know if any of them is related to this issue.
 
Top