Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

High "Services" memory consumption

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

jenksdrummer

Member
Joined
Jun 7, 2011
Messages
199
1620918790936.png


Image says it all. Doing some testing with HCIBench (VMWare ova) to determine what kinds of performance I can expect before taking this box to a temporary production (subplanting a Nimble Array during a migration). System is running 12U3.1. Was running with dedupe vdev (NVME mirror), Meta VDEV (SATA SSD mirror), and 12x 10TB Z2 array. This is after testing it and ditching the pool and building it fresh without dedupe; suspecting it could be related, but that image is *after* the pool nuke and rebuild (NVMe as Meta VDEV, no dedupe, omit SATA SSD); so thought is that if that was in any way related to dedupe that it would have flushed that out as well at that time.

Top:
1620918987432.png


Only "Services" running, according to the GUI at least is iSCSI and SMART; flushed both and still there. Rebooting it seems to resolve, but concerned that putting this into production I'd see this behavior occur and rebooting won't be an option.
 
Last edited:

sretalla

Wizened Sage
Joined
Jan 1, 2016
Messages
4,270
You can see it better with htop, but I expect that it's python3.8 and therefore the middlewared that's the culprit.

If you restart it (service middlewared restart) does that improve anything? (at least temporarily?)
 

jenksdrummer

Member
Joined
Jun 7, 2011
Messages
199
You can see it better with htop, but I expect that it's python3.8 and therefore the middlewared that's the culprit.

If you restart it (service middlewared restart) does that improve anything? (at least temporarily?)
Apologies; box is offline and getting packed up soon. I'll keep these suggestions in mind if I see the issue persisting once deployed again.

Odd thing is that this is a clean install with only iSCSI enabled, so there's not a lot of venues for which middleware would be giving the issue.
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
Just to keep the thread going : I have the same issue.
Restarting jails , services and middleware has no effect.
It takes about 10 days to look like what the OP has posted.....
 

ddaenen1

Member
Joined
Nov 25, 2019
Messages
216
Just to keep the thread going : I have the same issue.
Restarting jails , services and middleware has no effect.
It takes about 10 days to look like what the OP has posted.....
I have the same thing. Services keep creeping up after every reboot. Went all the way up to 17GiB middle of last month. Now already back up to 8GiB after 17 days uptime.

1621277383331.png
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
I just went through my memory reporting and can see that this issue did not occur until 12-U2.1 was install on the 24th February.
This is a home server in a relatively stable environment.
TrueNas Memory.png


The spikes are reboots primarily for firmware upgrades.
I have just rebooted into U1.1 and will monitor !
 
Last edited:

Mlovelace

Neophyte Sage
Joined
Aug 19, 2014
Messages
1,069
I just went through my memory reporting and can see that this issue did not occur until 12-U2.1 was install on the 24th February.
This is a home server in a relatively stable environment.
View attachment 47241

The spikes are reboots primarily for firmware upgrades.
I have just rebooted into U1.1 and will monitor !
Are you running any jails?
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
Are you running any jails?
HDHomerun Server
Mariadb 10.4.17 for Kodi
PY-Medusa
RslSync
Transmission 3
URBackup

BaseJails at 12.2-P4
Restarting jails makes no difference.

Services

SMB
NFS
SSH
Smart

Restarting services has no effect.
The last time I tried to restart MiddlewareD the server did not come back and I had to do a hardware reset.
 
Last edited:

Mlovelace

Neophyte Sage
Joined
Aug 19, 2014
Messages
1,069
HDHomerun Server
Mariadb 10.4.17 for Kodi
PY-Medusa
RslSync
Transmission 3
URBackup

BaseJails at 12.2-P4
Restarting jails makes no difference.
I was seeing something similar till I upgraded my jails to the latest (12.2-P6). Reboot and weeks later it hasn't reappeared.
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
I was seeing something similar till I upgraded my jails to the latest (12.2-P6). Reboot and weeks later it hasn't reappeared.
I'll let it run for a week or two at 12-U1.1 then I will try the jail upgrade.
 

jenksdrummer

Member
Joined
Jun 7, 2011
Messages
199
FWIW, on another box, I'm only running SMB/CIFS, same result.

Nuked the pool (pool previously had metadatavdevs and dedupe vdevs, going without this round), built a new pool.

1621602970366.png


All I've done since nuking the pool was build a new one and then create a datastore tree that I like to use for representing shares, and then create the shares. There's 6 shares.
Of note, before nuking hte pool, I did copy it all off to another freenas box via SMB (didn't want to do a zfs send/receive, also looking at changing dataset sizing, from what I understand, that doesn't change the dataset record size when using that method)

Only running service according to the UI is SMART and SMB; restarted SMB, no change.

1621603306443.png


1621603334165.png


Interesting, collectd may be a bit of a culprit.....and nopes. stop/start of collectd didn't make any difference.

Also killed/restarted middlewared and no change.

Worth noting, this is "wired" memory. My googlese mentions this is kernel memory; so it's likely there's a leak somewhere either in the kernel itself or in a module.
 
Last edited:

Sirius

Member
Joined
Mar 1, 2018
Messages
39
I'm also having similar issues. Only running SMB, SSH and SMART. No jails and no VMs.

I tried restarting middlewared and that didn't help.

Screen Shot 2021-05-25 at 4.32.42 pm.png
 

sretalla

Wizened Sage
Joined
Jan 1, 2016
Messages
4,270
I'm also having similar issues. Only running SMB, SSH and SMART. No jails and no VMs.
You're not having issues... SMB does caching of its own and is included in the services count. Your chart looks fine.
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
I'll let it run for a week or two at 12-U1.1 then I will try the jail upgrade.
So I have upgraded the jails to 12.2-P6 and gone to 12.0-U4 and the laundry still climbs. It seems to be at specific times but so far I have been unable to correlate that to network activity.
Screenshot (43).png
 
Last edited:

Mlovelace

Neophyte Sage
Joined
Aug 19, 2014
Messages
1,069
So I have upgraded the jails to 12.2-P6 and gone to 12.0-U4 and the laundry still climbs. It seems to be at specific times but so far I have been unable to correlate that to network activity.
View attachment 47680
Mine was doing the same after the 12.0-U4 update. Turns out there is a jail upgrade of 12.2-P8 that came with 12.0-U4. I applied the jail upgrade and it's back to normal.

Wish I could say the jail patch was the silver bullet that'll fix your problem, but I've seen a lot of these posts and they weren't all running jails. The 12.0-U4 was supposed to be the fix, so looks like the devs don't have this nailed down.
 

Alecmascot

Neophyte Sage
Joined
Mar 18, 2014
Messages
695
I've gone to 12.2-P8 and will see what happens.
 
Top