Middlewared having a memory leak in 22.02.0?

[Whiskydrinker]

The question is how much memory should the middlewared normally consume. Because right now it feels like it's leaking memory to me.

My server has 16 gigs of RAM and currently runs TrueNAS-SCALE-22.02.0.

As I'm writing this my current uptime is 1 day 15:42. Before this reboot my uptime was pretty close to 16 days. And at the end of it middlewared was using close to 35% of the memory in htop and the system was telling me that it was running out of memory. So I rebooted it.

Now after the reboot and when the whole system had settled in after a few minutes middlewared was using 6.3% of memory in htop. Right now we are yet again at 10.2%. This looks like the middlewared's memory consumption is growing by approximately 0.1% per hour uptime. This growth also fits nicely with the last uptime's memory consumption of middlewared at the end.

So can somebody else here check his memory consumption of middlewared using htop and look after how much uptime how much memory is being used by it?

 

[mjtbrady]

I haven't dug into it yet, but I observed what I think is the same behavior yesterday.

Can't remember exactly what the up time was (something like two weeks), but according to htop middlewared was consuming 65% of RAM and growing. top reported the process as asyncio_loop, which I assume is a middlewared thread. This is on a test system that is doing nothing. No users, no shares, no apps. no VMs.

This system only has 8GB of RAM so things were starting to fail. A reboot took about 15 minutes to complete. I assume due to the lack of RAM.

The system has now been up for 24 hours and middlewared/asyncio_loop is now consuming 20.6% of RAM and growing.

 

[Whiskydrinker]

Yes, asyncio_loop is part of middlewared. With your system having 8 gigs of RAM 65% of memory consumption after around two weeks of uptime is in the same ballpark as 35% on mine with 16 gigs of RAM.

Since my posting from yesterday my middlewared has grown to 12.5% of memory right now.

 

[xygnal]

I'm on the same version, having the same problem.

2577 root 20 0 7524600 4.0g 9016 S 0.7 25.5 1056:28 asyncio_loop

Rebooting, or directly killing the process, drops the memory usage until it has time to climb again.

 

[sretalla]

Consuming memory isn't necessarily leaking it. Processes that cache or build indexes can certainly show linear growth in some circumstances and not be leaky.

I don't run asyncio on my SCALE systems, so can't say anything other than the middleware takes a good chunk of memory on a system that's been running for 26 days which runs containers (about 16%), which is almost identical to another which has been up for 27 days and runs VMs. Both have 64GB. Also identical (in GB, not %) to another that has 32GB and runs nothing but shares that aren't accessed.

It's certainly worth a ticket if somebody wants to upload their dumps and ask iX to look at it.

 

[anodos]

Consuming memory isn't necessarily leaking it. Processes that cache or build indexes can certainly show linear growth in some circumstances and not be leaky.

I don't run asyncio on my SCALE systems, so can't say anything other than the middleware takes a good chunk of memory on a system that's been running for 26 days which runs containers (about 16%), which is almost identical to another which has been up for 27 days and runs VMs. Both have 64GB. Also identical (in GB, not %) to another that has 32GB and runs nothing but shares that aren't accessed.

It's certainly worth a ticket if somebody wants to upload their dumps and ask iX to look at it.

Maybe worth checking on master to see if it's still there first. There were some memory leak fixes that have gone in after 22.02 IIRC.

 

[xygnal]

Consuming memory isn't necessarily leaking it. Processes that cache or build indexes can certainly show linear growth in some circumstances and not be leaky.

I don't run asyncio on my SCALE systems, so can't say anything other than the middleware takes a good chunk of memory on a system that's been running for 26 days which runs containers (about 16%), which is almost identical to another which has been up for 27 days and runs VMs. Both have 64GB. Also identical (in GB, not %) to another that has 32GB and runs nothing but shares that aren't accessed.

It's certainly worth a ticket if somebody wants to upload their dumps and ask iX to look at it.

I didn't know you could run without asyncio, or are you saying you run without middlewared and what it provides all together?


--

For anyone else having this issue, have you expanded beyond the stock application catalogs?

 

[sretalla]

I didn't know you could run without asyncio, or are you saying you run without middlewared and what it provides all together?

I don't see any process in the list named asyncio_loop. I see plenty named middlewared.

I haven't done anything to enable asyncio, so unless it's somehow on by default, I guess it's not on.

 

[Ixian]

I don't see any process in the list named asyncio_loop. I see plenty named middlewared.

I haven't done anything to enable asyncio, so unless it's somehow on by default, I guess it's not on.

Asyncio isn't something you enable or disable. It's a master control, of sorts, for Python that provides concurrency for services among other things and it's heavily leveraged by Middlewared.

asyncio — Asynchronous I/O

Hello World!: asyncio is a library to write concurrent code using the async/await syntax. asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance n...

docs.python.org

High memory/CPU consumption by Asyncio is likely the symptom of another problem, not the root cause.

 

[xygnal]

I don't see any process in the list named asyncio_loop. I see plenty named middlewared.

I haven't done anything to enable asyncio, so unless it's somehow on by default, I guess it's not on.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2577 root 20 0 7588420 4.1g 8968 S 28.2 26.4 1132:47 asyncio_loop

root@truenas[/mnt/raidpool/freenas/docker/netdata]# ps -ef | grep 2577
root 2577 1 3 Feb26 ? 18:52:48 middlewared

 

[mjtbrady]

I have disabled Kubernetes by unsetting the pool for applications. Memory usage now appears to be stable with middlewared/asyncio_loop only consuming 3.8% of RAM on my test system (8G RAM) after a couple of days up time. So I am guessing that whatever the root cause is, it is in the Kubernetes set up.

Unfortunately, this and the issues caused by the number of ZFS snapshots make TrueNAS Scale completely unusable for me. Moving onto to looking at other solutions now.

 

[xygnal]

I expect it's probably the catalogging that does it. That is why I wondered how many people with this problem had added extra catalogs on top of the base.

 

[mjtbrady]

I had the Truecharts catalog added.

 

[truecharts]

We cannot reproduce this behavior, but even so it is not the catalog that is causing it but rather the parser inside SCALE that does so ;-)
(the file-reader vs the file itself, basically)

 

[Whiskydrinker]

I have also Truecharts added as a catalog.

BTW my current uptime is a little more than 14.5 days and middlewared currently consumes about 4.5 Gigabytes of RAM and growing.

 

[anyuser]

I also had Truecharts installed and saw this memory behaviour. I have since done a fresh install of SCALE and have not added any app catalogs and services memory is stable at approx 2.5GB so far

 

[helbo15]

I'm fairly new to Truenass.

I have installed version TrueNAS-22.02.0.
The only services I've enabled is S.M.A.R.T. and SMB.
I've got one VM running still trying to get a 2nd up but my own misconfiguration in that one means I keep turning it off, deleting it and making a new one.
Anyway I've had this running now for about 19 days.
in that time I've seen a steady increase in use of memory usage from services.
I've got a very small home setup with:
4TB hdd's x2
32GB DDR4
ryzen 5 3400G

essentially all I want from it is a samba share and a few linux VM's
But the services memory keeps rising up everytime I do anything.
I've lost 1 GB ram to Services in just the last half an hour due to me creating a VM and Deleting the VM to recreate it again.

Considering to issue a reboot now just to get the memory consumption down again but it is not really a viable long term solution.

Anyone knows what I can do to change this behaviour ?

 

[LarsR]

I've lost 1 GB ram to Services in just the last half an hour due to me creating a VM and Deleting the VM to recreate it again

You do realize that the memory for your vm is used by a system process and therefore is counted as system memory.
If you shutdown your vm the memory will be freed

 

[helbo15]

You do realize that the memory for your vm is used by a system process and therefore is counted as system memory.
If you shutdown your vm the memory will be freed

but that should still not result in me losing an extra GB of memory every time I turn off a VM, Delete Said VM. and create a new Identical VM. then start it up. only to see that suddenly I've lost 1 more GB memory than was used when the first VM ran...

 

[pabloalcantara]

I´m having the same memory leak on truenas.
I resolved the hard (and bad) way. every couple of day, I have an cronjob that restart the Middlewared