Scale memory utilization

[rmont]

I upgraded to SCALE recently and I'm curious about the memory utilization.

My previous CORE server was allocating most memory to ZFS cache, while SCALE is mostly using memory for apps.

This looks more reasonable to me because the server is mostly running apps, but I remember reading that ZFS likes to have a lot of memory allocated to cache.
So now I'm curious which server is doing the right thing...

 

[morganL]

ZFS cache generally takes whatever RAM is left over.
What services are you running? That determines what RAM is used...

 

[NugentS]

I think you could do with some more memory, poor ZFS is being starved

 

[Xinefus]

ZFS cache generally takes whatever RAM is left over.
What services are you running? That determines what RAM is used...

When you say whatever RAM is left over.... How exactly is that calculated?

I'm running into the same issue here, however I only have 1 service currently running Plex through Apps. My memory allocation looks very similar to OP... Is this something to be expected? I though that I would have enough RAM to do at least a couple of things at once...

 

[morganL]

When you say whatever RAM is left over.... How exactly is that calculated?

I'm running into the same issue here, however I only have 1 service currently running Plex through Apps. My memory allocation looks very similar to OP... Is this something to be expected? I though that I would have enough RAM to do at least a couple of things at once...

View attachment 54046

You have 6.7GB allocated to ZFS... if you add more services, that ZFS RAM will shrink. If it gets too low, performance will reduce, but you will see that.

 

[Xinefus]

Thanks for the quick reply! Really appreciate the explanation.

 

[artlessknave]

When you say whatever RAM is left over.... How exactly is that calculated?

its calculated by ZFS using whatever RAM isn't in use. it's not any more complex than that from a user/admin perspective. if anything else asks for the RAM, ZFS will free some up for it.
I agree on the "need more RAM" part though. if anything happens and your pool needs to eat RAM to survive (yes, it does this when things go really wrong), it will try and use disk swap, but that's slow as hell, because that swap is usually on the same disks as the pool that it's trying to rescue.

 

[jlw52761]

its calculated by ZFS using whatever RAM isn't in use. it's not any more complex than that from a user/admin perspective. if anything else asks for the RAM, ZFS will free some up for it.
I agree on the "need more RAM" part though. if anything happens and your pool needs to eat RAM to survive (yes, it does this when things go really wrong), it will try and use disk swap, but that's slow as hell, because that swap is usually on the same disks as the pool that it's trying to rescue.

I have to disagree only due to observation. I have a system with 128GB RAM, with a couple of VMs running, and my ZFS Cache ranges from 20GB to 100GB, with it changing on the fly. I have between 55GB and 10GB free RAM at all times, so ZFS isn't taking all available RAM under SCALE, it seems to be dynamic based on some algorithm that is in flux.

Here's the point-in-time, and below, you can see where the ARC grew while I was moving a lot of data over, then it went down around 28-Feb due to a reboot, but since then the system hasn't been rebooted and you can see the ARC grow, then reduce itself for some reason. I haven't removed any load from this NAS, it's been constant, so I don't know why SCALE reduced the ARC size. Since I don't have L2ARC, the tracking or ARC to ZFS Cache on the dashboard is 1:1.

So just based on observation, wither it's the port of OpenZFS to Debian, or some other difference, SCALE definitely dynamically adjusts the ARC up and down, and doesn't always use all available RAM. I personally like this, in a way, but would be interested to understand the algorithm behind it.

 

[artlessknave]

its calculated by ZFS using whatever RAM isn't in use. it's not any more complex than that from a user/admin perspective. if anything else asks for the RAM, ZFS will free some up for it.

I have to disagree only due to observation.

fair enough, that was an oversimplification. from our perspective, as users, ZFS RAM usage is intended to be transparent and automatic.

zfs will dynamically use whatever RAM is available to try and intelligently cache files that are accessed (ARC) and writes to pools (collected into transaction groups and flushed to disk in regular intervals, usually like 5 seconds, as contiguous writes when possible).

if you have enough RAM to cache everything that is accessed reasonably frequently, then it will not use all the RAM, but whatever is most useful. I believe this is what you're seeing.

if, however, you are pounding the server with enterprise usage, like a massive database or huge VM stores, then you would see different behavior with ZFS stretching it's algorithmic muscles to get the most reliable performance from what it was fed (hardware).

 

[morganL]

I have to disagree only due to observation. I have a system with 128GB RAM, with a couple of VMs running, and my ZFS Cache ranges from 20GB to 100GB, with it changing on the fly. I have between 55GB and 10GB free RAM at all times, so ZFS isn't taking all available RAM under SCALE, it seems to be dynamic based on some algorithm that is in flux.

View attachment 54116

Here's the point-in-time, and below, you can see where the ARC grew while I was moving a lot of data over, then it went down around 28-Feb due to a reboot, but since then the system hasn't been rebooted and you can see the ARC grow, then reduce itself for some reason. I haven't removed any load from this NAS, it's been constant, so I don't know why SCALE reduced the ARC size. Since I don't have L2ARC, the tracking or ARC to ZFS Cache on the dashboard is 1:1.

View attachment 54117


So just based on observation, wither it's the port of OpenZFS to Debian, or some other difference, SCALE definitely dynamically adjusts the ARC up and down, and doesn't always use all available RAM. I personally like this, in a way, but would be interested to understand the algorithm behind it.

The algorithm in Linux/SCALE is more complex than FreeBSD
It will get more tuning now that we have got to the RELEASE stage.

 

[jlw52761]

The algorithm in Linux/SCALE is more complex than FreeBSD
It will get more tuning now that we have got to the RELEASE stage.

That does explain the observation then, especially when comparing the other box not running SCALE using a fairly fixed amount of RAM all the time.
I do have to say my ARC hit rate is 99%, rarely dipping below that to 98%, so more than enough there. I wonder what the OP is seeing for hit/miss rate on their ARC. That's what I always understand to be the true indicator if more RAM is needed for ARC or if L2ARC is needed.

 

[artlessknave]

true indicator if more RAM is needed for ARC or if L2ARC is needed

yup. if you have 99% ARC hits, then L2ARC is just going to use RAM for no reason.
you can never have too much RAM, really, particularly because if something DOES go wrong with a pool, sometimes zfs needs a crapton of RAM to fix it fast (this is one of the reasons there is 2GB of swap created per disk, so that if something goes sour, and you dont have sufficient RAM, there is at least some swap to work with, but swapping on the disks ZFS is trying to rescue is very much not ideal.)

 

[mrcbns]

I think too services RAM usage seems buggy. I have 2 mirrored 5Tb disks with 8Gb of RAM. I deployed a minio truechart for store datasets mostly. Over some uploads for like 200mb of small files dataset (~15min) services RAM is constantly growing up for no reason, then hors after there is still only 0.2Gb of ram available. To me there is no reason for this. I disable all running truechart apps (minio) so services RAM is not going down enough. It frees 0.5 Gb only. I check swap and It used ~2MB. After I reboot everything goes normal again as seen on this graph:

How is a minio server for an upload of 200Mb dataset eats 3Gb of RAM forever and most important, Who is taking 4Gb of ram for running.

I am unable to track who is taking my RAM :(

But it feels unstable.

 

[Mathew]

I have a few apps running and after a few weeks I noticed that I always had a bunch of memory free so to not be so wasteful, I set my ARC to 96GB. ZFS seems much happier since doing so.

I'd post a screenshot if I could but I seem to not have that option.

 

[sretalla]

I am unable to track who is taking my RAM :(

htop

 

[Kasazn]

Just want to chip in, previous SCALE installations did not have issues like these. I am experiencing what OP is experiencing right now. Today I had to restart my server after a week and the usage is all taken up by services.

Build is on my signature.

 

[anodos]

Just want to chip in, previous SCALE installations did not have issues like these. I am experiencing what OP is experiencing right now. Today I had to restart my server after a week and the usage is all taken up by services.

Build is on my signature.

NAS-116105 / 22.12 / Memory improvements for catalogs by sonicaj · Pull Request #9137 · truenas/middleware

This PR adds following changes: Do not cache entire contents of catalogs. Just caching names of the apps is sufficient with some other metadata. Remove retrieving versions from catalog.items as it...

github.com

We've just recently fixed an issue with memory usage in charts that could be significantly exacerbated by third-party repos.

 

[neofusion]

That's great!
There is more to it since I've never setup much less used charts on my system and it still sees Services memory usage grow without end. There might be some correlation to having the UI open vs. just leaving it closed, I am not sure.

 

[anodos]

That's great!
There is more to it since I've never setup much less used charts on my system and it still sees Services memory usage grow without end. There might be some correlation to having the UI open vs. just leaving it closed, I am not sure.

There was an additional fix related to memory usage here: https://github.com/truenas/middleware/pull/9125

TL;DR you can maybe take a nightly for a spin and see if usage is stabilized. Otherwise wait for 22.02.2. Above fix (unlike charts one) shouldn't be hard to hot-patch.

 

[neofusion]

There was an additional fix related to memory usage here: https://github.com/truenas/middleware/pull/9125

TL;DR you can maybe take a nightly for a spin and see if usage is stabilized. Otherwise wait for 22.02.2. Above fix (unlike charts one) shouldn't be hard to hot-patch.

That shows promise, I'll try it out in 22.02.2.