Scale memory utilization

[MountainMan]

There was an additional fix related to memory usage here: https://github.com/truenas/middleware/pull/9125

TL;DR you can maybe take a nightly for a spin and see if usage is stabilized. Otherwise wait for 22.02.2. Above fix (unlike charts one) shouldn't be hard to hot-patch.

Will this fix (#9125) be backported to 22.02.2 ?

I see #9137 was, but #9125 only seems to show 22.12.

 

[anodos]

9125 is against the stable/angelfish branch. It's already merged for 22.02.2.

 

[inman.turbo]

instead of scaling back ARC when a vm started which needs some of that memory, scale will just warn you, then crash your system (out of memory)

 

[morganL]

instead of scaling back ARC when a vm started which needs some of that memory, scale will just warn you, then crash your system (out of memory)

Please "report-a-bug".... we are looking at several non-optima behaviours with ZFS and Linux. ARC management can be improved.

 

[Daisuke]

ZFS cache generally takes whatever RAM is left over.

I always get about 100G free, next time I would probably not use that much RAM. I added more, for VMs.

 

[morganL]

I always get about 100G free, next time I would probably not use that much RAM. I added more, for VMs.

View attachment 60723 View attachment 60722

I've since learned that ZFS with Linux gets less RAM to use as ARC. The operating system keeps more RAM for its own use and is not as dynamic as FreeBSD. We're reviewing settings/changes to improve this.

 

[kiler129]

I've since learned that ZFS with Linux gets less RAM to use as ARC. The operating system keeps more RAM for its own use and is not as dynamic as FreeBSD. We're reviewing settings/changes to improve this.

Is there any tunable for that? I see a similar behavior as @Daisuke - if I give the VM 100GB ram it will use ~50 for ARC, if I give it 64GB it will not go over ~32GB for ARC.

 

[Fiberton]

Is there any tunable for that? I see a similar behavior as @Daisuke - if I give the VM 100GB ram it will use ~50 for ARC, if I give it 64GB it will not go over ~32GB for ARC.

I have 512 gigs of ram and the arc does as per normal consuming 252 gigs of ram. Would like to bump that to say 400. I would love to have a tuneable to set the arc size aswell.

 

[emsicz]

On my system this is still the case. I'm running with 6x16 ECC memory and I have ~ 10 VMs running. Usually it's something like 45 gigs of ZFS cache and 45 of Services (Vritualization shows 0.00 available), according to dashboard. When I need to spin up additional VM, I get the prompt to confirm overcommit (the prompt asks me if I understand the danger without really saying anything as to what the danger is). If I do that, the VM spins up just fine and in the dashboard, I can see the ZFS cache reducing itself slowly until I get about 10 gigs of unused RAM (the Virtualization tab then suddenly stops showing 0.00 memory available and shows something like 10 gigs available).

The behavior isn't very intuitive, but stability-wise I've never had a single problem with this particular thing.

 

[morganL]

On my system this is still the case. I'm running with 6x16 ECC memory and I have ~ 10 VMs running. Usually it's something like 45 gigs of ZFS cache and 45 of Services (Vritualization shows 0.00 available), according to dashboard. When I need to spin up additional VM, I get the prompt to confirm overcommit (the prompt asks me if I understand the danger without really saying anything as to what the danger is). If I do that, the VM spins up just fine and in the dashboard, I can see the ZFS cache reducing itself slowly until I get about 10 gigs of unused RAM (the Virtualization tab then suddenly stops showing 0.00 memory available and shows something like 10 gigs available).

The behavior isn't very intuitive, but stability-wise I've never had a single problem with this particular thing.

Your system is pretty balanced..... it's the storage-only use-cases that are impacted.

 

[im.thatoneguy]

I've since learned that ZFS with Linux gets less RAM to use as ARC. The operating system keeps more RAM for its own use and is not as dynamic as FreeBSD. We're reviewing settings/changes to improve this.

Is it safe to tune it up if you have lots of RAM (256GB) and no VMs or large apps? Having 128GB of RAM sitting idle is a lot of hardware doing nothing in a storage server.

 

[artlessknave]

Is it safe to tune it up if you have lots of RAM (256GB)

I'm reasonably sure there is currently no way to do this on SCALE

 

[NugentS]

Actually you can - just can't remember how. However its get reset at almost any opportunity, such as starting a VM etc

 

[john60]

I have 128G of memory.
http://192.168.1.80/reportsdashboard/memory reports on 35G

While the dashboard reports

How do these correlate?

What is consuming the Services? I has a vm (ubuntu) and a docker (mysql) but both are deleted.

Code:

root@truenas[~]# cat /proc/meminfo
MemTotal:       131892916 kB
MemFree:         4784196 kB
MemAvailable:    8251392 kB
Buffers:            5636 kB

Code:

root@truenas[~]# free
               total        used        free      shared  buff/cache   available
Mem:       131892916   122236024     4841248       64840     4815644     8284868
Swap:        4190072        2840     4187232

Are memory reports accurate in TrueNAS-SCALE-22.12.0 ?

 

[sammael]

I'm just a homelab enthusiast without any professional connection to IT so I might be just misunderstanding some basic concept, but ever since I upgraded to SCALE on 64GB ram system, there has never been less than 16GB of free ram regardless of how many apps or vm's are running and the ARC never goes above 32GB.

I might not even need it in my use case, but I have to wonder why there needs to be 16GB of ram just sitting idly all the time? It doesn't seem to be utilized by the linux os underneath either by what `free -h says`. I'm just at a loss why the overwhelming advice on this everywhere is "just leave it be" despite there being methods of increasing it, but then why did I buy 4 16GB sticks of ram if the system consistenly leaves 1 free ? (I know that's not exactly what is happening but you get my drift)

 

[NugentS]

Its annoying but ZFS on Linux (aka Scale) will only use 50% of available memory for ARC.

I am hoping that IX will work some wonder and improve that

 

[dir_d]

Any updates on this? I switched from Core to Scale and half my Ram is not used.

 

[NugentS]

Its a feature. ZFS on Debian only uses 50% at the moment

 

[Etorix]

Any updates on this? I switched from Core to Scale and half my Ram is not used.

What's your uptime? It will take time, and use, to populate >200 GB of ARC…

 

[john60]

What's your uptime? It will take time, and use, to populate >200 GB of ARC…

8 days, and memory

I still am curious about services memory. What is this? I have no vm or docker apps.