what is consuming 62,8BiB of services memory

[john60]

Only have one VM, but it is stopped

 

[Whattteva]

Are you using CORE or SCALE?

Also, try to run htop and hit M once it launches and you can see what is taking up a lot of RAM.

 

[john60]

There is more than 50 or so /k3s server.

then a boatload of middlewared

then a boat load of dockerd

everything else shows 0.0%

Note: I even deleted the VM and the AP, so nothing but truenas.

It looks like something is leaking memory.

Code:

root@truenas[~]# cat /proc/meminfo
MemTotal:       131892916 kB
MemFree:         4764708 kB
MemAvailable:    8195764 kB
Buffers:            5636 kB
Cached:           325924 kB
SwapCached:          948 kB
Active:           182948 kB
Inactive:        2959396 kB
Active(anon):      80196 kB
Inactive(anon):  2773548 kB
Active(file):     102752 kB
Inactive(file):   185848 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4190072 kB
SwapFree:        4186976 kB
Dirty:              3684 kB
Writeback:             4 kB
AnonPages:       2639300 kB
Mapped:           296668 kB
Shmem:             64832 kB
KReclaimable:    4442724 kB
Slab:           117233012 kB
SReclaimable:    4442724 kB
SUnreclaim:     112790288 kB
KernelStack:       26684 kB
PageTables:        20368 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    70136528 kB
Committed_AS:    7628892 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     2709876 kB
VmallocChunk:          0 kB
Percpu:            64896 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1353728 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:         262144 kB
CmaFree:          117456 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1118860 kB
DirectMap2M:    84748288 kB
DirectMap1G:    50331648 kB
root@truenas[~]# 5~

Code:

root@truenas[~]# free
               total        used        free      shared  buff/cache   available
Mem:       131892916   122465812     4638372       64848     4788732     8078148
Swap:        4190072        3096     4186976
root@truenas[~]# 

 

[Whattteva]

Ah yes... This is the reason I don't use SCALE. It's bloated and I consider it BETA software.

To your question, k3s is the kubernetes process for the TrueCharts apps hosting.

 

[Whattteva]

Code:

root@truenas[~]# free
               total        used        free      shared  buff/cache   available
Mem:       131892916   122465812     4638372       64848     4788732     8078148
Swap:        4190072        3096     4186976
root@truenas[~]#

You should run free -h instead so it's more readable.

 

[john60]

Code:

root@truenas[~]# free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi       116Gi       4.4Gi        63Mi       4.6Gi       7.7Gi
Swap:          4.0Gi       3.0Mi       4.0Gi
root@truenas[~]# 

 

[Whattteva]

How many TrueCharts apps are you running?

 

[john60]

Do not know how to configure true charts?
So what ever is configured by default.

 

[john60]

Just watched a video of truecharts. Never used this.

Only default applications, and they are not used.

I had one vm, but deleted it.
I also had one custom app mySQL, but deleted it too.

So plain NAS with NFS,SMB, automated replication (not running at the moment), automated snapshots (not running at the moment), cloud sync tasks (not running at the moment), and automated smart test (not running at the moment).

Looks like a memory leak.

 

[Whattteva]

Wow, you have no apps and it's running all those k3s processes? Definitely a bug.

 

[Whattteva]

Just watched a video of truecharts. Never used this.

Only default applications, and they are not used.

I had one vm, but deleted it.
I also had one custom app mySQL, but deleted it too.

So plain NAS with NFS,SMB, automated replication (not running at the moment), automated snapshots (not running at the moment), cloud sync tasks (not running at the moment), and automated smart test (not running at the moment).

Looks like a memory leak.

Is it doing this still even after a reboot?

 

[john60]

A reboot is my next step. Waiting for a window. Maybe you were correct in waiting on that scale to mature.

 

[Whattteva]

A reboot is my next step. Waiting for a window. Maybe you were correct in waiting on that scale to mature.

I myself experienced a similar issue with usage, but on the CPU side. k3s was constantly hovering at 10% CPU cycle even when it's literally doing nothing (no clients connected) and no real production data. I saw that and said "Nope, not ready for prime time yet". 10% is absolutely bonkers since my CORE install idles at under 1%. Moreover, this issue persists even across reboots.

I'm not alone in this as other people have confirmed here and here. There is even a GitHub issue for it here, though it was closed as they saw it as non-issue apparently and it's supposed to idle that high. That just seems crazy to me that it's using that much idle out of 2 cores of Xeon Silver.

 

[winnielinnie]

@Whattteva, no offense, but I think you're being a pessimist. There are so many things you can do with excess heat generated by a CPU that never truly idles.

• Cook eggs. Just place a frying pan over the exhaust fan on the top of the chassis.
• Heat up the room. A great way to subsidize your HVAC during the winter months.
• Dry your hands. No need to waste paper towels. The warm air will suffice.

Unfortunately, Core denies us of these features, with such nonsense of idling at 0-1% during no activity.

 

[Whattteva]

@Whattteva, no offense, but I think you're being a pessimist. There are so many things you can do with excess heat generated by a CPU that never truly idles.

• Cook eggs. Just place a frying pan over the exhaust fan on the top of the chassis.
• Heat up the room. A great way to subsidize your HVAC during the winter months.
• Dry your hands. No need to waste paper towels. The warm air will suffice.

Unfortunately, Core denies us of these features, with such nonsense of idling at 0-1% during no activity.

Wow, you are absolutely correct. I really should start thinking outside the box. When you look at it that way, I should really appreciate the versatility it affords me.