What is Services memory?

[winnielinnie]

I assume it's due to running it in paranoid mode which invokes this function for each deletion.

Whoa! I can imagine how much RAM is being used if you're using "cmp" on multi-gigabyte files.

That's a really wasteful use of RAM to simply figure out which files are safe to delete.

Is there any way you can change your process for managing duplicate files, or perhaps forgo the "paranoid" option?

EDIT: My guess is that rmlint's script (specifically the use of "cmp" on multi-gigabyte files), requires a lot of RAM, which shoves away much ARC (as ZFS does this by design).

Then to further add confusion, the graphical dashboard is either not correctly denoting "free" memory, or perhaps the data used by "cmp" is still lingering in RAM as a non-ARC buffer/cache.

 

[CJRoss]

Whoa! I can imagine how much RAM is being used if you're using "cmp" on multi-gigabyte files.

That's a really wasteful use of RAM to simply figure out which files are safe to delete.

Is there any way you can change your process for managing duplicate files, or perhaps forgo the "paranoid" option?

EDIT: My guess is that rmlint's script (specifically the use of "cmp" on multi-gigabyte files), requires a lot of RAM, which shoves away much ARC (as ZFS does this by design).

Then to further add confusion, the graphical dashboard is either not correctly denoting "free" memory, or perhaps the data used by "cmp" is still lingering in RAM as a non-ARC buffer/cache.

I'd really rather not change the workflow. As I mentioned, I'm not too concerned about the amount of ram it's using. My issue is that the ARC doesn't seem to recover and stays in the squashed state until a reboot. It almost seems like the ARC would rather evict things than try and use any Inactive memory.

Any suggestions on how to clear any non-ARC buffer/cache?

 

[CJRoss]

Some interesting details in this thread. It looks like any attempt to unmount the partition will immediately free up the memory. The unmount does not have to succeed.

Solved - FreeBSD Inactive memory

Hello, Out of 10 machines one of the machine 90% memory is being put into the Inactive memory. Lot of analysis done still we are not able to solve the issue without rebooting the OS. Someone please help me to fix the issue. Mem: 78M Active, 1503M Inact, 160M Wired, 112M Buf, 260M Free Thanks...

forums.freebsd.org

 

[winnielinnie]

Any suggestions on how to clear any non-ARC buffer/cache?

I don't think it's as straight-forward in FreeBSD as it is on Linux. (The FreeBSD gurus in here might know more about this.)

However, this doesn't seem like an ideal approach. Let's say you clear the cache used by "cmp".

Let's say this is regarded as truly free/unused RAM.

Let's say you begin to use the server and it fills up the ARC, eating into this free'd up RAM.

The next time you run the rmlint script, it's all undone. Meaning that leaving the "cmp" buffer is preferable to clearing it for the sake of clearing it, regardless of whether or not the jail is currently running or is restarted.

 

[CJRoss]

I don't think it's as straight-forward in FreeBSD as it is on Linux. (The FreeBSD gurus in here might know more about this.)

However, this doesn't seem like an ideal approach. Let's say you clear the cache used by "cmp".

Let's say this is regarded as truly free/unused RAM.

Let's say you begin to use the server and it fills up the ARC, eating into this free'd up RAM.

The next time you run the rmlint script, it's all undone. Meaning that leaving the "cmp" buffer is preferable to clearing it for the sake of clearing it, regardless of whether or not the jail is currently running or is restarted.

The next time cmp gets called, it would be for different data as the original files that were compared had their duplicates deleted. Therefore it's not really useful to keep that cmp data in memory compared to ARC data.

Also, the Inactive data doesn't seem to ever get cleared from what I can tell. For whatever reason, ARC just doesn't put enough pressure on it to force Inactive memory to be freed. If I run rmlint on different files, it just keeps adding to the Inactive total.

 

[awasb]

I'm in no way a guru, but over the years _using_ FreeBSD I came to the impression, that inactive mem gets freed when needed. As long as the system won't swap, I'd still say that it's a problem of colorization.

 

[CJRoss]

I'm in no way a guru, but over the years _using_ FreeBSD I came to the impression, that inactive mem gets freed when needed.

The problem is that the ARC doesn't appear to be a valid need in the eyes of the system.

 

[awasb]

Ad hoc. Yes. I‘d wait a bit. (And check mounts.)

 

[CJRoss]

Ad hoc. Yes. I‘d wait a bit. (And check mounts.)

I'm not sure what you mean by this.

 

[awasb]

Did you try to u(n)mount the zfs data shares in the jail before suspending?

 

[winnielinnie]

Did you try to u(n)mount the zfs data shares in the jail before suspending?

But the real datasets will still be mounted in the server, regardless of the jail's status or mountpoints.

 

[awasb]

Yes. For sure. But "what if" the ARC cache for the jail mount won't get freed/allocated to the "main"/host ARC? I don't quite remember, but weren't there some options, to control ZFS/ARC from inside a jail vs. host managed ARC?

 

[CJRoss]

Did you try to u(n)mount the zfs data shares in the jail before suspending?

Are you talking about running umount or editing the config? I just tried running umount and got operation not permitted and nothing changed memory wise.

 

[CJRoss]

A plex jail schedule task and a syncthing sync both kicked off but ARC is still struggling.

Code:

ARC size (current):                                    16.5 %   10.3 GiB
        Target size (adaptive):                        16.6 %   10.4 GiB
        Min size (hard limit):                          3.2 %    2.0 GiB
        Max size (high water):                           31:1   62.8 GiB
        Most Frequently Used (MFU) cache size:         84.2 %    7.5 GiB
        Most Recently Used (MRU) cache size:           15.8 %    1.4 GiB
        Metadata cache size (hard limit):              75.0 %   47.1 GiB
        Metadata cache size (current):                  6.5 %    3.1 GiB
        Dnode cache size (hard limit):                 10.0 %    4.7 GiB
        Dnode cache size (current):                    14.9 %  717.5 MiB

 

[awasb]

Can you u(n)mount from the host? Something like

Code:

zfs unmount /mnt/data/jail-mounts/zfsmount

Please check the mounts afterwards.

Btw. for checking/observing other resources:

GitHub - ozgurkazancci/jstats: A small tool for FreeBSD systems - lists RAM, CPU and disk space usage of the jails running in the host system.

A small tool for FreeBSD systems - lists RAM, CPU and disk space usage of the jails running in the host system. - ozgurkazancci/jstats

github.com

 

[CJRoss]

Can you u(n)mount from the host? Something like

Code:

zfs unmount /mnt/data/jail-mounts/zfsmount

Please check the mounts afterwards.

That kicked back "not a mount point". Not sure what you mean by checking the mounts.

Btw. for checking/observing other resources:

GitHub - ozgurkazancci/jstats: A small tool for FreeBSD systems - lists RAM, CPU and disk space usage of the jails running in the host system.

A small tool for FreeBSD systems - lists RAM, CPU and disk space usage of the jails running in the host system. - ozgurkazancci/jstats

github.com

That doesn't show any of the jails that aren't running.

 

[awasb]

No, but it will show the exact mem while running.

Cocerning "check mounts" ... something like

Code:

mount | grep jail-mount

If there is nothing after shutting down the jail, I'd wait some time[*] to see whether the RAM gets freed or not.

[*] Rather days than hours, depends on the "traffic".

 

[CJRoss]

There's no jail mount for that dataset. The only thing for that jail is the base jail dirs.

That said, it's not looking good for my ARC.

Code:

ARC size (current):                                    10.4 %    6.5 GiB
        Target size (adaptive):                        13.1 %    8.2 GiB
        Min size (hard limit):                          3.2 %    2.0 GiB
        Max size (high water):                           31:1   62.8 GiB
        Most Frequently Used (MFU) cache size:         48.7 %    2.4 GiB
        Most Recently Used (MRU) cache size:           51.3 %    2.6 GiB
        Metadata cache size (hard limit):              75.0 %   47.1 GiB
        Metadata cache size (current):                  6.4 %    3.0 GiB
        Dnode cache size (hard limit):                 10.0 %    4.7 GiB
        Dnode cache size (current):                    16.9 %  817.0 MiB

 

[winnielinnie]

There's no jail mount for that dataset. The only thing for that jail is the base jail dirs.

You're saving / managing your data inside the jail's "root" dataset? (That is, within the iocage/jails/jailname/root dataset)

 

[CJRoss]

You're saving / managing your data inside the jail's "root" dataset? (That is, within the iocage/jails/jailname/root dataset)

There's no data to rmlint.

The jail is configured to mount the datasets I want to clean, but when the jail isn't running, the datasets aren't mounted. To clarify, I'm not editing the config. There's just nothing mounted other than the base jail dirs when it's not running.