arc_prune high CPU usage

[Cellobita]

I have migrated one of my customers from Core 13.0-U6.1 to SCALE 23.10.2. Their usage is mainly a (very) old and cranky Btrieve-based accounting app; after the migration, they are experiencing intermitent, but frequent very high CPU usage from a process called arc_prune, which seriously impacts the server performance. This did not happen on Core.

The server has an i3-3250 CPU @ 3.50GHz, 16GB of RAM and SSD boot and data drives; there are no apps installed, just two plain SMB shares.

I have done some initial investigation and tried setting /sys/module/zfs/parameters/zfs_arc_max and /sys/module/zfs/parameters/zfs_arc_min to 6442450944 (6GB), after reading somewhere that "setting both to the same value effectively disables ARC prune". It did not work.

The server is consistently showing 2 to 3GB free memory.

Can anyone suggest additional tuning here? For budgetary reasons a hardware upgrade is not possible over the next several months.

 

[Elliot Dierksen]

I use CORE, i would say you don't have enough RAM. that system is old enough that you ought to be able to expand the memory pretty inexpensively.

 

[chuck32]

Where is this app running? Inside a VM? What is the memory allocation?

16 GB should be enough if it's just used as a file server for the app. My suspicion would be the app itself uses too much memory, hence arc needs to be pruned (in scale it uses up to 50 % by default) otherwise the system would run out of memory.

Probably more memory will solve this, but let's rule out everything else first.

 

[Cellobita]

Where is this app running? Inside a VM? What is the memory allocation?

16 GB should be enough if it's just used as a file server for the app. My suspicion would be the app itself uses too much memory, hence arc needs to be pruned (in scale it uses up to 50 % by default) otherwise the system would run out of memory.

Probably more memory will solve this, but let's rule out everything else first.

The app doesn't run on the server, it is a fileshare-based thing: the workstations run the executable from the server, and it then opens multiple DB files on the server with some kind of lock sharing mechanism. Think Access, with no SQL frontend - but older.

It did run without issues on Core.

 

[Cellobita]

I use CORE, i would say you don't have enough RAM. that system is old enough that you ought to be able to expand the memory pretty inexpensively.

Can't do, the mobo RAM capacity is maxed out as it is; however, the system consistently shows me more than 2GB RAM free. Is this indication unreliable?

 

[Davvo]

SCALE (up to dragonfish iirc) doesn't allow you to use more than 50% of the system's RAM for caching purposes. Might be related.

 

[Cellobita]

SCALE (up to dragonfish iirc) doesn't allow you to use more than 50% of the system's RAM for caching purposes. Might be related.

If I understand the /sys/module/zfs/parameters/zfs_arc options correctly, mine is limited to 6 out of 16GB, after having typed the following at the shell:

echo 6442450944 >> /sys/module/zfs/parameters/zfs_arc_max

echo 6442450944 >> /sys/module/zfs/parameters/zfs_arc_min

The dashboard cache amount is stable at 6GB, after issuing them, but I still get the arc_prune process at high CPU utilization. According to this it shouldn't be happening.

 

[Cellobita]

On FreeBSD this was an issue, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275063, fixed in 14.0-RELEASE-p2.

I wonder if the bug could somehow be related to my issue, and a similar fix applicable, even if the underlying OS in SCALE is Linux; not a developer, so can't begin to guess from a technical perspective...