High Memory Utilization?

[rjkboyle]

Hi all,
I noticed bad performance issues, and I'm seeing "swap_pager _getswapspace(*): failed" errors on my screen.

System Specs:
FreeNAS-11.2-BETA1
Intel Xeon CPU E5-2640 v4 @2.40
Memory 130922MB
12x 4TB in 2 vdevs.

I'm only running one Bhyve with 32GB assigned memory.

But I'm assuming I'm having some sort of memory issues, because swap space is full. But why am I using swap if I have 128GB of RAM?
I just recently upgraded to 11.2 from 11.1, and this seems to be a new issue.

Sorry if I haven't provided enough detail, happy to answer any questions.

 

[jro]

I would suggest checking out top and/or htop to try and figure out what's taking up all that memory. You can access top directly from the web UI by clicking "Display System Processes" on the left menu. You can access htop via the shell.

edit: it looks like the web UI version of top is not interactive, so you won't be able to sort by memory usage. If you run top or htop in the shell, you'll be able to sort by memory usage.

 

[kdragon75]

10GB of SWAP! holy cow! I cant wait to see the top output!

 

[rjkboyle]

Top didn't show anything using much RAM other than Bhyve which was using the assigned 32GB.
I was able to determine that ARC was using ~87GB of RAM which wasn't leaving much for anything else.
I just turned on AutoTune and rebooted to see where the system set the limit - I found an obscure thread referencing systems with high (over 100GB) of RAM seeing issues like this.

Any input is appreciated.

 

[rjkboyle]

So at one point in time Cyberjock posted about how AutoTune helps system with a lot of RAM. Ref
That doesn't seem to be the case anymore


Ran AutoTune and got the following.

Results seem slightly better?
I'm trying to max out ARC usage, and I'm still seeing it using Swap space.

Is there a reason why it should be going into Swap with 128Gb of physical RAM?

 

[kdragon75]

Try this:
top -w
Then, sort processes according to their swap usage pressing "o",
then write "swap" and press RETURN.

 

[rjkboyle]

Ah! NFS?

Edit: I was trying to utilize a lot of performance so was pulling a lot of data via NFS with the intention to fill up ARC and hit swap? If that logic makes any sense ha.

 

[kdragon75]

Hmm 2531M used but only 31M listed...

 

[Elliot Dierksen]

Is there a reason why it should be going into Swap with 128Gb of physical RAM?

None that comes to mind. After making big changes to the amount of memory in my FreeNAS, I did follow Cyberjock's guidance on that. What worked for me was:

1) disable autotune and reboot
2) delete all autotune tunable and reboot
3) enable autotune and reboot

Kind of disruptive, but everything seemed happy afterwards. That said, I only use my FreeNAS for storage. If you are running jails/iocages, you may need to cap the amount of memory it tires to grab for the ARC. Any gains from stuff in the ARC are more than offset by the performance cost of swapping.

 

[rjkboyle]

Thanks for the reply Elliot.
I never had autotune enabled or any tunables before just enabling it today.

I think I might let it run for a bit and continue to monitor. But then maybe I need to lower the max ARC.

arc_summary show the following.

Before Autotune:
ARC Size: 92.46% 84.71 GiB
Target Size: (Adaptive) 95.53% 87.52 GiB
Min Size (Hard Limit): 16.87% 15.45 GiB
Max Size (High Water): 5:1 91.62 GiB

After Autotune:
ARC Size: 90.34% 82.78 GiB
Target Size: (Adaptive) 92.34% 84.60 GiB
Min Size (Hard Limit): 16.87% 15.45 GiB
Max Size (High Water): 5:1 91.62 GiB

Doesn't seem like it really changed anything so I wonder if I'm just going to see the same issues.

Maybe I need to try adding 'vfs.zfs.arc_max' and setting it to 80GiB or something. Because autotune set it to 123552000000 which I'm assuming is 123GB?

 

[kdragon75]

Did you reboot after enabling auto tune? It would seem you ARC filled rather fast.

 

[rjkboyle]

Sure did!
I started a couple transfers of ~500GB each to different system using NFS. Watched as the ARC filled gradually up to 90%. Once it got there then I saw the swap start to be ulitized.

 

[Elliot Dierksen]

I started a couple transfers of ~500GB each to different system using NFS. Watched as the ARC filled gradually up to 90%. Once it got there then I saw the swap start to be ulitized.

I suspect (but am not positive) that autotune is going to work based on the assumption that it is just doing storage related things. I have never had to do any manual tuning of FreeNAS, so take this with a grain of salt. I would reduce the max ARC size by at least the amount of RAM that you are allocating to any jail/iocages.

 

[rjkboyle]

Thanks Elliot! I think you're probably right.

 

[pro lamer]

Did it help? (background: I'm considering building a E5 rig to be able to have lots of RAM)

 

[rjkboyle]

It sure did!
Autotune did a good enough job it seemed. However I did manually calculate a max arc size and set that. Since then I haven't seen the box use swap, and no more performance issues.