Memory issues with fine running FreeNAS machine. Help!

[watoursw]

I've been running a freenas box for the past 12 months, but over the last 2 months, the system has been acting strange. It randomly stops responding, shuts down the plex plugin and has just been a nightmare. I've been researching the symptoms and all things hint to the swap memory usage being the culprit. Appreciate if the community can help debug this issue and point me in the right direction. Here are some key pieces of information:

• I have 5 * 3 TB drives running in Raid-Z2 with 8GB RAM and the purpose of the box is solely to run 1-2 video streams on Plex
• The issue of ad-hoc shutdown (server becoming unresponsive) started in March'19; the system had been running fine at that point for ~10 months
• When I look at the logs, the problem always coincides with swap memory being fully used up
• Doubling the RAM to 16 GB did not address the problem
• In memory utilization chart below, the system used to have 10 GB of swap memory but it somehow seemed to have decreased to 5 GB

Appreciate any help!

 

[dlavigne]

What's the full output of ifconfig within code tags? Which FreeNAS version and Plex version? Anything in /var/log/messages around the time the issue occurs?

 

[watoursw]

Thank you for the reply.
FreeNAS version is 11.2-U4.1 (i used to run an older version but upgraded recently to see if the upgrade fixes the issue but it hasn't helped so far)
Plex version is 1.10.1.4602

Here's the messages in /var/log/messages around the last time the problem happened

here's the ifconfig output

 

[Elliot Dierksen]

16GB is the bare minimum for FreeNAS only. If you are running virtual machines, you need more RAM. The output you posted confirms that with the "out of swap space" messages.

 

[glauco]

Have you discovered which processes are using so much RAM?
htop is perfect for this purpose. Launch it and sort processes by RAM usage.

 

[watoursw]

16GB is the bare minimum for FreeNAS only. If you are running virtual machines, you need more RAM. The output you posted confirms that with the "out of swap space" messages.

I do not use VMs. I agree with your assessment; based on all the research I have done, the issue seems to be that the 8 GB RAM is inadequate. So i upgraded to 16 GB but that did not solve the problem. Before I buy even more RAM, i want to ensure I'm treating the problem (and not just the symptom). The system ran perfectly well for 10 months with just 8 GB and now, it doesn't work with even 16 GB with no change change in the usage. I cannot explain why that would be.

Have you discovered which processes are using so much RAM?
htop is perfect for this purpose. Launch it and sort processes by RAM usage.

Ran htop today under some basic load and didn't notice anything unusual (memory wise at least). My CPU seems to be fully loaded while transcoding a high res video but i guess that is expected.

 

[glauco]

htop says you still only have 8 GB of RAM. Perhaps the additional 8 GB module wasn't recognized?

 

[SweetAndLow]

Do you have any tunables set? Delete all of them!

 

[watoursw]

htop says you still only have 8 GB of RAM. Perhaps the additional 8 GB module wasn't recognized?

i removed the 8 GB stick i had added as it made the problem worse (the system would run out of memory more frequently)

Do you have any tunables set? Delete all of them!

No tunables set. On that note, also tried autotune but it didn't help; so deleted them all.

 

[Elliot Dierksen]

I do not use VMs

I should have said jails there. You mention transcoding, so I am going to guess that you have Plex or something like that running in a jail. In any case, it still looks like you are way low on RAM. Not to be cavalier with your $$$, but I wouldn't think that going to 32GB would be too pricey. One thing I have seen in other threads is that autotune assumes the FreeNAS is the only thing there, and it tries to use all the RAM. If you have a jail running, that will obviously need RAM as well. You may need to manually tune down the max ARC size.

 

[Yorick]

I’d say ARC as well. I have your setup, pretty much - basic SMB, plex, and one VM that uses 1 GiB. I use under 3 GiB for all processes, and around 25 GiB for ARC.

Total in this server is 32GiB, only because that seemed like a sane amount when building it.

With 16GiB, restricting ARC to 10GiB should work, and 12GiB would be the max you could get away with.

It’s possible something else is running that’s taking ram, htop should help in identifying that. My htop shows 2.77 GiB used, and top shows 25 GiB of ARC.

 

[watoursw]

You may need to manually tune down the max ARC size.

With 16GiB, restricting ARC to 10GiB should work, and 12GiB would be the max you could get away with.

Thank you. I'll try this for a week and see how it goes. A basic question:

What should I set vfs.zfs.arc_max value to limit it to 10 GB? does vfs.zfs.arc_max="10000M" mean 10 GB?

 

[Yorick]

You should be able to just do =“10G” . Expressed in M it’d be =“10240M” if you want to be precise.

Edit: That's as a tunable, loader presumably. If you want it to take effect right away without a reboot, this on shell:
sysctl -a vfs.zfs.arc_max=10737418240

Edit 2: Oh funny, the human-friendly numbers in loader subtract a GiB. 10G ends up as 9663676416, 9 GiB. Not a big deal, but still funny.

 

[Yorick]

I am wondering: Should the default value of arc_max be considered a bug, or a feature? It's 1 GiB below max memory, which is going to cause swapping if it ever gets there. 4 GiB below max memory would be a more conservative setting. I'd even take 3 GiB below, it'd be an improvement for systems that have been built with smaller amounts of RAM.