NAS Using Swap When SMB In Use

[oli_kester]

Hi all, I'm having an issue with my TrueNAS server, any help is much appreciated!

Every night, my laptop runs a backup job to the NAS' SMB endpoint. This causes the RAM usage to increase, which then causes the Plex plugin to be killed due to an 'out of swap space' error. Every day I have to restart my NAS / Plex plugin to fix this. The interesting thing is that the jail/plugin are still marked as 'UP' in the UI.

What's strange is that I used to have this setup working with 8GB RAM - I only had to increase that when my NAS ran into performance issues when updating from FreeNAS to TrueNAS CORE. It had been running fine for months, but only recently started having this problem.

This latest issue has come about with no changes to configuration, software or hardware. I have freed up some space to get the storage usage back down to 82%, but this hasn't helped either.

My NAS is running with the following specs:

• Dell Optiplex 390
• Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz
• 16GB RAM
• 2x 4TB HDD , RAID 1
• 2x 32GB Flash Drives used for booting
• TrueNAS-12.0-U8.1

I have a single jail/plugin running Plex. I also have a single VM running with minimal resources allocated - the problem doesn't go away if I stop the VM.

The issue looks to be cause by a spike in ZFS cache usage, which causes the system to use lots of swap space. I have tried tuning the `vfs.zfs.arc_max ` parameter, which doesn't appear to fix the issue sadly. I have also tried autotune.

I looked at what was using the swap and I found 'NGINX core', along with a few other processes.

Any ideas please?

Some logs:

Code:

Aug 18 04:46:06
nas390 swap_pager_getswapspace(2): failed
Aug 18 04:46:26
nas390 swap_pager_getswapspace(2)[813]: Last message 'failed' repeated 1 times,
suppressed by syslog-ng on nas390.nest
Aug 18 04:46:26
nas390 kernel: pid 1944 (Plex Media Server), jid 1, uid 972, was killed: out of
swap space
Aug 18 04:46:26
nas390 kernel[813]: Last message 'pid 1944 (Plex Media' repeated 1 times,
suppressed by syslog-ng on nas390.nest
Aug 18 04:46:26
nas390 kernel: pid 1312 (bhyve), jid 0, uid 0, was killed: out of swap space
Aug 18 04:46:26
nas390 kernel[813]: Last message 'pid 1312 (bhyve), ji' repeated 1 times,
suppressed by syslog-ng on nas390.nest
Aug 18 04:46:26
nas390 kernel: pid 1163 (collectd), jid 0, uid 0, was killed: out of swap space
Aug 18 04:46:26
nas390 kernel[813]: Last message 'pid 1163 (collectd),' repeated 1 times,
suppressed by syslog-ng on nas390.nest
Aug 18 04:46:26
nas390 swap_pager_getswapspace(32): failed
Aug 18 04:46:26
nas390 swap_pager_getswapspace(32)[813]: Last message 'failed' repeated 6
times, suppressed by syslog-ng on nas390.nest

 

[jgreco]

I used to have this setup working with 8GB RAM -

Holy crap, you've had this running along with Plex on 8GB RAM? That's remarkable IMO.

Please note that the minimum required memory for TrueNAS CORE was boosted to 16GB awhile back, and this number isn't intended to allow for any jails, Plex, VM's, or other memory stressors. It's just meant for basic fileservice. Your system's swapping is its way of desperately asking you to add more physical memory in order to sustain the demand.

I know it's tempting to run stuff on the smallest amount of resources possible, but I suggest bearing in mind that ZFS was written with the assumption that it could consume large amounts of resources without problem. When you start adding competition for memory such as VM's or Plex into the equation, it is easy to run the system out of memory. This leads to poor NAS performance and/or processes being killed now and then.

 

[Etorix]

I have freed up some space to get the storage usage back down to 82%, but this hasn't helped either.

If I understand correctly that your pool is 82% full, fragmentation may increase the burden of taking new data in.
I'm afraid you're looking at a two-pronged hardware update here: More RAM and bigger drives (or more drives). Plus moving to a safer geometry than raidz1.

 

[oli_kester]

Holy crap, you've had this running along with Plex on 8GB RAM? That's remarkable IMO.

Please note that the minimum required memory for TrueNAS CORE was boosted to 16GB awhile back, and this number isn't intended to allow for any jails, Plex, VM's, or other memory stressors. It's just meant for basic fileservice. Your system's swapping is its way of desperately asking you to add more physical memory in order to sustain the demand.

I know it's tempting to run stuff on the smallest amount of resources possible, but I suggest bearing in mind that ZFS was written with the assumption that it could consume large amounts of resources without problem. When you start adding competition for memory such as VM's or Plex into the equation, it is easy to run the system out of memory. This leads to poor NAS performance and/or processes being killed now and then.

Thanks for the super quick reply jgreco, and for the guidance. Sounds like things have gotten to a point where upgrade is unavoidable. Unfortunately, that motherboard won't accept any more than 16GB RAM, so the whole system will need replacing.

I've noticed my system has snapshots from FreeNAS 11.x. Would it be possible to roll back to v11 until I can upgrade my hardware, or would this cause chaos?

Thanks again

 

[oli_kester]

If I understand correctly that your pool is 82% full, fragmentation may increase the burden of taking new data in.
I'm afraid you're looking at a two-pronged hardware update here: More RAM and bigger drives (or more drives). Plus moving to a safer geometry than raidz1.

Thanks Etorix, that's right. I actually had it running around that level of storage usage in the past with no issues. But in any case, it sounds like an upgrade is very much needed at some point.

 

[Etorix]

I've noticed my system has snapshots from FreeNAS 11.x. Would it be possible to roll back to v11 until I can upgrade my hardware, or would this cause chaos?

If you have NOT upgraded pool features since 11.x I should be possible to roll back. You may always try, and revert to 12.0-U8 if the pool fails to import.
But it seems that time has come for a general upgrade.

 

[oli_kester]

After observing things for the past two days, it looks like I have fixed this issue - it actually ended up being a problem with the Plex plugin and nothing to do with my TrueNAS config or hardware.

Before reporting the issue here, I had already clicked 'update' several times on the Plex plugin and jail to try and ensure it was up-to-date. The UI gives no feedback here to tell you the status of the update, which I find strange - much better doing it through the shell. I updated several times through the shell to ensure I was on the latest TrueNAS official release.

I originally thought that it was the overnight SMB backup that was causing the memory issues, but I have since discovered that at 4AM my Plex server is conifigured to perform maintenance tasks. All of the RAM issues were starting around then.

I tried limiting the resources available to the plugin using the steps in the Jail memory allocation forum post. This did not help unfortunately.

As a last ditch attempt, I thought I would try manually updating the Plex plugin using the legendary PMS_Updater. I used to do this, but decided it might be best to stick to the official release if possible. However - now that I've done this I have my VM, Plex and file services all running on TrueNAS happily, with no tasks being killed. All hail the PMS_Updater!

There is a guide on how to use the tool on Cracked Console.