FreeNAS completely unresponsive lots of messages in console with "swap_pager_getswapspace(32): failed"

[Mannekino]

I woke up this morning to my FreeNAS server being completely dead. I launched IPMI and saw a lot of messages in the console with:

Code:

swap_pager_getswapspace(32): failed

I wasn't able to type anything so I decided to do a reboot through the IPMI controls. I've never experienced this before. I've looked at some graphs and saw this:

I would like to know what went wrong. I've looked at /var/log/messages but I can't find anything before the first swap error started showing.

This is all in the log before it started:

Code:

Oct  8 00:00:00 freenas syslog-ng[974]: Configuration reload finished;
Oct  9 00:00:00 freenas syslog-ng[974]: Configuration reload request received, reloading configuration;
Oct  9 00:00:00 freenas syslog-ng[974]: Configuration reload finished;
Oct  9 03:45:00 freenas ZFS: vdev state changed, pool_guid=xxx vdev_guid=xxx
Oct  9 23:33:57 freenas swap_pager_getswapspace(32): failed
Oct  9 23:33:57 freenas swap_pager_getswapspace(32): failed

 

[Mannekino]

Bump... I haven't had any issues since the reboot and I also did the upgrade to 11.3-U5.

In the release notes I found a fix related to this issue and I'm wondering if this is what I experienced.

https://jira.ixsystems.com/browse/NAS-107315

 

[Mannekino]

It happened again unfortunately and I think Plex might be the culprit. Before the system became completely unresponsive and I managed to get a quick look at htop. I just did another test and kept htop open and saw the memory usage of Plex going up pretty fast. About 5 minutes into this particular video file Plex stopped playback and memory usage reached about 60%. After I stopped Plex using iocage stop plex the memory was released again.

Anything thought on how I should proceed with this? I already updated Plex to the latest version I'm on 1.20.2.3402 right now.

Graph of the test I did just now

 

[Elliot Dierksen]

You don't have enough RAM for your workload. That is what the messages about swapping are telling you. If you stop the Plex service, I bet is it more responsive. If this is your purpose for this server, then you need to invest in some more RAM. You could also tune down the amount of RAM FreeNAS tries to use for ZFS cache. You will likely need to do that anyway if you have a service or jail that needs a lot of memory. FreeNAS tries to allocate all that is available for cache unless you stop it from doing do via tunables.

 

[Mannekino]

I don't know why you conclude I don't have enough RAM, 32 GB is more than enough for my purposes and I haven't had any issues like this before. I've narrowed it down to a memory leak issue with Plex. It only happened when I was viewing a certain video file that was being transcoded by Plex. It took about 10 minutes for the memory leak to reach the level where it would take up all my RAM and swap space.

 

[anodos]

I don't know why you conclude I don't have enough RAM, 32 GB is more than enough for my purposes and I haven't had any issues like this before. I've narrowed it down to a memory leak issue with Plex. It only happened when I was viewing a certain video file that was being transcoded by Plex. It took about 10 minutes for the memory leak to reach the level where it would take up all my RAM and swap space.

Can you use `top` to figure out which process is consuming all the RAM?

 

[HoneyBadger]

It only happened when I was viewing a certain video file that was being transcoded by Plex.

"Only with a specific file" might indicate an issue in the decode process for Plex. Do you know what codec is being used in the source file? (H.265/HEVC maybe?)

 

[Mannekino]

Can you use `top` to figure out which process is consuming all the RAM?

It was the Plex Transcoder process that was leaking memory. I haven't been able to reproduce the issue since last week. I also upgraded Plex to the latest version. I'm running the same video file right now through Plex and the memory usage is stable.

"Only with a specific file" might indicate an issue in the decode process for Plex. Do you know what codec is being used in the source file? (H.265/HEVC maybe?)

Viewing the same file now and this is what I'm seeing in Tautulli:

Maybe a memory leak issue was present in the previous version of Plex I was running and it has been fixed now? Dunno, I'm 4:30 minutes into the same file I was viewing last week and by now the memory for the Transcoder proces was up to 35% or so, it's still holding stable at 1.4% now.

 

[HoneyBadger]

Glad it's been solved. I poked in the Alert Settings but I don't see a way to set up "alert if swap is used or greater than percentage" - that would be a good thing to have if it's possible to configure somehow.

 

[anodos]

Glad it's been solved. I poked in the Alert Settings but I don't see a way to set up "alert if swap is used or greater than percentage" - that would be a good thing to have if it's possible to configure somehow.

Maybe enhancement request on Jira?