jails dying

[georgelza]

Hi guys
Just noticed... originally thought it was related to plex... dying, so was going to ask on that forum (not that they've ever helped me resolve anything)... discovered it's not just plex, qbittorrent also dies...

but Unifi Controller did not, so not all jails, but some, seems to be ever day...

TrueNAS Core 12.0-U6

Any suggestions where the logs are, anyone aware of a known problem.

G

 

[sretalla]

Any suggestions where the logs are

Dying isn't very specific and which logs to look at will depend heavily on what you mean by it.

/var/log/messages or /var/log/middlewared.log might be places to look, but if the jail is still "running" from the TrueNAS perspective, I don't see why anything would be logged there.

If the plex service is stopping or hanging, you'll need to look into the plex logs (from the settings | manage | troubleshooting | download logs).

 

[georgelza]

It works, can be access, then it does not, accessing web page of plex and of qbittorrent just spins wheels, restarting it restores the service.

G

 

[sretalla]

With 8GB of RAM, I wouldn't be surprised to find that you're having memory contention issues after those 2 potentially memory-hungry apps have been running a while.

You may see something in dmesg to indicate something about swap.

Unless you're really unable to source more memory, I would suggest just upgrading it for happy motoring.

 

[georgelza]

they've been running happily and stable for the last 12 months,

To test the theory I will shut qbittorrent down and see if prometheus stays happy.

G

 

[rvassar]

@strella - His .sig says 2 x 8Gb, so 16Gb.... But it's still worth investigating.

I have the Plex and Zoneminder plugins configured with 16Gb of RAM, and its getting memory pressure. There's seems to have been a big jump in middleware / overhead between 11.3 and 12.0u3. I'm actually making plans to either migrate the Zoneminder stuff elsewhere or source 32Gb of RAM.

 

[georgelza]

so with qbittorrent Plex died again, can't imagine this being memory related.
found the following in /var/log/messages

Dec 11 03:19:26 vaultx swap_pager_getswapspace(9): failed
Dec 11 03:19:26 vaultx swap_pager_getswapspace(3): failed
Dec 11 03:19:26 vaultx swap_pager_getswapspace(20): failed
Dec 11 03:19:27 vaultx kernel: pid 86617 (Plex Media Server), jid 10, uid 972, was killed: out of swap space
Dec 11 03:19:28 vaultx kernel[1095]: Last message 'pid 86617 (Plex Medi' repeated 1 times, suppressed by syslog-ng on vaultx.tinmanza.com
Dec 11 03:19:28 vaultx swap_pager_getswapspace(32): failed
Dec 11 03:19:29 vaultx swap_pager_gets

jid 10 is Plex
G

 

[Samuel Tai]

You're out of swap space, which is a symptom of memory pressure. How are your pool disks partitioned? Are you using the default 2 GiB swap partitions on each pool disk?

 

[georgelza]

how where do I see this.
don't recall changing anything so guessing it will be a yes.
strange that I never had this problem... up to the recent upgrade !.
G

 

[Samuel Tai]

To see the partitioning of your pool disks, run gpart show, and look for the freebsd-swap partitions and their sizes.

FreeNAS 9 would stripe all the partitions together, so your swap size was additive over all your pool disks.
FreeNAS 11 changed this to stripes of 2-way mirrors, since there were crashes due to failures of one or more stripes. This meant total system swap was halved.
TrueNAS 12 changed this to stripes of 3-way mirrors, since there were crashes due to failures of both members of a mirror. Unfortunately, this reduced total system swap even more.

On my system with a 4-way RAIDZ2 pool, this meant my total system swap went from 8 GB (FreeNAS 9) to 4 GB (FreeNAS 11) to 2 GB (TrueNAS 12), since with only 4 disks, my swap is now 1 stripe of 3, with 1 spare.

To see your effective swap size and utilization, run swapinfo. My swap is almost never used, because I only have 1 custom basejail for borgbackup, and 1 VM for Plex.

 

[rvassar]

Dec 11 03:19:27 vaultx kernel: pid 86617 (Plex Media Server), jid 10, uid 972, was killed: out of swap space

This is kind of the end game in "memory pressure". The system is killing things to stay alive. The line above flat out states "I killed your Plex server because I'm out of memory and can't fake it with swap space". On a linux system they call it the kernel OOM reaper, and you can tweek it's behavior on a process by process basis by adjusting /proc/<pid>/oom_score_adj. I haven't looked to see if FreeBSD / TrueNAS has a similar tunable. But all you'd be doing is selecting another process to get killed.

So Samual has pointed out how the swap space has been cut over the years to manage device failures, but he didn't explain any further. Swap space is an allocation on the disk where the system kernel can "park" running program's memory allocation. The program, in this case your Plex jail, occupies some amount of real physical memory, which is shared by everything else running on the system. Each running program see's it's memory as wholly it's own, and exists in a virtualized address space from 0 to X bytes of memory. The OS see's all the different allocations and manages where they are, can keep a running score of each virtual address space page by page, and if addresses go idle, or are not going to be scheduled, it can map those virtual addresses to "swap space", and park them out of the disk, freeing up real RAM to fit more running programs. If a swapped program tries to access a page of its address space that's been moved to disk, it generates a page fault, and the kernel MMU has to select some other program to get kicked out of memory, so it can import the needed pages and allow the program to schedule on to the CPU.

If you now have a picture of your running Plex server as a block of swiss cheese with a bunch of holes, I've been successful here. The "holes" are parked out on the swap space partitions.

The problem here is this is supposed to be a transient or temporary compromise. If your Plex server is idle, and the system needs to run a scrub, Plex gets sent to the closet, and the scrub runs at 3am, and you were none the wiser. It works... Until it doesn't. In the bad old days you could get your command shell swapped out and the system would start "thrashing on swap", and you couldn't get back in to kill your program and keep the server running. That's why the OOM reaper & those tunables got created. But the real fix is to not use swap. It's there to back fill for the short term. It's quite slow, and it has a hard ending as you've discovered.

What you really need is more memory, or move some workloads off your NAS.

 

[danb35]

What you really need is more memory

This. If you're using any significant amount of swap, you need more RAM. And that isn't surprising; 8 GB is the minimum for TrueNAS alone, and Plex likes its RAM too.

If it's any consolation, Plex is much better in this regard than it used to be--I remember it exhausting my 32 GB, my swap, and crashing my system, some years back--and doing that with some frequency. I haven't had that problem for a long time (even before I upgraded my system to 128 GB). But I'd consider 16 GB a minimum for a system that's going to run Plex.

 

[Jailer]

But I'd consider 16 GB a minimum for a system that's going to run Plex.

I am in total agreement with this and would go one step further and say that 16GB is the minimum to run any jails if you want a trouble free experience.

 

[Samuel Tai]

OP has 16 GB, but for his situation, he's bursting at the seams with 16 GB. And no wonder: he's running Plex, qbitorrent, and Unifi. I suspect he's got another couple of plugins/jails on top of those. With 16 GB, he can only really comfortably run 2 jails.

 

[georgelza]

nope, only those 3 jails.
and I've been running for a year now, all stable with this exact setup, only last last week/2 after my last upgrade that I've now got problems.
I am currently only running 2 jails, I shut qbittorrent down... and it's (Plex) still having problems/creashing.

Unifi has been stable.

G

 

[kherr]

I had something similar, and what it turned out to be after a lot of troubleshooting .... VNET up and quit working. I found a fix in the forum archives ......

YMMV .....

 

[rvassar]

nope, only those 3 jails.
and I've been running for a year now, all stable with this exact setup, only last last week/2 after my last upgrade that I've now got problems.
I am currently only running 2 jails, I shut qbittorrent down... and it's (Plex) still having problems/creashing.

Unifi has been stable.

G

The message "killed: out of swap space" is pretty diagnostic. Samuel explained what happened to the swap space partitions as you upgraded.

 

[georgelza]

s

To see the partitioning of your pool disks, run gpart show, and look for the freebsd-swap partitions and their sizes.

FreeNAS 9 would stripe all the partitions together, so your swap size was additive over all your pool disks.
FreeNAS 11 changed this to stripes of 2-way mirrors, since there were crashes due to failures of one or more stripes. This meant total system swap was halved.
TrueNAS 12 changed this to stripes of 3-way mirrors, since there were crashes due to failures of both members of a mirror. Unfortunately, this reduced total system swap even more.

On my system with a 4-way RAIDZ2 pool, this meant my total system swap went from 8 GB (FreeNAS 9) to 4 GB (FreeNAS 11) to 2 GB (TrueNAS 12), since with only 4 disks, my swap is now 1 stripe of 3, with 1 spare.

To see your effective swap size and utilization, run swapinfo. My swap is almost never used, because I only have 1 custom basejail for borgbackup, and 1 VM for Plex.

o ran the command, kept seeing 2gb next to freenas_swap,
so can I increase this, looking at solutions before having to spend money.
as mentioned, I've been stable for a nearly a year now with those 3 jails running, now with one shut down when not using it crasher (seems to normally happen during the night, I can restart if now and it will be stable, until some time tonight).
Will start pricing memory,
just irritating as it was stable, until the recent upgrade,
G

 

[georgelza]

so got to comment, with purse you expect things to improve, but seriously.. I"ve now shut down all my jails, except for plex, and it's still dying, over night...

the platform use to run stable with all 3 jails enabled, now one and I have this, this is not on...

G

 

[rvassar]

so got to comment, with purse you expect things to improve, but seriously.. I"ve now shut down all my jails, except for plex, and it's still dying, over night...

the platform use to run stable with all 3 jails enabled, now one and I have this, this is not on...

G

Might be interesting to try and plant a script in the Plex jail and try and record what's happening prior to the OOM event that gets it killed. Have you added a significant amount of media, photos, music, etc? Is it trying to index something, load preview media, transcode to other formats? I have a HD Home Run receiver attached to mine, and I set it to encode recordings for mobile.

Anything recent changes like that perhaps? It's unlikely to change the overall recommendation to not run with swap continuously in use, you're still going to want more RAM, but it might allow you to tune things a bit in your favor and avoid the crash.