jails dying

[Jailer]

Some additional info would be helpful as well such as version of TrueNAS you are running? Was the probelmatic jails installs done via the plugins or manually? What version of FreeBSD are the jails running?

 

[georgelza]

Some additional info would be helpful as well such as version of TrueNAS you are running? Was the probelmatic jails installs done via the plugins or manually? What version of FreeBSD are the jails running?

version information was shared on very op. (TrueNAS-12.0-U6)
hmm, original plex was installed via plug in, and then updated along the way, applies to Plex and Unifi, although with Unify think thats out the door as the upgrade from 5->6 was not a click upgrade button.
qbittorrent have not been upgraded from original.
Don't think this is related to the jail, they have been version stable, this started with the most recent TrueNAS upgrade.

G

 

[georgelza]

more than happy to... if yo can share script and instructions how you want it installed, if it can help someone else.

made the call and ordered some more RAM, was lucky and got a special from my HW store where I normally get all my bits.

Plex, no real change, just organic growth of the media volume, no real significant sudden increase,
qbittorrent well there is nothing to change there, hehehehe
G

Might be interesting to try and plant a script in the Plex jail and try and record what's happening prior to the OOM event that gets it killed. Have you added a significant amount of media, photos, music, etc? Is it trying to index something, load preview media, transcode to other formats? I have a HD Home Run receiver attached to mine, and I set it to encode recordings for mobile.

Anything recent changes like that perhaps? It's unlikely to change the overall recommendation to not run with swap continuously in use, you're still going to want more RAM, but it might allow you to tune things a bit in your favor and avoid the crash.

 

[georgelza]

Interesting...

G

 

[Samuel Tai]

See also this other thread: https://www.truenas.com/community/t...t-02-00-00-every-day-out-of-swap-space.93106/. OP discovered there was a problem with his Plex metadata causing swap to balloon.

 

[georgelza]

my posting of the diagram was a "question", we're saying I'm running out of swop, but if we look at those graphs no sign of that

G

 

[Samuel Tai]

The graphs only have 5 minute resolution (which is the default for RRD). Your console log is definitive. In fact, your swap graph is already very bad; normal behavior is to have 0 swap utilization, with very, very infrequent swap usage. In your case, you're using swap constantly.

Since Plex dies, you should look at your Plex overnight maintenance activities.

 

[georgelza]

well, lets see how it behaves now, additional 16GB RAM was just installed, 4 x 8GB, all slots now filled.

G

 

[georgelza]

well, the MEMORY upgrade did NOT help...

G

 

[georgelza]

... as said... the wheels started coking off at the last update, see attached.
let me see if there is a "new'er" version of Plex also.

G

 

[rvassar]

more than happy to... if yo can share script and instructions how you want it installed, if it can help someone else.

Sorry to hear you're still having trouble. This is really looking like some software is going seriously off the rails. The problem is it may not be Plex itself. It's just Plex that's getting clobbered by it. I kind of draw the line at doing custom scripting here in the forums. What I would do is leave something running that tails to a log file that will survive the event.

On a Linux system I'd run something like "top -b > /mnt/pool/logfile.txt", which will record all the running processes, memory use etc... But the FreeBSD version of top exits after one pass in batch mode, and I don't know the other flags that alter this behavior. Maybe someone else has another suggestion here? The htop command has a bunch of memory sorting capabilities, but it doesn't have a batch mode. The BSD version of /bin/ps has some pretty good memory reporting too. The goal is to catch the errant process as the event happens, and get a clue where to go looking...

 

[georgelza]

something is def errant, if you look at that 4 week swap space usage... don't think this is just me using swop, because not enough RAM, something fundamentally change a couple of weeks ago, which is the root cause. I got double the RAM now and still...

anyone that know how to deep diagnose this, would be appreciated,
for as like all support systems first question is mos always, are you running the most recent stable... well I'm busy running update as we speak.
I did Plex yesterday, which it seems did not help by itself as it died last night again,
did notice yesterday in some log (can't remember what where)...but seems something happened at at 03:30, need to see if I can scale back in logs, and see if this is repeated daily...
G

 

[georgelza]

Correction, it seems to be daily at approx 03:03... see 2 diagrams attached.

But this does not look like a big enough bump in swap to cause the problem...

can swap be increased ?

 

[rvassar]

Right, but that chart has 5 minute granularity, so there may be spikes... But I'm still kind of hung up on the "why", and it has me wondering if there's something in the jail config. I tried "iocage get memoryuse plex" and found mine had a value of "off". I'm wondering if yours has some limit set somewhere. Can you run "iocage get all plex" and post the result here? This is assuming you jail is named "plex".

 

[Samuel Tai]

The time lines up with the daily Plex database maintenance (between 2-5 AM). You should look at the earlier thread I posted https://www.truenas.com/community/t...t-02-00-00-every-day-out-of-swap-space.93106/, where there was a problem with Plex metadata that needed to be fixed in that poster's media to resolve the database issue.

 

[georgelza]

Excuse the spelling error on the jail name...

george@vaultx:~ $ iocage get memoryuse PexMediaServer
off
george@vaultx:~ $

and

george@vaultx:~ $ iocage get all PexMediaServer
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:0
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:1
boot:1
bpf:0
children_max:0
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:172.16.10.1
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:PexMediaServer
host_hostuuid:PexMediaServer
host_time:0
hostid:031b021c-040d-05ce-0706-1c0700080009
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|172.16.10.26/24
ip4_saddrsel:0
ip6:new
ip6_addr:none
ip6_saddrsel:0
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/PexMediaServer/data
jail_zfs_mountpoint:none
last_started:2021-12-16 05:45:47
localhost_ip:none
login_flags:-f root
mac_prefix:06aedb
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:plexmediaserver
plugin_repository:https://github.com/freenas/iocage-ix-plugins.git
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.1-RELEASE-p13
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:pluginv2
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:06aedb756232 06aedb756233
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off
george@vaultx:~ $

Right, but that chart has 5 minute granularity, so there may be spikes... But I'm still kind of hung up on the "why", and it has me wondering if there's something in the jail config. I tried "iocage get memoryuse plex" and found mine had a value of "off". I'm wondering if yours has some limit set somewhere. Can you run "iocage get all plex" and post the result here? This is assuming you jail is named "plex".

 

[rvassar]

Ok, so there's no hard memory limit set. Your jail config and mine are essentially identical. That implies that whatever is happening is eating the 16G you added and then chewing several Gb into the swap space.

I think you're going to need to pursue the link @Samuel Tai post above and figure out what the Plex DB is doing as part of it's nightly maintenance.

 

[georgelza]

I've opened a thread in the Plex discussion forum. Will advise what I get back from that side...

FYI....

Dec 15 03:13:05 vaultx kernel: pid 2010 (Plex Media Server), jid 1, uid 972, was killed: out of swap space
Dec 15 03:13:05 vaultx kernel[1095]: Last message 'pid 2010 (Plex Media' repeated 1 times, suppressed by syslog-ng on vaultx.xxxxxx.com
Dec 15 09:25:59 vaultx kernel: pid 32784 (Plex Media Server), jid 5, uid 972: exited on signal 6
Dec 16 03:15:06 vaultx kernel: pid 44663 (Plex Media Server), jid 7, uid 972, was killed: out of swap space
Dec 16 03:15:06 vaultx kernel[1095]: Last message 'pid 44663 (Plex Medi' repeated 1 times, suppressed by syslog-ng on vaultx.xxxxxx.com

G