Plex service stopping inside jail

[itractus]

I can see a handfull of people are (or were) having issues with the plex service stopping inside the bsd jail. As am I.

I'm running the lastest truenas, with a manually created jail and installed the plex server. It works really well but randomly stops the service. When I manually restart the service plex works again. This happens daily. I'm not adept enough at BSD to troubleshoot my own way out of this so I need some pointers.

Probably there is an issue with the jail, I assume this since the plex logs don't show any big errors that are abnormal.

Thanks in advance.

 

[itractus]

It seems to be related to a swap issue as per the dmesg.
That is in turn related to the arc something?? I need some help here. I'm out of depth.

 

[Patrick M. Hausen]

The NAS system running out of swap is caused by "something" consuming too much memory. Most probably Plex in your case.
You could set up a cron job that does ps awux regularly and directs the output to a file. The column labelled "RSS" tells you how much actual memory every process uses.

 

[itractus]

It doesn't look like anything is hogging all the memory. I have 16Gigs installed and a 14.5TB pool.

Code:

USER         PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED        TIME COMMAND
root          11 799.0  0.0       0    128  -  RNL  Sun15   31898:29.14 [idle]
kclijsters 66176   5.7  0.4  232808  65008  -  S    09:58       2:10.97 /usr/local/sbin/smbd --daemon
root           0   1.3  0.1       0  11792  -  DLs  Sun15      17:40.21 [kernel]
root        1654   0.8  0.0       0     16  -  DL   Sun15       0:06.80 [aiod10]
root        1656   0.3  0.0       0     16  -  DL   Sun15       0:06.08 [aiod12]
root         324   0.2  5.7 1161408 942248  -  S    Sun15      34:00.63 python3.8: middlewared (python3.8)
root        1650   0.1  0.0       0     16  -  DL   Sun15       0:06.44 [aiod6]
root           1   0.0  0.0    9932    292  -  ILs  Sun15       0:00.04 /sbin/init --
root           2   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto]
root           3   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 0]
root           4   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 1]
root           5   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 2]
root           6   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 3]
root           7   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 4]
root           8   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 5]
root           9   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 6]
root          10   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [audit]
root          12   0.0  0.0       0    464  -  WL   Sun15       5:50.46 [intr]
root          13   0.0  0.0       0     48  -  DL   Sun15       0:04.04 [geom]
root          14   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [crypto returns 7]
root          15   0.0  0.0       0     48  -  DL   Sun15       1:09.85 [cam]
root          16   0.0  0.0       0     48  -  DL   Sun15       0:00.16 [ctl]
root          17   0.0  0.0       0    800  -  DL   Sun15       0:18.34 [usb]
root          18   0.0  0.0       0     16  -  DL   Sun15       0:00.05 [soaiod1]
root          19   0.0  0.0       0     16  -  DL   Sun15       0:00.05 [soaiod2]
root          20   0.0  0.0       0     16  -  DL   Sun15       0:00.05 [soaiod3]
root          21   0.0  0.0       0     16  -  DL   Sun15       0:00.05 [soaiod4]
root          22   0.0  0.0       0   2240  -  DL   Sun15       1:48.01 [zfskern]
root          23   0.0  0.0       0     16  -  DL   Sun15       2:33.64 [pf purge]
root          24   0.0  0.0       0     16  -  DL   Sun15       0:16.36 [rand_harvestq]
root          25   0.0  0.0       0     48  -  DL   Sun15       1:20.34 [pagedaemon]
root          26   0.0  0.0       0     16  -  DL   Sun15       0:00.60 [vmdaemon]
root          27   0.0  0.0       0    128  -  DL   Sun15       0:22.41 [bufdaemon]
root          28   0.0  0.0       0     16  -  DL   Sun15       1:49.84 [syncer]
root          29   0.0  0.0       0     16  -  DL   Sun15       0:01.50 [vnlru]
root         307   0.0  0.0   10916   1356  -  Ss   Sun15       0:00.13 /sbin/devd -q
root         310   0.0  0.0   18264   3084  -  Is   Sun15       0:00.03 /usr/local/sbin/zfsd
root         323   0.0  0.0   10844    460  -  Is   Sun15       0:00.00 daemon: /usr/local/bin/middlewared[324] (daemon)
root         328   0.0  0.0   21532   7252  -  I    Sun15       0:00.05 /usr/local/bin/python3.8 -c from multiprocessing.resource_tracker import main;main(11)
root         521   0.0  0.2   59520  33552  -  I    Sun15       0:00.61 python3.8: /usr/local/bin/python3.8 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_f
root         548   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [ftcleanup]
root         552   0.0  0.0       0     16  -  DL   Sun15       0:00.00 [g_mp_kt]
root        1076   0.0  0.0   19472      0  -  IW   -           0:00.00 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog.pid
root        1084   0.0  0.0   11172    240  -  Ss   Sun15       0:00.25 /usr/sbin/rpcbind
root        1243   0.0  0.0   18008   3108  -  Ss   Sun15       0:20.12 /usr/sbin/mountd -rS /etc/exports /etc/zfs/exports
root        1249   0.0  0.0   17240   2388  -  Is   Sun15       0:00.01 nfsd: master (nfsd)
root        1250   0.0  0.0   10784    484  -  S    Sun15       0:00.43 nfsd: server (nfsd)
root        1253   0.0  0.0  279456    212  -  Is   Sun15       0:00.20 /usr/sbin/rpc.statd
root        1256   0.0  0.0   17336   2448  -  Ss   Sun15       0:00.45 /usr/sbin/rpc.lockd
ntpd        1301   0.0  0.0   18956   1340  -  Ss   Sun15       0:11.52 /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
root        1337   0.0  0.0  217208   1948  -  Is   Sun15       0:03.67 /usr/local/sbin/smbd --daemon
root        1345   0.0  0.0   46352   3712  -  Ss   Sun15       0:02.35 /usr/local/sbin/winbindd --daemon
root        1350   0.0  0.0   47508   4636  -  I    Sun15       0:00.50 winbindd: domain child [TRUENAS] (winbindd)
root        1356   0.0  0.0  173892   1276  -  I    Sun15       0:00.22 /usr/local/sbin/smbd --daemon
root        1357   0.0  0.0  173916   1680  -  S    Sun15       0:00.19 /usr/local/sbin/smbd --daemon
root        1358   0.0  0.0   47940   2728  -  I    Sun15       0:00.28 winbindd: idmap child (winbindd)
root        1369   0.0  0.0   10844    424  -  Is   Sun15       0:00.00 daemon: /usr/local/bin/wsdd.py[1370] (daemon)
daemon      1370   0.0  0.1   72396  11568  -  I    Sun15       0:00.33 python3 /usr/local/bin/wsdd.py (python3.8)
root        1385   0.0  0.0   14796   1748  -  Is   Sun15       0:00.22 /usr/local/sbin/smartd -n -i 1800 -c /usr/local/etc/smartd.conf -p /var/run/smartd-daemon.pid
root        1391   0.0  0.0   46408    972  -  I    Sun15       0:00.28 winbindd: domain child [BUILTIN] (winbindd)
root        1392   0.0  0.0   38936   4136  -  Ss   Sun15       3:37.93 /usr/local/bin/rrdcached -s www -l /var/run/rrdcached.sock -p /var/run/rrdcached.pid
root        1402   0.0  0.0   35500      0  -  IWs  -           0:00.00 nginx: master process /usr/local/sbin/nginx
www         1403   0.0  0.0   37724   3452  -  S    Sun15       0:11.76 nginx: worker process (nginx)
messagebus  1408   0.0  0.0   12480    772  -  Is   Sun15       0:00.00 /usr/local/bin/dbus-daemon --system
root        1425   0.0  0.0   10844    688  -  Is   Sun15       0:00.00 daemon: /usr/local/sbin/collectd[15718] (daemon)
root        1470   0.0  0.0   11400    804  -  Ss   Sun15       0:00.78 /usr/sbin/cron -s
root        1645   0.0  0.0       0     16  -  DL   Sun15       0:06.11 [aiod1]
root        1646   0.0  0.0       0     16  -  DL   Sun15       0:06.38 [aiod2]
root        1647   0.0  0.0       0     16  -  DL   Sun15       0:06.59 [aiod3]
root        1648   0.0  0.0       0     16  -  DL   Sun15       0:06.43 [aiod4]
root        1649   0.0  0.0       0     16  -  DL   Sun15       0:06.53 [aiod5]
root        1651   0.0  0.0       0     16  -  DL   Sun15       0:06.34 [aiod7]
root        1652   0.0  0.0       0     16  -  DL   Sun15       0:06.64 [aiod8]
root        1653   0.0  0.0       0     16  -  DL   Sun15       0:05.99 [aiod9]
root        1655   0.0  0.0       0     16  -  DL   Sun15       0:06.05 [aiod11]
root        1657   0.0  0.0       0     16  -  DL   Sun15       0:06.68 [aiod13]
root        1658   0.0  0.0       0     16  -  DL   Sun15       0:06.80 [aiod14]
root        1659   0.0  0.0       0     16  -  DL   Sun15       0:06.41 [aiod15]
root        1660   0.0  0.0       0     16  -  DL   Sun15       0:06.30 [aiod16]
root        3370   0.0  0.0   28568   5604  -  Is   Sun17       0:00.76 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog.pid
root       15655   0.0  0.9  207256 145840  -  I    Mon02       1:17.51 python3.8: middlewared (worker) (python3.8)
root       15657   0.0  0.9  207312 146540  -  I    Mon02       1:16.27 python3.8: middlewared (worker) (python3.8)
root       15658   0.0  0.9  206304 145096  -  I    Mon02       1:10.92 python3.8: middlewared (worker) (python3.8)
root       15659   0.0  0.9  208032 146960  -  I    Mon02       1:09.12 python3.8: middlewared (worker) (python3.8)
root       15660   0.0  0.9  206848 146080  -  I    Mon02       1:05.87 python3.8: middlewared (worker) (python3.8)
root       15718   0.0  0.2   88292  36064  -  S    Mon02      31:13.86 /usr/local/sbin/collectd -f
root       32241   0.0  0.0   11428   1780  -  SsJ  Mon20       0:00.30 /usr/sbin/syslogd -c -ss
root       32313   0.0  0.0   11248    984  -  SsJ  Mon20       0:00.62 /usr/sbin/cron -J 15 -s
avahi      32389   0.0  0.0   12844   2700  -  S    Mon20       0:03.98 avahi-daemon: running [truenas.local] (avahi-daemon)
972        32397   0.0  0.9  245128 155680  -  INJ  Mon20       0:57.72 Plex Plug-in [com.plexapp.system] /usr/local/share/plexmediaserver-plexpass/Resources/Plug-ins-d0ce30438/
972        37384   0.0  0.3   98832  52504  -  IJ   Tue02       0:45.48 Plex Plug-in [com.plexapp.agents.localmedia] /usr/local/share/plexmediaserver-plexpass/Resources/Plug-ins
root       66165   0.0  0.2  223292  26116  -  S    09:57       0:00.31 /usr/local/sbin/smbd --daemon
root       66177   0.0  0.3  232168  49528  -  S    09:58       0:26.68 /usr/local/sbin/smbd --daemon
root       66891   0.0  0.0   11236   2248  -  S    10:47       0:00.08 iostat -d -I -w 2 -x
root       67062   0.0  0.0   19040   7568  -  Is   10:53       0:00.01 sshd: /usr/local/sbin/sshd [listener] 0 of 10-100 startups (sshd)
root       67152   0.0  0.0   19584   8176  -  Ss   10:54       0:00.02 sshd: root@pts/1 (sshd)
972        67308   0.0  0.5  273512  90528  -  IsJ  10:59       0:01.59 /usr/local/share/plexmediaserver-plexpass/Plex_Media_Server (Plex Media Server)
972        67329   0.0  0.3  126384  52660  -  INJ  11:00       0:02.17 Plex Plug-in [com.plexapp.system] /usr/local/share/plexmediaserver-plexpass/Resources/Plug-ins-d0ce30438/
972        67334   0.0  0.1   46984  17512  -  SJ   11:00       0:00.07 /usr/local/share/plexmediaserver-plexpass/Plex Tuner Service /usr/local/share/plexmediaserver-plexpass/Re
972        67335   0.0  0.3   95636  49668  -  IJ   11:00       0:01.69 Plex Plug-in [com.plexapp.agents.imdb] /usr/local/share/plexmediaserver-plexpass/Resources/Plug-ins-d0ce3
root        1545   0.0  0.1   49252  21612 v0  Is+  Sun15       0:00.32 python3 /etc/netcli (python3.8)
root        1546   0.0  0.0   10884    456 v1  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv1
root        1547   0.0  0.0   10884    456 v2  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv2
root        1548   0.0  0.0   10884    456 v3  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv3
root        1549   0.0  0.0   10884    456 v4  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv4
root        1550   0.0  0.0   10884    456 v5  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv5
root        1551   0.0  0.0   10884    456 v6  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv6
root        1552   0.0  0.0   10884    456 v7  Is+  Sun15       0:00.00 /usr/libexec/getty Pc ttyv7
root       67154   0.0  0.0   13608   4084  1  Ss   10:54       0:00.01 -zsh (zsh)
root       67370   0.0  0.0   11968   2840  1  R+   11:02       0:00.00 ps awux
root       67235   0.0  0.2   52608  40648  2  Is   10:58       0:00.57 /usr/local/bin/python3.8 /usr/local/bin/iocage console -f plex
root       67267   0.0  0.0   12048   2780  2  IJ   10:58       0:00.01 login [pam] (login)
root       67268   0.0  0.0   13376   3664  2  I+J  10:58       0:00.01 -csh (csh)

 

[itractus]

Looking in this a little further, it looks like there is no swap in my system. I set my swap to 40GiB in the GUI, but checking "swapinfo" shows nothing. I'm sure this is not normal behavior.... right?

 

[ThreeDee]

I just "upgraded" from a 128GB M.2 NVMe to a 256GB one and now I am having issues with Plex randomly stopping too.

Plex has never had issues before, but now on this new bigger drive just for jails (Plex, Ubiquiti Controller) .. I'm having issues. Ubiquiti hasn't randomly stopped working, just Plex.

12.2 p6 jail and manual install of plexpass 1.23.1.4528-c0513eb4c

I'm going to reinstall and see if it's just something I booger'd up during install or from using PMS_Updater.sh script

 

[Patrick M. Hausen]

Looking in this a little further, it looks like there is no swap in my system. I set my swap to 40GiB in the GUI, but checking "swapinfo" shows nothing. I'm sure this is not normal behavior.... right?

Swap space is reserved when you create the pool. And obviously you did with swap space set to 0. You cannot change the size after the fact.

 

[itractus]

So, can I somehow create a swap file and use that?

 

[Patrick M. Hausen]

Not in a supported way in the UI, but you could use a POSTINIT task with a shell script that does that.

 

[itractus]

Swap space is reserved when you create the pool. And obviously you did with swap space set to 0. You cannot change the size after the fact.

So, creating a swap file is what i'll do. I used the following commands to create a 20Gig swapfile:

Code:

dd if=/dev/zero of=/usr/swap0 bs=1G count=20
chmod 0600 /usr/swap0
echo "md99    none    swap    sw,file=/usr/swap0,late    0    0" >> /etc/fstab
swapon -aL

This enables the swap file, now I just need to:

use a POSTINIT task with a shell script that does that.

How would I go about doing that?

I'm also going to see if plex now stop quitting the service.

 

[Patrick M. Hausen]

I would not write into the /usr directory because that will end up in your boot-pool. I'd use /mnt/<yourpoolname>/swap0 or similar.

To have that executed automatically after reboot, put all of these commands into a shell script, then use the UI "Tasks --> Init/Shutdown Scripts" and add that shell script as a task of type POSTINIT. Place the shell script somewhere on your pool, too. Don't write anywhere in the system directories or your changes will be gone after the next update.

 

[itractus]

awsome, thanks for the help. Now i'm going to let this run for a while to see if it solves the issue.

 

[itractus]

So after moving the swap file to the pool, it persists and my script looks as follows:

Code:

echo "md99    none    swap    sw,file=/mnt/NAS/Backend/swap0,late    0    0" >> /etc/fstab
swapon -aL

A quick reboot shows the swap file is correctly attached. Now I'm going to do some testing to see if there is still an issue with plex

 

[itractus]

After some testing, it seems to not work.
Not only does the service stop, I also cannot dmesg anymore, getting the following error when I do:

The jail now also shows as 'down' in the morning after a night of running plex.

Any pointers?

 

[Patrick M. Hausen]

You cannot call dmesg inside a jail. Try on the NAS system.

 

[ThreeDee]

After having my Plex crash seemingly every night around 2 am'ish .. It seemed to have been hardware related to my 3700x on my ASRock x470D4U.

I turned off C-States, PBO and another "feature" that auto boosts/overclocks my CPU of which in itself wasn't so much an issue as was the extra voltage that came with those boosts causing CPU to get into the 70+ c's with seemingly not much going on. Now it's sitting under 28-30c and my Plex didn't crash last night.

 

[itractus]

It's not a CPU issue, nor a bios issue. I've not had issues with the same host using different software. It is definitly swap space related though... I'm still getting that error.

 

[Patrick M. Hausen]

It's not a CPU issue, nor a bios issue. I've not had issues with the same host using different software. It is definitly swap space related though... I'm still getting that error.

So the Plex processes are using too much memory. That's a Plex problem, not a TrueNAS one. Possibly there is a memory leak (bug) or your system has too little memory for your workload. Restarting Plex every night could be a possible workaround.

 

[itractus]

That would be my guess, I have 16Gigs in the system now, and don't really want to add spend money to add higher density modules to an old desktop. I've got 20Gigs set up as swap, i've ran plex server for years and never really needed more than 8 gigs, so it seems there is an issue alright. Is there a way i can track what's using the ram every, say, 5 mins? write that to a file and keep that somewhere to check out when it crashes. This way I can isolate what's happening.
thanks for the help

 

[Patrick M. Hausen]

Of course.

ps awwux >>/mnt/your-pool/some/scratch/space/ps.txt

in a cron job every five minutes.