Restarting crashed jail takes down whole server

[JoeB]

I dont know where to start with diagnosing this, but it's happened a few times now.

CouchPotato stops responding to http, so I restart the jail from the FN web GUI (it's installed in a jail, not as a plugin).
The restart of the jail hangs, then the server stops serving to all http requests from all jails, plex goes down on the TV and the FN web GUI stops serving pages too.

After about 3 mins, the web GUI comes back to life, with a jail stopped: virtualbox. I'm not sure if this was running before the crash, but it should have been.

Has this happened to anyone else? What can I do to fix it, it's anoying sure, but not a huge issue.

 

[dlavigne]

Anything related in /var/log/messages?

 

[JoeB]

Not that i can see, it just looks like a restart occurred at 11:06:30 - about when i tried to restart the jail.

Code:

Dec 23 09:09:26 JOE-FREENAS kernel: pid 17329 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 09:23:44 JOE-FREENAS kernel: pid 18567 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 09:24:05 JOE-FREENAS kernel: pid 18622 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 09:24:17 JOE-FREENAS kernel: pid 18623 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 09:36:44 JOE-FREENAS kernel: pid 19739 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 09:53:51 JOE-FREENAS kernel: pid 21184 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:06:44 JOE-FREENAS kernel: pid 22399 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:23:52 JOE-FREENAS kernel: pid 23888 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:24:05 JOE-FREENAS kernel: pid 23941 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:24:17 JOE-FREENAS kernel: pid 23943 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:36:48 JOE-FREENAS kernel: pid 25069 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:38:44 JOE-FREENAS kernel: pid 25205 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:38:56 JOE-FREENAS kernel: pid 25210 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:39:17 JOE-FREENAS kernel: pid 25263 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:39:22 JOE-FREENAS kernel: pid 25267 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 10:54:18 JOE-FREENAS kernel: pid 26569 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 11:01:25 JOE-FREENAS kernel: pid 27289 (Plex Media Scanner), uid 972: exited on signal 11
Dec 23 11:06:30 JOE-FREENAS syslog-ng[1422]: syslog-ng starting up; version='3.6.4'
Dec 23 11:06:30 JOE-FREENAS Copyright (c) 1992-2016 The FreeBSD Project.
Dec 23 11:06:30 JOE-FREENAS Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Dec 23 11:06:30 JOE-FREENAS	 The Regents of the University of California. All rights reserved.
Dec 23 11:06:30 JOE-FREENAS FreeBSD is a registered trademark of The FreeBSD Foundation.
Dec 23 11:06:30 JOE-FREENAS FreeBSD 10.3-STABLE #0 r295946+07c41cd(9.10-STABLE): Wed Nov  9 00:19:25 UTC 2016
Dec 23 11:06:30 JOE-FREENAS root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
Dec 23 11:06:30 JOE-FREENAS FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Dec 23 11:06:30 JOE-FREENAS CPU: AMD Turion(tm) II Neo N54L Dual-Core Processor (2196.39-MHz K8-class CPU)
Dec 23 11:06:30 JOE-FREENAS Origin="AuthenticAMD"  Id=0x100f63  Family=0x10  Model=0x6  Stepping=3
Dec 23 11:06:30 JOE-FREENAS Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Dec 23 11:06:30 JOE-FREENAS Features2=0x802009<SSE3,MON,CX16,POPCNT>
Dec 23 11:06:30 JOE-FREENAS AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
Dec 23 11:06:30 JOE-FREENAS AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId>
Dec 23 11:06:30 JOE-FREENAS SVM: NP,NRIP,NAsids=64
Dec 23 11:06:30 JOE-FREENAS TSC: P-state invariant
Dec 23 11:06:30 JOE-FREENAS real memory  = 17716740096 (16896 MB)
Dec 23 11:06:30 JOE-FREENAS avail memory = 16459726848 (15697 MB)
Dec 23 11:06:30 JOE-FREENAS Event timer "LAPIC" quality 400
Dec 23 11:06:30 JOE-FREENAS ACPI APIC Table: <HP	 ProLiant>
Dec 23 11:06:30 JOE-FREENAS FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Dec 23 11:06:30 JOE-FREENAS FreeBSD/SMP: 1 package(s) x 2 core(s)
Dec 23 11:06:30 JOE-FREENAS cpu0 (BSP): APIC ID:  0
Dec 23 11:06:30 JOE-FREENAS cpu1 (AP): APIC ID:  1
Dec 23 11:06:30 JOE-FREENAS random: <Software, Yarrow> initialized
Dec 23 11:06:30 JOE-FREENAS WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Dec 23 11:06:30 JOE-FREENAS ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20150515/tbfadt-644)
Dec 23 11:06:30 JOE-FREENAS ioapic0 <Version 2.1> irqs 0-23 on motherboard
Dec 23 11:06:30 JOE-FREENAS kbd1 at kbdmux0
Dec 23 11:06:30 JOE-FREENAS aesni0: No AESNI support.
Dec 23 11:06:30 JOE-FREENAS padlock0: No ACE support.
Dec 23 11:06:30 JOE-FREENAS cryptosoft0: <software crypto> on motherboard
Dec 23 11:06:30 JOE-FREENAS acpi0: <HP ProLiant> on motherboard

 

[dlavigne]

Were you able to figure this out? I wonder if you have a bad RAM stick as signal 11 is often a sign of bad or insufficient RAM.

 

[DrKK]

as signal 11 is often a sign of bad or insufficient RAM.

Not in this case Dru, more than likely.

It is fairly common for anyone running Plex on FreeBSD to get these signal-11 bomb-outs on the plexmediascanner service. I have no idea why it happens. I get it all the time too. It's plex related somehow.

 

[DrKK]

That being said, the OP appears to be asking a lot of a 16GB RAM server, with his virtual box, and relative to RAM size, large'ish pool.

 

[nojohnny101]

I'll chime in about the "pid XXXX (Plex Media Scanner), uid XXX: exited on signal 11" is about.

I never previously got these on my plex server install until I checked the option in the plex server settings to have plex automatically scan changes to the media folders it was watching to detect changes and add media to the library without having to force a rescan.

After I enabled this option, I get the above message every morning in my "daily security output" emails from the freenas box. Coincidentally, the number of "exited on signal 11" messages I get corresponds to the number of libraries I have setup in plex.