Server crashing while compiling mysql56 within pluginjail

[Kris Dahl]

I've encountered what I believe is unusual behavior while setting up a new jail, and am in need of some direction/guidance in order to troubleshoot the issue further in hopes of discovering the root cause.

Being new to Freenas, I don't have any error message to provide, only a screenshot which illustrates the "Running Processes" as well as the command executing within the SSH windows, at the time of the crash. The jail appears to be processing the command as I would expect and then just dies. The entire server crashes towards the end (80 to 90%) of compiling mysql56 within a new pluginjail.
cd /usr/ports/databases/mysql56-server && make config-recursive install clean

When it crashes, all services become unresponsive, including the main Freenas web interface. The only way to bring it back is power it down by holding the power button. Power it back on and it's as if nothing happened, until I run the command again. I've tried to "lighten the load" by stopping all other jails, same issue. I successfully executing other commands, specifically the install of postgresql, no problems.
cd /usr/ports/databases/postgresql94-server && make config-recursive install clean

At this point in time I believe the issue could lie in one of the following, I have taken a stab at ordering them in terms of likelyhood.
1 - Hardware overheating
2 - Hardware failure
3 - There's a problem with my installation
4 - Freenas bug

My main issue is, I have no logging to examine, I should say, that I know of. Is there some sort of logging that could be enabled to monitor the temp of CPU/Chipset? Is there logging enabled by default that I could/should examine? I have checked the console, once I've brought services back up, nothing indicating an issue exists.

I am certain the hardware specs are sufficient for the load, the intent of this server is for home entertainment use. However, unlike it's typical workload, I have noticed that compiling does utilize a number of the 8 cores available. Even so, the percent usage of each core appears to be less than 20%.

Any help will be greatly appreciated, please find my HW spec below.

• FreeNAS Build - FreeNAS-9.3-STABLE-201605170422
• Gigabyte GA-990FXA-UD3
  
• AMD FX-8120
  
• 16GB (2x8) DDR3 1866 RAM
• 3x2TB HDs RAIDZ1
• Builtin AMD SB950 HD Controller
• Builtin Realtek RTL8111E LAN

Thanks,
Kris

 

[MrToddsFriends]

At this point in time I believe the issue could lie in one of the following, I have taken a stab at ordering them in terms of likelyhood.
1 - Hardware overheating
2 - Hardware failure
3 - There's a problem with my installation
4 - Freenas bug

Is your system able to run prime95 without stability problems?
https://forums.freenas.org/index.php?threads/what-cpu-testing-software-to-use.42633/#post-277319

Did you run memtest86 during the burn-in test of the system?
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Do the system crashes occur each time at the same point in the mysql56 build process?

 

[Kris Dahl]

Is your system able to run prime95 without stability problems?
https://forums.freenas.org/index.php?threads/what-cpu-testing-software-to-use.42633/#post-277319

Did you run memtest86 during the burn-in test of the system?
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Thanks for your input. I had run both tests prior to installation, both past. However I was not aware that I could run both after install, and since it's possible that things could change.

I have kicked off Prime, and checking WCPU it's easy to see that this is putting vastly more stress on the CPUs than the make operation, which we all knew. I'll let it run for the afternoon for good measure but since it had previously past I won't do a 24 hour stress test.

I'll report back once tests have completed.

Thanks again, as this will be good start for troubleshooting #1 and #2.

Do the system crashes occur each time at the same point in the mysql56 build process?

No, it crashes at different points during the make operation, but generally >80%. In the original screen shot it made it 8 steps into 91%, I just reran it and it made it 6 steps into 96%.

So close, yet so far away. Bahaha :(

 

[Sakuru]

Anything in /data/crash?

 

[Kris Dahl]

Yes, there are 4 events within the /data/crash folder, although I would have expected more.

Examining the *.last files I've found the following, let me know if the entire dump file would be of use.

info.last
Dump header from device /dev/dumpdev
Architecture: amd64
Architecture Version: 1
Dump Length: 121344B (0 MB)
Blocksize: 512
Dumptime: Thu Sep 22 05:31:06 2016
Hostname: freenas.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 9.3-RELEASE-p31 #0 r288272+e7e804d: Mon May 16 21:29:59 PDT 2016
root@build3.ixsystems.com:/tank/home/stable-builds/FN/objs/os-base/amd64/tank/home/stable-builds/FN/FreeBSD/src/sys Panic String:
Dump Parity: 4135627384
Bounds: 2
Dump Status: good

textdump.tar.last
... towards end of file...
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0x18
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff80cfb36a
stack pointer = 0x28:0xfffffe04967f8820
frame pointer = 0x28:0xfffffe04967f8860
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 82852 (python2.7)

root@build3.ixsystems.com:/tank/home/stable-builds/FN/objs/os-base/amd64/tank/home/stable-builds/FN/FreeBSD/src/sys/FREENAS.amd64