Web GUI (nginx) hung?

[rmccullough]

My setup:
FreeNAS 11.2 (BETA2)
12-bay Supermicro CSE-826A-R1200LPB
2x 920Watt Power Supply PWS-920P-1R Platinum
Supermicro X9DRi-LN4F+

• 2x Intel Xeon E5-2620 v1 HEx (6) Core @ 2.0GHz
  
• 4x Intel® i350 GbE controller
• 32GB ECC PC3-10600

32GB SATADOM (boot drive)
2x LSI 9210-8i
9 x 2TB SAS 6GB/S Hitachi GST ULTRASTAR


I think I have another issue (drive dying), but the WebGUI became unresponsive today. I read that I can restart the services using these commands:
service nginx restart
service django restart

However, it seemed to hang while trying to restart the pid (3193) for nginx. I also tried a kill -9 3193, but it did not seem to make a difference. So I connected using IMPI and tried to reboot. This also hung waiting for pid 3193. I finally used IPMI to do a power reset. However, I wanted to know if there was something else I should have tried?

 

[Ericloewe]

You might want to make sure your boot device isn't crapping out on you. This is a frequent symptom and some SATA DOMs are pretty nasty when it comes to reliability.

 

[rmccullough]

@Ericloewe any advice on how I can test my SATA DOM boot device? Thanks.

 

[Ericloewe]

If it reports SMART data, have a look at it.

Make sure you have an aggressive scrub schedule for the boot pool, too.

 

[rmccullough]

Here is my boot scrub:

Code:

Boot Pool Condition : ONLINE
Size: 29.5G
Used: 872M(2%)
Last Scrub Run on: September 8th 2018, 3:45:11 am
Automatic scrub interval (in days)
7

Regarding the SMART data for the SATA DOM, I don't really know what to look for but this looks ok to me other than that it has limited SMART capabilities:

Code:

freenas# smartctl -a /dev/ada0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:	 SATADOM-SV 3SE
Serial Number:	20150907AA0853999054
Firmware Version: S130710
User Capacity:	32,017,047,552 bytes [32.0 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Sat Sep  8 19:01:42 2018 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Disabled.
Total time to complete Offline
data collection:				(   32) seconds.
Offline data collection
capabilities:					(0x00)		 Offline data collection not supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x00) Error logging NOT supported.
										General Purpose Logging supported.
SCT capabilities:			  (0x0039) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x0000   000   000   000	Old_age   Offline	  -	   0
  2 Throughput_Performance  0x0000   000   000   000	Old_age   Offline	  -	   0
  3 Spin_Up_Time			0x0000   000   000   000	Old_age   Offline	  -	   0
  5 Reallocated_Sector_Ct   0x0000   000   000   000	Old_age   Offline	  -	   0
  7 Unknown_SSD_Attribute   0x0000   000   000   000	Old_age   Offline	  -	   0
  8 Unknown_SSD_Attribute   0x0000   000   000   000	Old_age   Offline	  -	   0
  9 Power_On_Hours		  0x0000   000   000   000	Old_age   Offline	  -	   367
 10 Unknown_SSD_Attribute   0x0000   000   000   000	Old_age   Offline	  -	   0
 12 Power_Cycle_Count	   0x0000   000   000   000	Old_age   Offline	  -	   39
  1 Raw_Read_Error_Rate	 0x0000   000   000   000	Old_age   Offline	  -	   0
168 Unknown_Attribute	   0x0000   000   000   000	Old_age   Offline	  -	   0
169 Unknown_Attribute	   0x0000   000   000   000	Old_age   Offline	  -	   0
  1 Raw_Read_Error_Rate	 0x0000   000   000   000	Old_age   Offline	  -	   0
175 Program_Fail_Count_Chip 0x0000   100   000   000	Old_age   Offline	  -	   0
192 Power-Off_Retract_Count 0x0000   000   000   000	Old_age   Offline	  -	   0
194 Temperature_Celsius	 0x0000   036   100   000	Old_age   Offline	  -	   36 (3 100 0 0 0)
197 Current_Pending_Sector  0x0000   000   000   000	Old_age   Offline	  -	   0
240 Unknown_SSD_Attribute   0x0000   000   000   000	Old_age   Offline	  -	   0
170 Unknown_Attribute	   0x0003   100   100   ---	Pre-fail  Always	   -	   1703936
173 Unknown_Attribute	   0x0012   100   100   ---	Old_age   Always	   -	   589825
229 Unknown_Attribute	   0x0002   100   100   ---	Old_age   Always	   -	   8830595015064
236 Unknown_Attribute	   0x0002   100   100   ---	Old_age   Always	   -	   0
235 Unknown_Attribute	   0x0002   100   000   ---	Old_age   Always	   -	   0
176 Erase_Fail_Count_Chip   0x0000   100   000   ---	Old_age   Offline	  -	   0

Read SMART Log Directory failed: Input/output error

SMART Error Log not supported

SMART Self-test Log not supported

Selective Self-tests/Logging not supported

I also tried to run smartctl -t short /dev/ada0 but it reported an error that there is an existing test with 150% remaining:

Code:

freenas# smartctl -t short /dev/ada0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Self-test functions not supported

Can't start self-test without aborting current test (150% remaining),
add '-t force' option to override, or run 'smartctl -X' to abort test.

I tried to force it and it still didn't seem to want to go.