Critical error but I cannot find in GUI

Status
Not open for further replies.

mguebert

Dabbler
Joined
Dec 30, 2015
Messages
17
I am seeing a critical error on DA 3, however when I look at most recent scrub status or volume status everything shows healthy. So I cannot determine, do I have a issue or not? I tried to reboot and still see the error in the dashboard.

7-20-2017 1-13-05 PM.jpg
7-20-2017 1-15-11 PM.jpg
7-20-2017 1-13-55 PM.jpg
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
You clearly have an issue. Its just that it hasn't (visibly) affected ZFS yet. That drive is on the way out, so replace it.

For our enlightenment, feel free to post the output of smartctl -x /dev/da3 - [CODE][/CODE] tags please!

To replace the drive, follow the instructions in the manual: http://doc.freenas.org/11/storage.html#replacing-a-failed-drive
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
Run a smart test from the command line on ada3. I would do a short one - it only takes a couple of minutes, followed by a long one.

Search the forums for the syntax.

Post the results here in code tags if you would like us to interpret the results.
 

mguebert

Dabbler
Joined
Dec 30, 2015
Messages
17
Thanks, here is the smart test data

Code:

Shell
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)															
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org														
																																  
=== START OF INFORMATION SECTION ===																								
Model Family:	 Seagate Barracuda 7200.14 (AF)																					
Device Model:	 ST3000DM001-1CH166																								
Serial Number:	Z1F3M630																										 
LU WWN Device Id: 5 000c50 06475a267																								
Firmware Version: CC27																											 
User Capacity:	3,000,592,982,016 bytes [3.00 TB]																				
Sector Sizes:	 512 bytes logical, 4096 bytes physical																			
Rotation Rate:	7200 rpm																										 
Form Factor:	  3.5 inches																										
Device is:		In smartctl database [for details use: -P show]																 
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b																			 
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)																			
Local Time is:	Thu Jul 20 15:00:48 2017 CDT																					 
SMART support is: Available - device has SMART capability.																		 
SMART support is: Enabled																										 
AAM feature is:   Unavailable																									 
APM feature is:   Disabled																										 
Rd look-ahead is: Enabled																										 
Write cache is:   Enabled																										 
ATA Security is:  Disabled, NOT FROZEN [SEC1]																					 
Wt Cache Reorder: Unavailable																									 
																																  
=== START OF READ SMART DATA SECTION ===																							
SMART overall-health self-assessment test result: PASSED																			
																																  
General SMART Values:																											 
Offline data collection status:  (0x82) Offline data collection activity															
										was completed without error.																
										Auto Offline Data Collection: Enabled.													 
Self-test execution status:	  (   0) The previous self-test routine completed													
										without error or no self-test has ever													 
										been run.																				 
Total time to complete Offline																									 
data collection:				(  592) seconds.																					
Offline data collection																											
capabilities:					(0x7b) SMART execute Offline immediate.															
										Auto Offline data collection on/off support.												
										Suspend Offline collection upon new														
										command.																					
										Offline surface scan supported.															
										Self-test supported.																		
										Conveyance Self-test supported.															
										Selective Self-test supported.															 
SMART capabilities:			(0x0003) Saves SMART data before entering															
										power-saving mode

Shell
										Supports SMART auto save timer.															
Error logging capability:		(0x01) Error logging supported.																	
										General Purpose Logging supported.														 
Short self-test routine																											
recommended polling time:		(   1) minutes.																					
Extended self-test routine																										 
recommended polling time:		( 356) minutes.																					
Conveyance self-test routine																										
recommended polling time:		(   2) minutes.																					
SCT capabilities:			  (0x3085) SCT Status supported.																	 
																																  
SMART Attributes Data Structure revision number: 10																				
Vendor Specific SMART Attributes with Thresholds:																				 
ID# ATTRIBUTE_NAME		  FLAGS	VALUE WORST THRESH FAIL RAW_VALUE															 
  1 Raw_Read_Error_Rate	 POSR--   114   099   006	-	77288640															 
  3 Spin_Up_Time			PO----   094   094   000	-	0																	 
  4 Start_Stop_Count		-O--CK   100   100   020	-	100																	
  5 Reallocated_Sector_Ct   PO--CK   100   100   010	-	0																	 
  7 Seek_Error_Rate		 POSR--   068   055   030	-	55925873423															
  9 Power_On_Hours		  -O--CK   062   062   000	-	33417																 
10 Spin_Retry_Count		PO--C-   100   100   097	-	0																	 
12 Power_Cycle_Count	   -O--CK   100   100   020	-	58																	
183 Runtime_Bad_Block	   -O--CK   100   100   000	-	0																	 
184 End-to-End_Error		-O--CK   100   100   099	-	0																	 
187 Reported_Uncorrect	  -O--CK   100   100   000	-	0																	 
188 Command_Timeout		 -O--CK   100   100   000	-	0 0 0																 
189 High_Fly_Writes		 -O-RCK   081   081   000	-	19																	
190 Airflow_Temperature_Cel -O---K   064   060   045	-	36 (Min/Max 25/39)													
191 G-Sense_Error_Rate	  -O--CK   100   100   000	-	0																	 
192 Power-Off_Retract_Count -O--CK   100   100   000	-	54																	
193 Load_Cycle_Count		-O--CK   075   075   000	-	50960																 
194 Temperature_Celsius	 -O---K   036   040   000	-	36 (0 15 0 0 0)														
197 Current_Pending_Sector  -O--C-   100   100   000	-	40																	
198 Offline_Uncorrectable   ----C-   100   100   000	-	40																	
199 UDMA_CRC_Error_Count	-OSRCK   200   200   000	-	0																	 
240 Head_Flying_Hours	   ------   100   253   000	-	14917h+56m+06.119s													
241 Total_LBAs_Written	  ------   100   253   000	-	70010988218															
242 Total_LBAs_Read		 ------   100   253   000	-	382707157290														 
							||||||_ K auto-keep																					
							|||||__ C event count																				 
							||||___ R error rate																					
							|||____ S speed/performance																			
							||_____ O updated online																				
							|______ P prefailure warning																			
																																  
General Purpose Log Directory Version 1																							
SMART		   Log Directory Version 1 [multi-sector log support]																 
Address	Access  R/W   Size  Description																						 
0x00	   GPL,SL  R/O	  1  Log Directory
0x02		   SL  R/O	  5  Comprehensive SMART error log																		
0x03	   GPL	 R/O	  5  Ext. Comprehensive SMART error log																 
0x06		   SL  R/O	  1  SMART self-test log																				 
0x07	   GPL	 R/O	  1  Extended self-test log																			 
0x09		   SL  R/W	  1  Selective self-test log																			 
0x10	   GPL	 R/O	  1  SATA NCQ Queued Error log																			
0x11	   GPL	 R/O	  1  SATA Phy Event Counters log																		 
0x21	   GPL	 R/O	  1  Write stream error log																			 
0x22	   GPL	 R/O	  1  Read stream error log																				
0x80-0x9f  GPL,SL  R/W	 16  Host vendor specific log																			
0xa1	   GPL,SL  VS	  20  Device vendor specific log																		 
0xa2	   GPL	 VS	4496  Device vendor specific log																		 
0xa8	   GPL,SL  VS	 129  Device vendor specific log																		 
0xa9	   GPL,SL  VS	   1  Device vendor specific log																		 
0xab	   GPL	 VS	   1  Device vendor specific log																		 
0xb0	   GPL	 VS	5176  Device vendor specific log																		 
0xbe-0xbf  GPL	 VS   65535  Device vendor specific log																		 
0xc0	   GPL,SL  VS	   1  Device vendor specific log																		 
0xc1	   GPL,SL  VS	  10  Device vendor specific log																		 
0xc4	   GPL,SL  VS	   5  Device vendor specific log																		 
0xe0	   GPL,SL  R/W	  1  SCT Command/Status																				 
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer																					
																																  
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)																	 
No Errors Logged																													
																																  
SMART Extended Self-test Log Version: 1 (1 sectors)																				
No self-tests have been logged.  [To run self-tests, use: smartctl -t]															 
																																  
SMART Selective self-test log data structure revision number 1																	 
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS																						
	1		0		0  Not_testing																								
	2		0		0  Not_testing																								
	3		0		0  Not_testing																								
	4		0		0  Not_testing																								
	5		0		0  Not_testing																								
Selective self-test flags (0x0):																									
  After scanning selected spans, do NOT read-scan remainder of disk.																
If Selective self-test is pending on power-up, resume after 0 minute delay.														
																																  
SCT Status Version:				  3																							 
SCT Version (vendor specific):	   522 (0x020a)																				 
SCT Support Level:				   1																							 
Device State:						Active (0)																					
Current Temperature:					36 Celsius																				 
Power Cycle Min/Max Temperature:	 25/39 Celsius																				 
Lifetime	Min/Max Temperature:	 15/40 Celsius																				 
Under/Over Temperature Limit Count:   0/0 
0xb0	   GPL	 VS	5176  Device vendor specific log																		 
0xbe-0xbf  GPL	 VS   65535  Device vendor specific log																		 
0xc0	   GPL,SL  VS	   1  Device vendor specific log																		 
0xc1	   GPL,SL  VS	  10  Device vendor specific log																		 
0xc4	   GPL,SL  VS	   5  Device vendor specific log																		 
0xe0	   GPL,SL  R/W	  1  SCT Command/Status																				 
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer																					
																																  
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)																	 
No Errors Logged																													
																																  
SMART Extended Self-test Log Version: 1 (1 sectors)																				
No self-tests have been logged.  [To run self-tests, use: smartctl -t]															 
																																  
SMART Selective self-test log data structure revision number 1																	 
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS																						
	1		0		0  Not_testing																								
	2		0		0  Not_testing																								
	3		0		0  Not_testing																								
	4		0		0  Not_testing																								
	5		0		0  Not_testing																								
Selective self-test flags (0x0):																									
  After scanning selected spans, do NOT read-scan remainder of disk.																
If Selective self-test is pending on power-up, resume after 0 minute delay.														
																																  
SCT Status Version:				  3																							 
SCT Version (vendor specific):	   522 (0x020a)																				 
SCT Support Level:				   1																							 
Device State:						Active (0)																					
Current Temperature:					36 Celsius																				 
Power Cycle Min/Max Temperature:	 25/39 Celsius																				 
Lifetime	Min/Max Temperature:	 15/40 Celsius																				 
Under/Over Temperature Limit Count:   0/0																						 
																																  
SCT Data Table command not supported																								
																																  
SCT Error Recovery Control command not supported																					
																																  
Device Statistics (GP/SMART Log 0x04) not supported																				
																																  
SATA Phy Event Counters (GP Log 0x11)																							 
ID	  Size	 Value  Description																								
0x000a  2		   16  Device-to-host register FISes sent due to a COMRESET														
0x0001  2			0  Command failed due to ICRC error																			
0x0003  2			0  R_ERR response for device-to-host data FIS																 
0x0004  2			0  R_ERR response for host-to-device data FIS																 
0x0006  2			0  R_ERR response for device-to-host non-data FIS															 
0x0007  2			0  R_ERR response for host-to-device non-data FIS															 
																	  
 
Last edited by a moderator:

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
ID # 197 and 198 correspond to the messages you included in your OP.

It does look like you ran any self tests in the nearly 4 years that the drive has been powered on.

The documentation explains how to configure self tests to run automatically on a schedule.

As @Ericloewe said earlier, we recommend that you replace the drive.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thanks, here is the smart test data

Code:

Shell
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)															
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org														
																																
=== START OF INFORMATION SECTION ===																								
Model Family:	 Seagate Barracuda 7200.14 (AF)																					
Device Model:	 ST3000DM001-1CH166																								
Serial Number:	Z1F3M630																										
LU WWN Device Id: 5 000c50 06475a267																								
Firmware Version: CC27																											
User Capacity:	3,000,592,982,016 bytes [3.00 TB]																				
Sector Sizes:	 512 bytes logical, 4096 bytes physical																			
Rotation Rate:	7200 rpm																										
Form Factor:	  3.5 inches																										
Device is:		In smartctl database [for details use: -P show]																
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b																			
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)																			
Local Time is:	Thu Jul 20 15:00:48 2017 CDT																					
SMART support is: Available - device has SMART capability.																		
SMART support is: Enabled																										
AAM feature is:   Unavailable																									
APM feature is:   Disabled																										
Rd look-ahead is: Enabled																										
Write cache is:   Enabled																										
ATA Security is:  Disabled, NOT FROZEN [SEC1]																					
Wt Cache Reorder: Unavailable																									
																																
=== START OF READ SMART DATA SECTION ===																							
SMART overall-health self-assessment test result: PASSED																			
																																
General SMART Values:																											
Offline data collection status:  (0x82) Offline data collection activity															
										was completed without error.																
										Auto Offline Data Collection: Enabled.													
Self-test execution status:	  (   0) The previous self-test routine completed													
										without error or no self-test has ever													
										been run.																				
Total time to complete Offline																									
data collection:				(  592) seconds.																					
Offline data collection																											
capabilities:					(0x7b) SMART execute Offline immediate.															
										Auto Offline data collection on/off support.												
										Suspend Offline collection upon new														
										command.																					
										Offline surface scan supported.															
										Self-test supported.																		
										Conveyance Self-test supported.															
										Selective Self-test supported.															
SMART capabilities:			(0x0003) Saves SMART data before entering															
										power-saving mode

Shell
										Supports SMART auto save timer.															
Error logging capability:		(0x01) Error logging supported.																	
										General Purpose Logging supported.														
Short self-test routine																											
recommended polling time:		(   1) minutes.																					
Extended self-test routine																										
recommended polling time:		( 356) minutes.																					
Conveyance self-test routine																										
recommended polling time:		(   2) minutes.																					
SCT capabilities:			  (0x3085) SCT Status supported.																	
																																
SMART Attributes Data Structure revision number: 10																				
Vendor Specific SMART Attributes with Thresholds:																				
ID# ATTRIBUTE_NAME		  FLAGS	VALUE WORST THRESH FAIL RAW_VALUE															
  1 Raw_Read_Error_Rate	 POSR--   114   099   006	-	77288640															
  3 Spin_Up_Time			PO----   094   094   000	-	0																	
  4 Start_Stop_Count		-O--CK   100   100   020	-	100																	
  5 Reallocated_Sector_Ct   PO--CK   100   100   010	-	0																	
  7 Seek_Error_Rate		 POSR--   068   055   030	-	55925873423															
  9 Power_On_Hours		  -O--CK   062   062   000	-	33417																
10 Spin_Retry_Count		PO--C-   100   100   097	-	0																	
12 Power_Cycle_Count	   -O--CK   100   100   020	-	58																	
183 Runtime_Bad_Block	   -O--CK   100   100   000	-	0																	
184 End-to-End_Error		-O--CK   100   100   099	-	0																	
187 Reported_Uncorrect	  -O--CK   100   100   000	-	0																	
188 Command_Timeout		 -O--CK   100   100   000	-	0 0 0																
189 High_Fly_Writes		 -O-RCK   081   081   000	-	19																	
190 Airflow_Temperature_Cel -O---K   064   060   045	-	36 (Min/Max 25/39)													
191 G-Sense_Error_Rate	  -O--CK   100   100   000	-	0																	
192 Power-Off_Retract_Count -O--CK   100   100   000	-	54																	
193 Load_Cycle_Count		-O--CK   075   075   000	-	50960																
194 Temperature_Celsius	 -O---K   036   040   000	-	36 (0 15 0 0 0)														
197 Current_Pending_Sector  -O--C-   100   100   000	-	40																	
198 Offline_Uncorrectable   ----C-   100   100   000	-	40																	
199 UDMA_CRC_Error_Count	-OSRCK   200   200   000	-	0																	
240 Head_Flying_Hours	   ------   100   253   000	-	14917h+56m+06.119s													
241 Total_LBAs_Written	  ------   100   253   000	-	70010988218															
242 Total_LBAs_Read		 ------   100   253   000	-	382707157290														
							||||||_ K auto-keep																					
							|||||__ C event count																				
							||||___ R error rate																					
							|||____ S speed/performance																			
							||_____ O updated online																				
							|______ P prefailure warning																			
																																
General Purpose Log Directory Version 1																							
SMART		   Log Directory Version 1 [multi-sector log support]																
Address	Access  R/W   Size  Description																						
0x00	   GPL,SL  R/O	  1  Log Directory
0x02		   SL  R/O	  5  Comprehensive SMART error log																		
0x03	   GPL	 R/O	  5  Ext. Comprehensive SMART error log																
0x06		   SL  R/O	  1  SMART self-test log																				
0x07	   GPL	 R/O	  1  Extended self-test log																			
0x09		   SL  R/W	  1  Selective self-test log																			
0x10	   GPL	 R/O	  1  SATA NCQ Queued Error log																			
0x11	   GPL	 R/O	  1  SATA Phy Event Counters log																		
0x21	   GPL	 R/O	  1  Write stream error log																			
0x22	   GPL	 R/O	  1  Read stream error log																				
0x80-0x9f  GPL,SL  R/W	 16  Host vendor specific log																			
0xa1	   GPL,SL  VS	  20  Device vendor specific log																		
0xa2	   GPL	 VS	4496  Device vendor specific log																		
0xa8	   GPL,SL  VS	 129  Device vendor specific log																		
0xa9	   GPL,SL  VS	   1  Device vendor specific log																		
0xab	   GPL	 VS	   1  Device vendor specific log																		
0xb0	   GPL	 VS	5176  Device vendor specific log																		
0xbe-0xbf  GPL	 VS   65535  Device vendor specific log																		
0xc0	   GPL,SL  VS	   1  Device vendor specific log																		
0xc1	   GPL,SL  VS	  10  Device vendor specific log																		
0xc4	   GPL,SL  VS	   5  Device vendor specific log																		
0xe0	   GPL,SL  R/W	  1  SCT Command/Status																				
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer																					
																																
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)																	
No Errors Logged																													
																																
SMART Extended Self-test Log Version: 1 (1 sectors)																				
No self-tests have been logged.  [To run self-tests, use: smartctl -t]															
																																
SMART Selective self-test log data structure revision number 1																	
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS																						
	1		0		0  Not_testing																								
	2		0		0  Not_testing																								
	3		0		0  Not_testing																								
	4		0		0  Not_testing																								
	5		0		0  Not_testing																								
Selective self-test flags (0x0):																									
  After scanning selected spans, do NOT read-scan remainder of disk.																
If Selective self-test is pending on power-up, resume after 0 minute delay.														
																																
SCT Status Version:				  3																							
SCT Version (vendor specific):	   522 (0x020a)																				
SCT Support Level:				   1																							
Device State:						Active (0)																					
Current Temperature:					36 Celsius																				
Power Cycle Min/Max Temperature:	 25/39 Celsius																				
Lifetime	Min/Max Temperature:	 15/40 Celsius																				
Under/Over Temperature Limit Count:   0/0
0xb0	   GPL	 VS	5176  Device vendor specific log																		
0xbe-0xbf  GPL	 VS   65535  Device vendor specific log																		
0xc0	   GPL,SL  VS	   1  Device vendor specific log																		
0xc1	   GPL,SL  VS	  10  Device vendor specific log																		
0xc4	   GPL,SL  VS	   5  Device vendor specific log																		
0xe0	   GPL,SL  R/W	  1  SCT Command/Status																				
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer																					
																																
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)																	
No Errors Logged																													
																																
SMART Extended Self-test Log Version: 1 (1 sectors)																				
No self-tests have been logged.  [To run self-tests, use: smartctl -t]															
																																
SMART Selective self-test log data structure revision number 1																	
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS																						
	1		0		0  Not_testing																								
	2		0		0  Not_testing																								
	3		0		0  Not_testing																								
	4		0		0  Not_testing																								
	5		0		0  Not_testing																								
Selective self-test flags (0x0):																									
  After scanning selected spans, do NOT read-scan remainder of disk.																
If Selective self-test is pending on power-up, resume after 0 minute delay.														
																																
SCT Status Version:				  3																							
SCT Version (vendor specific):	   522 (0x020a)																				
SCT Support Level:				   1																							
Device State:						Active (0)																					
Current Temperature:					36 Celsius																				
Power Cycle Min/Max Temperature:	 25/39 Celsius																				
Lifetime	Min/Max Temperature:	 15/40 Celsius																				
Under/Over Temperature Limit Count:   0/0																						
																																
SCT Data Table command not supported																								
																																
SCT Error Recovery Control command not supported																					
																																
Device Statistics (GP/SMART Log 0x04) not supported																				
																																
SATA Phy Event Counters (GP Log 0x11)																							
ID	  Size	 Value  Description																								
0x000a  2		   16  Device-to-host register FISes sent due to a COMRESET														
0x0001  2			0  Command failed due to ICRC error																			
0x0003  2			0  R_ERR response for device-to-host data FIS																
0x0004  2			0  R_ERR response for host-to-device data FIS																
0x0006  2			0  R_ERR response for device-to-host non-data FIS															
0x0007  2			0  R_ERR response for host-to-device non-data FIS															
																	
This drive hasn't failed yet, but it has these problems:
Code:
197 Current_Pending_Sector  -O--C-   100   100   000	-	40																	
198 Offline_Uncorrectable   ----C-   100   100   000	-	40 
...which is why FreeNAS issued the alert.

This drive is nearly 4 years old and has never been tested. If your other drives are of similar age, you need to check them ASAP as you may very well have additional drives showing early signs of nearing their end-of-life. If more than two disks fail in either of your RAIDZ2 vdevs you will lose all of your data!

it would be a really good idea to schedule both short and long SMART tests on all of your disks. In the WebGUI, go to Tasks->S.M.A.R.T. Tests to set these up.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
The drive also has a fairly high load cycle count, so it's being spun down. This is often harder on the drive than simply letting it spin.
 
Status
Not open for further replies.
Top