S.M.A.R.T. error, ran long test: what to do?

[stoffix]

Two days ago I got an email saying one of my drives failed a S.M.A.R.T. test:
"Device: /dev/ada2, Self-Test Log error count increased from 0 to 1"

I just ran a long test, and as far as I can tell, it completed with no errors:

Code:

root@filtjener:~ # smartctl -a /dev/ada2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD30EFRX-68AX9N0
Serial Number:	WD-WCC1T1305709
LU WWN Device Id: 5 0014ee 25e0497a1
Firmware Version: 80.00A80
User Capacity:	3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Wed Aug  9 00:41:58 2017 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(38580) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   2) minutes.
Extended self-test routine
recommended polling time:		( 387) minutes.
Conveyance self-test routine
recommended polling time:		(   5) minutes.
SCT capabilities:			  (0x70bd) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   46
  3 Spin_Up_Time			0x0027   179   169   021	Pre-fail  Always	   -	   6016
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   48
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   1
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   052   052   000	Old_age   Always	   -	   35181
 10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   48
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   8
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   39
194 Temperature_Celsius	 0x0022   122   109   000	Old_age   Always	   -	   28
196 Reallocated_Event_Count 0x0032   199   199   000	Old_age   Always	   -	   1
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   23

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	 35181		 -
# 2  Short offline	   Completed without error	   00%	 35173		 -
# 3  Short offline	   Completed without error	   00%	 35172		 -
# 4  Short offline	   Completed without error	   00%	 35171		 -
# 5  Short offline	   Completed without error	   00%	 35170		 -
# 6  Short offline	   Completed without error	   00%	 35169		 -
# 7  Short offline	   Completed without error	   00%	 35168		 -
# 8  Short offline	   Completed without error	   00%	 35167		 -
# 9  Short offline	   Completed without error	   00%	 35166		 -
#10  Short offline	   Completed without error	   00%	 35165		 -
#11  Short offline	   Completed without error	   00%	 35164		 -
#12  Short offline	   Completed without error	   00%	 35163		 -
#13  Extended offline	Completed: read failure	   90%	 35161		 309151832
#14  Short offline	   Completed without error	   00%	 35160		 -
#15  Short offline	   Completed without error	   00%	 35159		 -
#16  Short offline	   Completed without error	   00%	 35158		 -
#17  Short offline	   Completed without error	   00%	 35157		 -
#18  Short offline	   Completed without error	   00%	 35156		 -
#19  Short offline	   Completed without error	   00%	 35155		 -
#20  Short offline	   Completed without error	   00%	 35154		 -
#21  Short offline	   Completed without error	   00%	 35153		 -
1 of 1 failed self-tests are outdated by newer successful extended offline self-test # 1

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

When I built this server some years ago I bought a spare drive to have just in case a drive fails, but should I swap it straight away, or could this just be a one time occurance?

Should I wait for a while and see if more errors occurs?

I'm on Freenas 11.0-U2
the motherboard is a supermicro X9 with a Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz CPU.
I have 32GB ECC RAM and my storage consists of 6 x 3TB western digital Red disks in Raid-z2

 

[rs225]

The drive shows 1 relocated sector; that could have been the problem. Also interesting is your multi-zone error rate at 23. Since it is raidz2, I would say just watch it and make sure scrub is running on the expected schedule. But this might become a replace.

Not sure, but it looks like you are running short tests every hour, and long tests every day. That's excessive.

 

[stoffix]

Thanks, I'll keep an eye on it!
Yeah, I was running these tests this often.
after googling around since I got this email, I changed the schedule to short tests once every week, and long tests every other week. Anyway, thanks for pointing it out! :)

 

[Spearfoot]

Thanks, I'll keep an eye on it!
Yeah, I was running these tests this often.
after googling around since I got this email, I changed the schedule to short tests once every week, and long tests every other week. Anyway, thanks for pointing it out! :)

Hourly short tests is excessive, but a weekly interval is probably too long -- especially when you have older disks, one of which is already showing signs of impending failure.

I suggest running the short tests daily and the long tests weekly.

 

[danb35]

I suggest running the short tests daily and the long tests weekly.

That's my schedule, and it seems like a good balance.

 

[Spearfoot]

That's my schedule, and it seems like a good balance.

Great minds think alike! :D

 

[stoffix]

Thanks!
I've been away the last week, but I just changed my schedule to daily short tests, and weekly long tests.
I ran another smartctl on my disk (all of them) and there haven't been any more errors, so I think I'll trust the updated schedule to notice me in time if something happens. I already have a new disk ready for if/when I need to swap it ;)

 

[Stux]

I run my short tests on Monday and Wenesday mornings. Gives me two+ business days to organize a replacement.

Long tests forntightly over the weekends (it's a business server, so this doesn't impact performance)

 

[farmerpling2]

You can always replace the drive with your spare. run badblock intensively on it (2-3 days) and see what comes out.

If badblock is happy then you are probably fine. You can always add the spare as a "spare" if you so wish.

 

[stoffix]

I'm not sure I can do that without degrading my pool. I don't have any free sata ports in my file server, and my other computer with sata ports runs windows.

I guess I could unload the disk and run badblocks on it while the pool is in a degraded state couldn't I? Although I'm not sure I want to do that.

 

[farmerpling2]

So your choice is to wait until a failure and hopefully you will notice it quickly ad replace the drive

or...

You can be proactive. Use a spare to replace and take the drive over to your window machine and run the manufacturers diags on it and see if their software finds anything.

I prefer to control my destiny and would suggest that you should too. ;)