Replaced a failing drive, but still have a failed pool

Joined
Aug 5, 2013
Messages
2
SUPERMICRO MBD-X10SL7-F-O
32 GB ECC 4x Crucial 8 GB 240-Pin DDR3 SDRAM unbuffered ECC
EVGA SuperNOVA 1000 P2 220-P2-1000-XR 80+ Platnium 1000W


Hi all,

I love my freenas server and have no problems with it for years. I really like the thought and effort put into the web interface and all of the improvements.
I double-checked the documentation first and checked a couple of posts on what to do if a drive fails. I was getting a lot of "READ FPDMA QUEUED" from ata1 errors on my console screen.

I ran a smartctl -x /dev/ada1 report on my failing drive. The drive had a high read error rate so I replaced it with another HGST Deskstar Nas 4 TB drive. I set the pool to resilver, and I still have a degraded pool. When I check the option to replace /dev/gptid/815852e8-d05c-11e6-97c6-0cc47a32cde6

the option says Replacing disk /dev/g Member disk and a selector. No specific drive mentioned. I am worried I have messed up my pool somehow.

What I see on Pool Status:
RESILVER
Status: FINISHED
Errors: 0


RAIDZ2 0 0 0 DEGRADED
da4p2 0 0 0 ONLINE more_vert
da2p2 0 0 0 ONLINE more_vert
ada1p2 0 0 0 ONLINE more_vert
da3p2 0 0 0 ONLINE more_vert
/dev/gptid/815852e8-d05c-11e6-97c6-0cc47a32cde6 0 0 0 UNAVAIL more_vert
da0p2 0 0 0 ONLINE more_vert
da1p2 0 0 0 ONLINE more_vert
ada0p2 0 0 0 ONLINE more_vert
da5p2 0 0 0 ONLINE more_vert
da6p2 0 0 0 ONLINE more_vert



My setup:
FreeNAS 11.2- Release-U1
Intel Xeon CPU E3-1231v3

My Drives are 10xHGST Deskstar NAS HDD 4 TB drives
SUPERMICRO MBD-X10SL7-F-O
32 GB ECC 4x Crucial 8 GB 240-Pin DDR3 SDRAM unbuffered ECC
EVGA SuperNOVA 1000 P2 220-P2-1000-XR 80+ Platinum 1000W


Running a smartctl -x /dev/ata1 gives the following info.. this is on a "new" drive:

root@freenas:/media/Movies # smartctl -x /dev/ada1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: HGST Deskstar NAS
Device Model: HGST HDN724040ALE640
Serial Number: PK2338P4HDW2DC
LU WWN Device Id: 5 000cca 249d3f19a
Firmware Version: MJAOA5E0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Aug 6 17:36:39 2019 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 24) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 567) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 095 095 016 - 393220
2 Throughput_Performance P-S--- 136 136 054 - 80
3 Spin_Up_Time POS--- 125 125 024 - 621 (Average 613)
4 Start_Stop_Count -O--C- 100 100 000 - 233
5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0
7 Seek_Error_Rate PO-R-- 100 100 067 - 0
8 Seek_Time_Performance P-S--- 119 119 020 - 35
9 Power_On_Hours -O--C- 098 098 000 - 18013
10 Spin_Retry_Count PO--C- 100 100 060 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 180
192 Power-Off_Retract_Count -O--CK 095 095 000 - 7196
193 Load_Cycle_Count -O--C- 095 095 000 - 7196
194 Temperature_Celsius -O---- 157 157 000 - 38 (Min/Max 15/59)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 20
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning

General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 7 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x20 GPL R/O 1 Streaming performance log [OBS-8]
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
Device Error Count: 20 (device log contains only the most recent 4 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 20 [3] occurred at disk power-on lifetime: 17999 hours (749 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
84 -- 51 00 11 00 01 a8 91 f9 87 08 00 Error: ICRC, ABRT at LBA = 0x1a891f987 = 7123106183

Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 20 00 10 00 01 a8 91 f9 88 48 ff 1d+16:37:34.712 READ FPDMA QUEUED
60 00 20 00 c0 00 01 a8 91 f9 78 40 00 1d+16:37:34.711 READ FPDMA QUEUED
60 00 20 00 b8 00 01 a8 91 f9 b8 40 00 1d+16:37:34.689 READ FPDMA QUEUED
60 00 20 00 b0 00 01 a8 91 f9 58 40 00 1d+16:37:34.689 READ FPDMA QUEUED
60 00 20 00 a8 00 01 a8 91 fa 00 40 00 1d+16:37:34.688 READ FPDMA QUEUED

Error 19 [2] occurred at disk power-on lifetime: 17999 hours (749 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
84 -- 51 00 01 00 01 0a c5 01 4f 0a 00 Error: ICRC, ABRT at LBA = 0x10ac5014f = 4475650383

Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 10 00 00 00 01 0a c5 01 4f 4a ff 1d+15:40:41.413 READ FPDMA QUEUED
60 00 10 00 c0 00 01 0a c5 01 40 40 00 1d+15:40:41.412 READ FPDMA QUEUED
60 00 20 00 b8 00 01 0a c3 8e 30 40 00 1d+15:40:41.391 READ FPDMA QUEUED
60 00 08 00 b0 00 01 0a c4 50 00 40 00 1d+15:40:41.373 READ FPDMA QUEUED
ea 00 00 00 00 00 00 00 00 00 00 40 00 1d+15:40:41.324 FLUSH CACHE EXT

Error 18 [1] occurred at disk power-on lifetime: 17999 hours (749 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
84 -- 51 00 09 00 00 f4 c4 6f 97 04 00 Error: ICRC, ABRT at LBA = 0xf4c46f97 = 4106514327

Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 18 00 08 00 00 f4 c4 6f 98 44 ff 1d+15:40:29.455 READ FPDMA QUEUED
60 00 18 00 80 00 00 f4 c4 6f 88 40 00 1d+15:40:29.454 READ FPDMA QUEUED
60 00 18 00 78 00 00 f4 c4 6f 58 40 00 1d+15:40:29.449 READ FPDMA QUEUED
60 00 18 00 70 00 00 f4 c4 6e f8 40 00 1d+15:40:29.440 READ FPDMA QUEUED
60 00 08 00 68 00 00 f4 c4 6e b0 40 00 1d+15:40:29.431 READ FPDMA QUEUED

Error 17 [0] occurred at disk power-on lifetime: 17998 hours (749 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
84 -- 51 00 01 00 00 f6 ab 26 d7 06 00 Error: ICRC, ABRT at LBA = 0xf6ab26d7 = 4138411735

Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 10 00 00 00 00 f6 ab 26 d7 46 ff 1d+15:36:40.632 READ FPDMA QUEUED
60 00 10 00 48 00 00 f6 ab 26 c8 40 00 1d+15:36:40.631 READ FPDMA QUEUED
60 00 20 00 40 00 00 f9 cf 71 78 40 00 1d+15:36:40.459 READ FPDMA QUEUED
60 00 08 00 38 00 00 f6 a7 89 08 40 00 1d+15:36:40.449 READ FPDMA QUEUED
60 00 08 00 30 00 00 f6 a8 e3 d0 40 00 1d+15:36:40.445 READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 38 Celsius
Power Cycle Min/Max Temperature: 26/40 Celsius
Lifetime Min/Max Temperature: 15/59 Celsius
Under/Over Temperature Limit Count: 0/0

SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (125)

Index Estimated Time Temperature Celsius
126 2019-08-06 15:29 36 *****************
... ..( 4 skipped). .. *****************
3 2019-08-06 15:34 36 *****************
4 2019-08-06 15:35 37 ******************
5 2019-08-06 15:36 36 *****************
... ..( 14 skipped). .. *****************
20 2019-08-06 15:51 36 *****************
21 2019-08-06 15:52 37 ******************
22 2019-08-06 15:53 36 *****************
23 2019-08-06 15:54 37 ******************
24 2019-08-06 15:55 37 ******************
25 2019-08-06 15:56 36 *****************
26 2019-08-06 15:57 36 *****************
27 2019-08-06 15:58 37 ******************
28 2019-08-06 15:59 37 ******************
29 2019-08-06 16:00 37 ******************
30 2019-08-06 16:01 36 *****************
31 2019-08-06 16:02 36 *****************
32 2019-08-06 16:03 36 *****************
33 2019-08-06 16:04 37 ******************
34 2019-08-06 16:05 37 ******************
35 2019-08-06 16:06 36 *****************
... ..( 21 skipped). .. *****************
57 2019-08-06 16:28 36 *****************
58 2019-08-06 16:29 37 ******************
59 2019-08-06 16:30 36 *****************
... ..( 11 skipped). .. *****************
71 2019-08-06 16:42 36 *****************
72 2019-08-06 16:43 35 ****************
... ..( 17 skipped). .. ****************
90 2019-08-06 17:01 35 ****************
91 2019-08-06 17:02 36 *****************
... ..( 7 skipped). .. *****************
99 2019-08-06 17:10 36 *****************
100 2019-08-06 17:11 37 ******************
... ..( 7 skipped). .. ******************
108 2019-08-06 17:19 37 ******************
109 2019-08-06 17:20 38 *******************
... ..( 15 skipped). .. *******************
125 2019-08-06 17:36 38 *******************

SCT Error Recovery Control:
Read: Disabled
Write: Disabled

Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 2) ==
0x01 0x008 4 180 --- Lifetime Power-On Resets
0x01 0x018 6 70185524653 --- Logical Sectors Written
0x01 0x020 6 313432072 --- Number of Write Commands
0x01 0x028 6 76480132593 --- Logical Sectors Read
0x01 0x030 6 468503617 --- Number of Read Commands
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 8234 --- Spindle Motor Power-on Hours
0x03 0x010 4 8233 --- Head Flying Hours
0x03 0x018 4 7196 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 22 --- Read Recovery Attempts
0x03 0x030 4 0 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 464 --- Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 39 --- Current Temperature
0x05 0x010 1 36 N-- Average Short Term Temperature
0x05 0x018 1 35 N-- Average Long Term Temperature
0x05 0x020 1 59 --- Highest Temperature
0x05 0x028 1 15 --- Lowest Temperature
0x05 0x030 1 51 N-- Highest Average Short Term Temperature
0x05 0x038 1 25 N-- Lowest Average Short Term Temperature
0x05 0x040 1 36 N-- Highest Average Long Term Temperature
0x05 0x048 1 25 N-- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 2576 --- Number of Hardware Resets
0x06 0x010 4 2310 --- Number of ASR Events
0x06 0x018 4 20 --- Number of Interface CRC Errors
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 613 R_ERR response for data FIS
0x0003 2 613 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 4 R_ERR response for non-data FIS
0x0006 2 4 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0009 2 621 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 621 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS

I hope this isn't a bad drive as well. It sure looks bad, but I am wondering if maybe I have some other type of problem, as I don't get any errors in the console log. Any other places I should check? Is the pool mentioning a specific GUID or an element of the missing pool?
 
Top