Failing drive? Missing Spare?

[Fuganater]

Hey all.

I've gone though all the steps but I am confusing myself. My system has been running great for over a year and now a hikup. I had a drive fail a few weeks ago and my spare took over, great. Now I have a drive giving me read errors. I want to pop out the failed drive, where ever that is, reformat it and test it then try to put it back in. I think it failed due to frequent power outages. I know I know... get a freaking UPS. I will, eventually.

So below is what I have found but I can't put 2 and 2 together to find the failed drive, then find the "trouble" drive. I think da11 is my trouble drive but if that is so, then where is the failed drive?

And then to find the failed drive in my Supermicro Chassis... I think this is the right steps? http://serverfault.com/questions/261779/how-to-determine-which-disk-failed-in-a-freenas-zfs-setup

[root@C3PO] ~# glabel status
Name Status Components
gptid/0189e083-87c3-11e5-9257-0cc47a6bd0ac N/A da0p2
gptid/0450883e-87c3-11e5-9257-0cc47a6bd0ac N/A da1p2
gptid/f59a6e80-87c2-11e5-9257-0cc47a6bd0ac N/A da2p2
gptid/007e4b9a-87c3-11e5-9257-0cc47a6bd0ac N/A da3p2
gptid/ff728308-87c2-11e5-9257-0cc47a6bd0ac N/A da4p2
gptid/f9788d6d-87c2-11e5-9257-0cc47a6bd0ac N/A da5p2
gptid/fd5cb19c-87c2-11e5-9257-0cc47a6bd0ac N/A da6p2
gptid/f6a0ed65-87c2-11e5-9257-0cc47a6bd0ac N/A da7p2
gptid/f7a889f1-87c2-11e5-9257-0cc47a6bd0ac N/A da8p2
gptid/f8ad1426-87c2-11e5-9257-0cc47a6bd0ac N/A da9p2
gptid/0385f501-87c3-11e5-9257-0cc47a6bd0ac N/A da10p2
gptid/051cb5fd-87c3-11e5-9257-0cc47a6bd0ac N/A da11p2
gptid/028825dd-87c3-11e5-9257-0cc47a6bd0ac N/A da12p2
gptid/fa86dfdf-87c2-11e5-9257-0cc47a6bd0ac N/A da13p2
gptid/fb895231-87c2-11e5-9257-0cc47a6bd0ac N/A da14p2
gptid/fc8ac078-87c2-11e5-9257-0cc47a6bd0ac N/A da15p2
gptid/8ea42839-7f56-11e5-80f4-0cc47a6bd0ac N/A da16p1
gptid/8ead9abf-7f56-11e5-80f4-0cc47a6bd0ac N/A da16p2
gptid/8ecec9e7-7f56-11e5-80f4-0cc47a6bd0ac N/A da17p1
gptid/8ed85293-7f56-11e5-80f4-0cc47a6bd0ac N/A da17p2
gptid/8e122e33-eb89-11e5-b59f-0cc47a6bd0ac N/A ada0p2
gptid/8e4e2a75-eb89-11e5-b59f-0cc47a6bd0ac N/A ada1p2

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.39.00.00
Firmware version : 20.00.04.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : 0
Segment : 0
Bus : 3
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
Enclosure # : 2
Slot # : 0
SAS Address : 5003048-0-00b9-89c4
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0ZTXXXU
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Enclosure services device
Enclosure # : 2
Slot # : 0
SAS Address : 5003048-0-00b9-89fd
State : Standby (SBY)
Manufacturer : LSILOGIC
Model Number : SASX36 A.1
Firmware Revision : 7017
Serial No : x3655170
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device

Device is a Hard disk
Enclosure # : 2
Slot # : 1
SAS Address : 5003048-0-00b9-89c5
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A80
Serial No : WDWMC4M0958685
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 2
SAS Address : 5003048-0-00b9-89c6
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A80
Serial No : WDWMC4M1147112
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 3
SAS Address : 5003048-0-00b9-89c7
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0ZTX9D7
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 5
SAS Address : 5003048-0-00b9-89c9
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0NZ3FUC
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 6
SAS Address : 5003048-0-00b9-89ca
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A80
Serial No : WDWMC4M1147413
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 7
SAS Address : 5003048-0-00b9-89cb
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A80
Serial No : WDWMC4M1147045
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 8
SAS Address : 5003048-0-00b9-89cc
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0ZTXP9X
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 9
SAS Address : 5003048-0-00b9-89cd
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M3DHPZDT
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 10
SAS Address : 5003048-0-00b9-89ce
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M3DHPVC4
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 11
SAS Address : 5003048-0-00b9-89cf
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M6AKFX4C
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 12
SAS Address : 5003048-0-00b9-89d0
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A80
Serial No : WDWMC4M0973509
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 13
SAS Address : 5003048-0-00b9-89d1
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0NJK0XH
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 14
SAS Address : 5003048-0-00b9-89d2
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M3DHPY0L
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 15
SAS Address : 5003048-0-00b9-89d3
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0UK4A1K
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD

Device is a Hard disk
Enclosure # : 2
Slot # : 16
SAS Address : 5003048-0-00b9-89d4
State : Ready (RDY)
Size (in MB)/(in sectors) : 1907729/3907029167
Manufacturer : ATA
Model Number : WDC WD20EFRX-68E
Firmware Revision : 0A82
Serial No : WDWCC4M0NJKKTP
GUID : N/A
Protocol : SATA
Drive Type : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500605b0:013ca580
Numslots : 8
StartSlot : 0
Enclosure# : 2
Logical ID : 50030480:00b989ff
Numslots : 34
StartSlot : 0
--------------------------------------------------

[root@C3PO] ~# zpool status Vol1
pool: Vol1
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0 in 12h1m with 0 errors on Fri Mar 10 12:19:12 2017
config:

NAME STATE READ WRITE CKSUM
Vol1 DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/f59a6e80-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/f6a0ed65-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/f7a889f1-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/f8ad1426-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/f9788d6d-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/fa86dfdf-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/fb895231-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/fc8ac078-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gptid/fd5cb19c-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/051cb5fd-87c3-11e5-9257-0cc47a6bd0ac FAULTED 3 0 0 too many errors
gptid/ff728308-87c2-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/007e4b9a-87c3-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/0189e083-87c3-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/028825dd-87c3-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/0385f501-87c3-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
gptid/0450883e-87c3-11e5-9257-0cc47a6bd0ac ONLINE 0 0 0
spares
7999731173232720296 UNAVAIL was /dev/gptid/3bb41c07-ea35-11e5-99cf-0cc47a6bd0ac

 

[Fuganater]

Acutally it could be da6...

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 178 174 021 Pre-fail Always - 4058
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 147
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 068 068 000 Old_age Always - 23376
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 147
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 137
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 9
194 Temperature_Celsius 0x0022 111 106 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 001 000 Old_age Always - 381159
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

 

[Robert Trevellyan]

Looks like FAULTED matches the ID of da11. You'll have to find the UNAVAIL spare by a process of elimination. Match all the good drives with their serial numbers, make a hard-copy list, shut down the server and pull the drives by examining serial numbers.

 

[Fuganater]

Ok so the spare took over and is now gone... How do I find out what drive it replaced and where that drive is so I can pull it?

 

[Robert Trevellyan]

You have to make a list of the serial numbers of all the active drives, then look at each physical drive until you find the one that isn't on the list.

 

[Fuganater]

Oh joy... I wonder if this is any easier in FreeNAS Corral...

 

[darkwarrior]

Oh joy... I wonder if this is any easier in FreeNAS Corral...

Nope. Is strictly the same ;)

 

[Fuganater]

Ok so now and odd issue when trying to find the missing drive. da15 shows up twice and da16 is missing. I have 17 drives in the chassis. Also da4 has an error when running the script.

And looking at the drives in the GUI da4 is missing but da16 shows up. I'm not sure how to intemperate this issue.

 

[Robert Trevellyan]

In this case I would trust the list you see in the GUI.

 

[Fuganater]

So in this case da4 is my bad drive that the spare replaced?

 

[Robert Trevellyan]

Apparently. The point is that the list of serial numbers you're seeing in the GUI is likely the one you need to check against the physical disks.

 

[Mr_N]

This is why a list of all HDD serial numbers and their install locations in your system should be completed as you build your system and it'll save time in the long run especially if you have a significant number of drives :)