Possible Double faulted RaidZ1 / Data Loss

[DarthZenophon]

I've been searching the forums and have not found anything that seems to be the problem I'm experiencing, but I will try to provide what have been the most commonly requested pieces of information. I discovered this problem when I heard a loud noise coming from my NAS box - clearly I have a failed hard drive. I assumed that with my 4+1 RaidZ1 setup I should be able to replace the drive and rebuild my pool. However, when I logged into the FreeNAS GUI, I found the following alert:

"CRITICAL: The volume pool_1 (ZFS) state is unknown"

In addition, rather than a failed drive with a "REPLACE" button, one of my drives was missing from the drive list entirely. I performed a clean shutdown and powered the system back up, and the error remained. The zpool import command seems to indicate a double fault, though I'm hoping one of the drives may be somehow recoverable long enough for me to get my data. I have purchased two new 3TB WD Red drives to use as replacements. Please help!

Version: FreeNAS-9.3-STABLE-201504100216
CPU: Intel Pentium(R) CPU G630 @ 2.70 GHz
Memory: 16059 MB


ZPOOL IMPORT OUTPUT

[root@the-infosphere] ~# zpool import
pool: pool_1
id: 14787558555151179531
state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-3C
config:

pool_1 FAULTED corrupted data
raidz1-0 FAULTED corrupted data
gptid/68c40808-d5e3-11e1-bbdd-bc5ff436d656 ONLINE
12310388013365230236 UNAVAIL cannot open
gptid/69f71c38-d5e3-11e1-bbdd-bc5ff436d656 ONLINE
233958574599389935 FAULTED corrupted data
gptid/6b2ad21f-d5e3-11e1-bbdd-bc5ff436d656 ONLINE


ZPOOL STATUS OUTPUT

[root@the-infosphere] ~# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Thu Jun 18 03:45:23 2015
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/6ba7a970-f9b5-11e4-9e90-bc5ff436d656 ONLINE 0 0 0

errors: No known data errors


SMARTCTL OUTPUT

[root@the-infosphere] ~# smartctl -a /dev/ada0
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RTF2
LU WWN Device Id: 5 000c50 052b002ec
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 22 20:25:36 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

SMART Disabled. Use option -s with argument 'on' to enable it.
(override with '-T permissive' option)
[root@the-infosphere] ~# smartctl -a /dev/ada1
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RRNS
LU WWN Device Id: 5 000c50 052b0793d
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 22 20:25:42 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

SMART Disabled. Use option -s with argument 'on' to enable it.
(override with '-T permissive' option)
[root@the-infosphere] ~# smartctl -a /dev/ada2
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RSWQ
LU WWN Device Id: 5 000c50 052af4fc0
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Jul 22 20:25:54 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

SMART Disabled. Use option -s with argument 'on' to enable it.
(override with '-T permissive' option)
[root@the-infosphere] ~# smartctl -a /dev/ada3
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0T7JT
LU WWN Device Id: 5 000c50 05297bd14
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Jul 22 20:26:08 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

SMART Disabled. Use option -s with argument 'on' to enable it.
(override with '-T permissive' option)

 

[cyberjock]

Well, you are definitely in a bad way.

SMART is disabled on all of the disks. So none of that output tells us the health of your disks. You'll need to enable SMART support and then run those commands again. I think the argument is -s on to turn it on.

But you seem to have 4 disks "available", ada0 through ada3. So you need to find the disk that isn't listed there and make it work. If you can't then there is no hope for your zpool aside from professional data recovery (Got $20k+?).

 

[Robert Trevellyan]

I have purchased two new 3TB WD Red drives to use as replacements. Please help!

There's a chance of at least partial recovery (to copy important data elsewhere) if you can clone one or both of the bad drives to one of the new ones using ddrescue.

Just as a sanity check, are all the drives connected to the same controller?

It would be best if you provided full details of your setup as described in the forum rules.

 

[cyberjock]

There's a chance of at least partial recovery (to copy important data elsewhere) if you can clone one or both of the bad drives to one of the new ones using ddrescue.

Just as a sanity check, are all the drives connected to the same controller?

It would be best if you provided full details of your setup as described in the forum rules.

Cloning the disk labeled as faulted is a lost cause. ZFS has already determined the data on that disk is bad and cannot be used. He *must* get the disk that isn't being detected to work.

Or, you know, fix the corruption on the faulted disk. But let's be serious.. is that particularly likely? I'm gonna go with "no" on that one. ;)

 

[Robert Trevellyan]

ZFS has already determined the data on that disk is bad and cannot be used.

Is there any way to force ZFS to try again (after cloning)?

 

[Ericloewe]

Is there any way to force ZFS to try again (after cloning)?

If the disk is faulted, cloning it accomplishes nothing. At best, you can do something like move the platters to a new drive or something of the sort (depending on the failure). But a one-to-one copy of the faulted drive won't help.

 

[DarthZenophon]

Thanks for everyone's responses, I really appreciate your willingness to help me out!

Here is the output with SMART enabled. I have all of the drives hooked up directly to the motherboard, so they should be on the same controller. The board is an ASRock H77 Pro4-M. Can anyone shed any light on why the fifth drive would simply be missing? As it was part of my pool, I would have thought it would still be listed, but flagged as Removed or Faulted or Missing. I have a feeling that fifth drive is the one I heard the bad noises from, so I'm not overly optimistic about getting it connected again. There's nothing totally critical on this NAS that I'll be heartbroken about if I can't recover it, but there is a lot of media that would take a while to replace. If it isn't recoverable, I mainly just want to understand what went wrong so that I can do things correctly when I set it back up (aside from obviously setting up better email notifications, which probably could have saved me when the first drive went down).

[root@the-infosphere] ~# smartctl -a /dev/ada0 -s on
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RTF2
LU WWN Device Id: 5 000c50 052b002ec
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 24 12:41:30 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 584) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 331) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 169395632
3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1328
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 272
7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 10432108
9 Power_On_Hours 0x0032 079 079 000 Old_age Always - 19144
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 41
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 037 037 000 Old_age Always - 63
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 098 098 000 Old_age Always - 2
190 Airflow_Temperature_Cel 0x0022 070 043 045 Old_age Always In_the_past 30 (0 5 30 29 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 32
193 Load_Cycle_Count 0x0032 067 067 000 Old_age Always - 66298
194 Temperature_Celsius 0x0022 030 057 000 Old_age Always - 30 (0 21 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 56
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 56
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5036h+59m+16.867s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 12669231730603
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 8936763722311

SMART Error Log Version: 1
ATA Error Count: 50 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 50 occurred at disk power-on lifetime: 19062 hours (794 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 40 ff ff ff 4f 00 46d+21:45:43.377 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:43.377 READ FPDMA QUEUED
ef 02 00 00 00 00 40 00 46d+21:45:43.377 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 00 46d+21:45:43.377 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 00 46d+21:45:43.376 SET MULTIPLE MODE

Error 49 occurred at disk power-on lifetime: 19062 hours (794 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 40 ff ff ff 4f 00 46d+21:45:39.488 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:39.488 READ FPDMA QUEUED
ef 02 00 00 00 00 40 00 46d+21:45:39.488 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 00 46d+21:45:39.488 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 00 46d+21:45:39.488 SET MULTIPLE MODE

Error 48 occurred at disk power-on lifetime: 19062 hours (794 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 40 ff ff ff 4f 00 46d+21:45:35.616 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:35.616 READ FPDMA QUEUED
ef 02 00 00 00 00 40 00 46d+21:45:35.615 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 00 46d+21:45:35.615 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 00 46d+21:45:35.615 SET MULTIPLE MODE

Error 47 occurred at disk power-on lifetime: 19062 hours (794 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 40 ff ff ff 4f 00 46d+21:45:31.652 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:31.652 READ FPDMA QUEUED
ef 02 00 00 00 00 40 00 46d+21:45:31.652 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 00 46d+21:45:31.652 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 00 46d+21:45:31.651 SET MULTIPLE MODE

Error 46 occurred at disk power-on lifetime: 19062 hours (794 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 40 ff ff ff 4f 00 46d+21:45:27.507 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:27.483 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:27.483 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:27.482 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 46d+21:45:27.476 READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@the-infosphere] ~# smartctl -a /dev/ada1 -s on
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RRNS
LU WWN Device Id: 5 000c50 052b0793d
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 24 12:41:47 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 331) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 118 100 006 Pre-fail Always - 193761440
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1441
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 070 060 030 Pre-fail Always - 10938364
9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 13907
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 42
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1
190 Airflow_Temperature_Cel 0x0022 071 048 045 Old_age Always - 29 (Min/Max 29/29)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 33
193 Load_Cycle_Count 0x0032 067 067 000 Old_age Always - 67649
194 Temperature_Celsius 0x0022 029 052 000 Old_age Always - 29 (0 14 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 2438h+17m+47.093s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 184071241165096
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 27971326376233

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@the-infosphere] ~# smartctl -a /dev/ada2 -s on
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0RSWQ
LU WWN Device Id: 5 000c50 052af4fc0
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Jul 24 12:42:50 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 336) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 097 097 006 Pre-fail Always - 113729968
3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1443
5 Reallocated_Sector_Ct 0x0033 099 099 036 Pre-fail Always - 2160
7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 10151603
9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 13963
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 42
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 2922
188 Command_Timeout 0x0032 099 099 000 Old_age Always - 3 3 3
189 High_Fly_Writes 0x003a 098 098 000 Old_age Always - 2
190 Airflow_Temperature_Cel 0x0022 071 043 045 Old_age Always In_the_past 29 (0 1 29 29 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 33
193 Load_Cycle_Count 0x0032 086 086 000 Old_age Always - 28424
194 Temperature_Celsius 0x0022 029 057 000 Old_age Always - 29 (0 13 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4345h+50m+06.385s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 194556539965720
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 114417384469785

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[DarthZenophon]

Continued...


[root@the-infosphere] ~# smartctl -a /dev/ada3 -s on
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0T7JT
LU WWN Device Id: 5 000c50 05297bd14
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Jul 24 12:43:55 2015 EDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 329) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 100 006 Pre-fail Always - 142616120
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1321
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 12650999
9 Power_On_Hours 0x0032 080 080 000 Old_age Always - 18229
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 42
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 050 050 000 Old_age Always - 50
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 073 052 045 Old_age Always - 27 (Min/Max 27/27)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 33
193 Load_Cycle_Count 0x0032 066 066 000 Old_age Always - 69163
194 Temperature_Celsius 0x0022 027 048 000 Old_age Always - 27 (0 13 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5001h+52m+55.314s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 250928146667301
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 99549394681141

SMART Error Log Version: 1
ATA Error Count: 50 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 50 occurred at disk power-on lifetime: 13354 hours (556 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 4d+00:18:19.442 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:19.441 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:19.440 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:19.433 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:19.403 READ FPDMA QUEUED

Error 49 occurred at disk power-on lifetime: 13354 hours (556 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 4d+00:18:16.491 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:16.460 READ FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 4d+00:18:16.460 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:16.460 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:16.460 READ FPDMA QUEUED

Error 48 occurred at disk power-on lifetime: 13354 hours (556 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 4d+00:18:13.547 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:13.521 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:13.520 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:13.519 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:13.518 READ FPDMA QUEUED

Error 47 occurred at disk power-on lifetime: 13354 hours (556 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 4d+00:18:10.598 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:10.598 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:10.597 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:10.591 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:10.559 READ FPDMA QUEUED

Error 46 occurred at disk power-on lifetime: 13354 hours (556 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 c0 ff ff ff 4f 00 4d+00:18:07.913 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:07.913 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:07.913 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:07.913 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 4d+00:18:07.913 READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@the-infosphere] ~#

 

[Robert Trevellyan]

ada0 looks like it's failing
ada1 looks OK
ada2 looks worse than ada1
ada3 may be showing early signs of failure

I mainly just want to understand what went wrong so that I can do things correctly when I set it back up (aside from obviously setting up better email notifications, which probably could have saved me when the first drive went down).

This, plus suitable SMART testing and pool scrubbing schedules, plus RAIDZ2 instead of RAIDZ1, and you would probably be in good shape right now.

 

[Robert Trevellyan]

You also made an unfortunate choice of hard drive model, namely a specific 3TB Seagate model that turns out to be a lemon. No way you could have known that at the time...

 

[cyberjock]

ada0, ada2, and ada3 are all failing. So the chances of this "working out" are very nearly zero. :/

The fact you had no SMART functioning was a real killer for you as you would likely never know anything was going wrong until it was too late.

At this point you are looking at restoring from backup, spending serious money (think along the lines of $20k+), or simply kissing the data goodbye.

If the disk that is being undetected were detected, there's a slim chance you might be able to get some of your data back. But without it, your chances are zero.

 

[Ericloewe]

By the way, for future reference, please use CODE tags for CLI output. Fixed-width fonts and correct whitespace make that stuff much easier to read. Plus, XenForo defaults to a compact view that allows for less scrolling movement when using QUOTE, CODE or similar.