Dead drive/power loss related problem?

[HelloFreeNAS]

Just under a week ago I was trying to access my FreeNAS web gui but was receiving some sort of nginx error. I hadn't set up SSH and couldn't figure out a way to safely shut down my server. I tried orderly shutdown in asrock IPMI and pressing the physical power button on the case (which usually initiates a safe shutdown), neither of which worked.
I ended up having to do a force shutdown through IPMI.

When I turned the server back on I could access the web GUI just fine but I had a couple alarming alerts:
"CRITICAL: 4408 unreadable sectors", "CRITICAL: one or more devices has experienced an unrecoverable error"

Throughout the past 5ish days after turning the server back on in the morning these alerts have showed to up to 6056 unreadable sectors and sometimes "CRITICAL: one or more devices has experienced an unrecoverable error" is there and sometimes it isn't.
The mirror had also been changed to a degraded state at one point.
I just updated to 9.10 stable and it is currently back to only 4408 unreadable sectors and "CRITICAL: one or more devices has experienced an unrecoverable error" has disappeared again and the mirror is in an online state.

What I'm unsure about is whether these notifications are related to some sort of drive errors which resulted in the nginx errors or if the nginx error was just a bug and cutting the power started all this.

My drives are a WD Barracuda 3TB (the one with errors) and WD Red 3TB in a mirror
2x SanDisk 8GB Cruzer Fit for my boot sticks, mirrored
I also have 8GB of ECC RAM.

1. What can I do to test the drive and see whether it's a physical issue and the drive needs replacing?

2. If it is just related to the unsafe shutdown, what can I do to fix these errors?

Thanks

edit: gonna have a look at this:
https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/
ill update if I figure anything out

 

[HelloFreeNAS]

There is currently no checksum, read or write errors on the volume although there definitely was previously.

currently running a long SMART test and will report back the results. This will only take 6 hours but I'm in Australia and I think I'll be asleep when it finishes.

 

[danb35]

After the long SMART test finishes (on both disks), post the output of smartctl -a /dev/whatever for both disks (in code tags) and run a scrub of your pool. The pool errors are cleared when you reboot, but the scrub will find (and fix, if possible) them.

 

[HelloFreeNAS]

Very sorry for the late reply.

I have a new alert on freenas - CRITICAL: Device: /dev/ada0, Self-Test Log error count increased from 0 to 1

These are the results:

Seagate Barracuda (ada0, the suspect)-

Code:

smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.3-RELEASE amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1CH166
Serial Number:    W1F3E19T
LU WWN Device Id: 5 000c50 06a2e6125
Firmware Version: CC47
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Apr  5 22:02:04 2016 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (  609) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 347) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       198907800
  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   099   099   020    Old_age   Always       -       1586
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   063   057   030    Pre-fail  Always       -       300782496702
  9 Power_On_Hours          0x0032   079   079   000    Old_age   Always       -       18840
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       71
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   068   068   000    Old_age   Always       -       32
188 Command_Timeout         0x0032   100   095   000    Old_age   Always       -       165 165 178
189 High_Fly_Writes         0x003a   099   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0x0022   066   040   045    Old_age   Always   In_the_past 34 (22 196 38 24 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1634
194 Temperature_Celsius     0x0022   034   060   000    Old_age   Always       -       34 (0 15 0 0 0)
197 Current_Pending_Sector  0x0012   061   061   000    Old_age   Always       -       6544
198 Offline_Uncorrectable   0x0010   061   061   000    Old_age   Offline      -       6544
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10976h+19m+40.579s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       56667469703
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       48072535345

SMART Error Log Version: 1
ATA Error Count: 33 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 33 occurred at disk power-on lifetime: 18600 hours (775 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00      08:06:07.370  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      08:06:07.370  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      08:06:07.370  WRITE FPDMA QUEUED
  60 00 10 ff ff ff 4f 00      08:06:07.370  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:06:07.369  READ FPDMA QUEUED

Error 32 occurred at disk power-on lifetime: 18600 hours (775 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00      08:06:04.837  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      08:06:04.837  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00      08:06:03.798  WRITE FPDMA QUEUED
  60 00 10 ff ff ff 4f 00      08:06:03.798  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:06:03.798  READ FPDMA QUEUED

Error 31 occurred at disk power-on lifetime: 18600 hours (775 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00      08:06:00.193  WRITE FPDMA QUEUED
  60 00 10 ff ff ff 4f 00      08:06:00.193  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:06:00.193  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:06:00.192  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      08:06:00.152  READ LOG EXT

Error 30 occurred at disk power-on lifetime: 18600 hours (775 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00      08:05:56.589  WRITE FPDMA QUEUED
  60 00 10 ff ff ff 4f 00      08:05:56.588  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:05:56.588  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:05:56.588  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      08:05:56.495  READ LOG EXT

Error 29 occurred at disk power-on lifetime: 18600 hours (775 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00      08:05:52.778  WRITE FPDMA QUEUED
  60 00 10 ff ff ff 4f 00      08:05:52.770  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:05:52.744  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:05:52.743  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      08:05:52.743  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     18643         775977928
# 2  Short offline       Completed without error       00%     18643         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

WD Red (ada1)-

Code:

smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.3-RELEASE amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0F7R8M0
LU WWN Device Id: 5 0014ee 60550c6aa
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Apr  5 21:59:14 2016 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (39840) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 399) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   155   154   021    Pre-fail  Always       -       7250
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       77
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7657
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       75
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       27
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       157
194 Temperature_Celsius     0x0022   118   111   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      7495         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[danb35]

ada0 is definitely bad. ada1 looks fine. Have you run a scrub since the SMART tests completed? If so, what's the output of zpool status now?

You also need to schedule regular SMART tests--neither of those disks has seen one until just now. I run short daily and long weekly; you could go as long as a short weekly and long monthly.

 

[HelloFreeNAS]

Hmm.. no scrub run since Feb 28th. I'll run one now and update when it's done.
I'll also set up SMART and scrub scheduling. I should've read into that more.

Thanks for the help

 

[danb35]

Scrubs run every six weeks by default, so nothing since 28 Feb would be expected--but more frequently than that would be better.