Incorrect reporting of Critical Error

Status
Not open for further replies.
Shane Bolton

Shane Bolton

Dabbler
Joined
Jan 16, 2016
Messages
12
I just got this email:

The volume vol01 (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

I checked the volume in about 1 minute from receiving the email. Nothing wrong. Everything was OK.
 
Bidule0hm

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Please post the output of zpool status -v between code tags.
 
Shane Bolton

Shane Bolton

Dabbler
Joined
Jan 16, 2016
Messages
12
zpool status -v is below:




Code:
 
pool: vol01                                                                  
state: ONLINE                                                                 
  scan: resilvered 44K in 0h0m with 0 errors on Sun Feb 14 21:15:49 2016       
config:                                                                        
                                                                               
        NAME                                            STATE     READ WRITE CKS
UM                                                                             
        vol01                                           ONLINE       0     0   
0                                                                             
          raidz1-0                                      ONLINE       0     0   
0                                                                             
            gptid/ce2dad98-d2b4-11e5-8520-001fd0831e05  ONLINE       0     0   
0                                                                             
            gptid/cf5a43f1-d2b4-11e5-8520-001fd0831e05  ONLINE       0     0   
0                                                                             
            gptid/d079415f-d2b4-11e5-8520-001fd0831e05  ONLINE       0     0   
0                                                                             
                                                                               
errors: No known data errors
 
Bidule0hm

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
It has resilvered 44 kB with success so that's why the pool is now healthy again.

Now the question is why it had 44 kB corrupted. Can you post the full list of your hardware and the output of smartctl -a /dev/adaX for each drive, between code tags (4th button from the end in the editor, 3rd item in the list), please.
 
Shane Bolton

Shane Bolton

Dabbler
Joined
Jan 16, 2016
Messages
12
Config:

Intel Core 2 Duo.
8GB RAM
ada0, ada2, ada3 = 2TB Seagate = ZFS1 Volume
ada1 = 160GB (not used or configured.)

Code:
[root@nas02] ~# smartctl -a /dev/ada0
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (AF)
Device Model:     ST2000DL003-9VT166
Serial Number:    5YD1CYB4
LU WWN Device Id: 5 000c50 02f3ff133
Firmware Version: CC32
User Capacity:    2,000,397,852,160 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Feb 14 23:20:33 2016 AEDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  623) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 344) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   099   006    Pre-fail  Always       -       1234336
  3 Spin_Up_Time            0x0003   093   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   092   092   020    Old_age   Always       -       9058
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   030    Pre-fail  Always       -       724855745
  9 Power_On_Hours          0x0032   035   035   000    Old_age   Always       -       57298
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       173
183 Runtime_Bad_Block       0x0032   090   090   000    Old_age   Always       -       10
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   099   099   000    Old_age   Always       -       1
188 Command_Timeout         0x0032   100   097   000    Old_age   Always       -       8590065720
189 High_Fly_Writes         0x003a   070   070   000    Old_age   Always       -       30
190 Airflow_Temperature_Cel 0x0022   064   046   045    Old_age   Always       -       36 (Min/Max 24/39)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       123
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       9059
194 Temperature_Celsius     0x0022   036   054   000    Old_age   Always       -       36 (0 12 0 0 0)
195 Hardware_ECC_Recovered  0x001a   009   003   000    Old_age   Always       -       1234336
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       38875 (253 34 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1584165110
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1239827477

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     52987         -
# 2  Short offline       Completed without error       00%     52925         -
# 3  Short offline       Completed without error       00%     52873         -
# 4  Short offline       Completed without error       00%     52811         -
# 5  Short offline       Completed without error       00%     52755         -
# 6  Short offline       Completed without error       00%     52704         -
# 7  Short offline       Completed without error       00%     52648         -
# 8  Short offline       Completed without error       00%     52596         -
# 9  Short offline       Completed without error       00%     52531         -
#10  Short offline       Completed without error       00%     52473         -
#11  Short offline       Completed without error       00%     52406         -
#12  Short offline       Completed without error       00%     52362         -
#13  Short offline       Completed without error       00%     52303         -
#14  Short offline       Completed without error       00%     52270         -
#15  Short offline       Completed without error       00%     52233         -
#16  Short offline       Completed without error       00%     52193         -
#17  Short offline       Completed without error       00%     52139         -
#18  Short offline       Completed without error       00%     52087         -
#19  Short offline       Completed without error       00%     52063         -
#20  Short offline       Completed without error       00%     52039         -
#21  Short offline       Completed without error       00%     52015         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:
[root@nas02] ~# smartctl -a /dev/ada2
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (AF)
Device Model:     ST2000DL003-9VT166
Serial Number:    5YD1DQLA
LU WWN Device Id: 5 000c50 02f46937f
Firmware Version: CC32
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Feb 14 23:23:55 2016 AEDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  612) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 331) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   102   099   006    Pre-fail  Always       -       1132256
  3 Spin_Up_Time            0x0003   093   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   092   092   020    Old_age   Always       -       9047
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   030    Pre-fail  Always       -       732004161
  9 Power_On_Hours          0x0032   035   035   000    Old_age   Always       -       57454
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       202
183 Runtime_Bad_Block       0x0032   094   094   000    Old_age   Always       -       6
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   096   000    Old_age   Always       -       4295032884
189 High_Fly_Writes         0x003a   001   001   000    Old_age   Always       -       135
190 Airflow_Temperature_Cel 0x0022   063   045   045    Old_age   Always   In_the_past 37 (Min/Max 25/40)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       148
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       9048
194 Temperature_Celsius     0x0022   037   055   000    Old_age   Always       -       37 (0 13 0 0 0)
195 Hardware_ECC_Recovered  0x001a   030   003   000    Old_age   Always       -       1132256
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       38996 (92 68 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2280846010
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3234290346

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     53048         -
# 2  Short offline       Completed without error       00%     52984         -
# 3  Short offline       Completed without error       00%     52934         -
# 4  Short offline       Completed without error       00%     52873         -
# 5  Short offline       Completed without error       00%     52814         -
# 6  Short offline       Completed without error       00%     52764         -
# 7  Short offline       Completed without error       00%     52710         -
# 8  Short offline       Completed without error       00%     52654         -
# 9  Short offline       Completed without error       00%     52589         -
#10  Short offline       Completed without error       00%     52534         -
#11  Short offline       Completed without error       00%     52470         -
#12  Short offline       Completed without error       00%     52428         -
#13  Short offline       Completed without error       00%     52370         -
#14  Short offline       Completed without error       00%     52337         -
#15  Short offline       Completed without error       00%     52300         -
#16  Short offline       Completed without error       00%     52258         -
#17  Short offline       Completed without error       00%     52205         -
#18  Short offline       Completed without error       00%     52152         -
#19  Short offline       Completed without error       00%     52128         -
#20  Short offline       Completed without error       00%     52104         -
#21  Short offline       Completed without error       00%     52080         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@nas02] ~# smartctl -a /dev/ada3
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Green (AF)
Device Model:     ST2000DL003-9VT166
Serial Number:    5YD16KY3
LU WWN Device Id: 5 000c50 02f24f2af
Firmware Version: CC32
User Capacity:    2,000,397,852,160 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Feb 14 23:26:52 2016 AEDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  623) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 345) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   099   006    Pre-fail  Always       -       1039168
  3 Spin_Up_Time            0x0003   093   089   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   092   092   020    Old_age   Always       -       8414
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   030    Pre-fail  Always       -       701996343
  9 Power_On_Hours          0x0032   043   043   000    Old_age   Always       -       50388
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       208
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   098   000    Old_age   Always       -       4295032861
189 High_Fly_Writes         0x003a   001   001   000    Old_age   Always       -       1384
190 Airflow_Temperature_Cel 0x0022   066   046   045    Old_age   Always       -       34 (Min/Max 25/36)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       166
193 Load_Cycle_Count        0x0032   096   096   000    Old_age   Always       -       8419
194 Temperature_Celsius     0x0022   034   054   000    Old_age   Always       -       34 (0 14 0 0 0)
195 Hardware_ECC_Recovered  0x001a   009   003   000    Old_age   Always       -       1039168
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       35189 (212 188 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2033070110
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1158915606

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     49838         -
# 2  Short offline       Completed without error       00%     49814         -
# 3  Short offline       Completed without error       00%     49790         -
# 4  Short offline       Completed without error       00%     49769         -
# 5  Short offline       Completed without error       00%     49738         -
# 6  Short offline       Completed without error       00%     49705         -
# 7  Short offline       Completed without error       00%     49685         -
# 8  Short offline       Completed without error       00%     49662         -
# 9  Short offline       Completed without error       00%     49623         -
#10  Short offline       Completed without error       00%     49587         -
#11  Short offline       Completed without error       00%     49562         -
#12  Short offline       Completed without error       00%     49538         -
#13  Short offline       Completed without error       00%     49514         -
#14  Short offline       Completed without error       00%     49490         -
#15  Short offline       Completed without error       00%     49435         -
#16  Short offline       Completed without error       00%     49319         -
#17  Short offline       Completed without error       00%     49235         -
#18  Short offline       Completed without error       00%     49151         -
#19  Short offline       Completed without error       00%     49074         -
#20  Short offline       Completed without error       00%     48999         -
#21  Short offline       Completed without error       00%     48937         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@nas02] ~#
 
Bidule0hm

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
By full hardware list I wanted to say the full list, not just the CPU and RAM. What's the MB, PSU, ...? is there an HBA?, etc...

I don't see anything wrong with those drives but you didn't setup long SMART tests so I can't be sure. I recommend to launch a long test manually on each drive, wait for the end, and then post the output of smartctl again.

NB: 57 kh is more than 6.5 years. It's remarkable because the drives don't last more than 4-5 years usually. It also means the drives are probably close to the end of their life.
 
Status
Not open for further replies.

Similar threads

bernieke
Issues with 12.0-U1 resulting in kernel panic boot loop
Replies
4
Views
2K
bernieke
bernieke
J
  • Locked
Critical error! What to do next?
2 3
Replies
45
Views
6K
JFD
J
D
  • Locked
ZFS is in a degraded state
Replies
15
Views
3K
danb35
danb35
J
  • Locked
Hulp gevraagd: critical error
Replies
3
Views
2K
JFD
J
B
'Disk removed by administrator' 2 minutes later alert is cleared
Replies
9
Views
1K
ChrisRJ
ChrisRJ
Top