Unable to attach to new device due to status 0x6

[fungus1487]

I am running an older version of FreeNAS (shame on me) I will be posting with regards to migrating to the latest stable elsewhere. However I currently am receiving these in the log:

Code:

Sep  6 22:13:13 r2d2 kernel: (ada3:ahcich3:0:0:0): lost device
Sep  6 22:13:16 r2d2 root: ZFS: vdev I/O failure, zpool=live2 path=/dev/gptid/6e63434f-b1fc-11e3-abf4-f46d049411dd offset=270336 size=8192 error=6
Sep  6 22:13:16 r2d2 root: ZFS: vdev I/O failure, zpool=live2 path=/dev/gptid/6e63434f-b1fc-11e3-abf4-f46d049411dd offset=1998250844160 size=8192 error=6
Sep  6 22:13:16 r2d2 root: ZFS: vdev I/O failure, zpool=live2 path=/dev/gptid/6e63434f-b1fc-11e3-abf4-f46d049411dd offset=1998251106304 size=8192 error=6
Sep  6 22:13:31 r2d2 kernel: cam_periph_alloc: attempt to re-allocate valid device ada3 rejected
Sep  6 22:13:31 r2d2 kernel: adaasync: Unable to attach to new device due to status 0x6

As well as a warning:

Code:

WARNING: The volume live2 (ZFS) status is UNKNOWN: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

I'm pretty damn rusty with ZFS and FreeNAS since I configured this back in 2013(ish).

FreeNAS version is FreeNAS-8.2.0-RC1-x64 (r11876). Any help with this initial issue would be brilliant. I was considering the above zpool clear then scrubbing the volume.

 

[toadman]

Yea, looks like you lost a drive, at least temporarily. What does "zpool status" give you? And what was the configuration on that vdev and the pool? RAIDZ, RAIDZ2, mirror, etc? And can you confirm you can or cannot reach ada3? If yes, what is the smart status?

 

[fungus1487]

Thanks for the reply. Here is the zpool status, it is configured in RAIDZ1:

Code:

  pool: live2
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
scrub: scrub in progress for 0h29m, 51.92% done, 0h26m to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        live2                                           ONLINE       0     0     0
          raidz1                                        ONLINE       0     0     0
            gptid/6b896f3d-b1fc-11e3-abf4-f46d049411dd  ONLINE       0     0     0
            gptid/6c215d33-b1fc-11e3-abf4-f46d049411dd  ONLINE       0     0     0
            gptid/6e63434f-b1fc-11e3-abf4-f46d049411dd  ONLINE      33 1.64K     0
            gptid/6ef55a64-b1fc-11e3-abf4-f46d049411dd  ONLINE       0     0     0
            gptid/6f82dd85-b1fc-11e3-abf4-f46d049411dd  ONLINE       0     0     0

errors: No known data errors

Weirdly I get status ONLINE for that disk but it looks like it isn't reachable. See image:

 

[toadman]

You can try a smartctl -a /dev/ada0 to see if it's reachable. If not, then I would offline it (if you don't have another disk available in the system and need to pull it and put in a new one) or replace it (if you already have a drive in the system). The freenas docs have the procedure spelled out.

You'll be in a precarious position during the resilver obviously as you've got a RAIDZ.

 

[fungus1487]

You can try a "smartctl -a /dev/ada0" to see if it's reachable. If not, then I would offline it (if you don't have another disk available in the system and need to pull it and put in a new one) or replace it (if you already have a drive in the system). The freenas docs have the procedure spelled out.

You'll be in a precarious position during the resilver obviously as you've got a RAIDZ.

I guess you meant "smartctl -a /dev/ada3" ?

 

[toadman]

oops! Yes, ada3. A bad case of cut and paste. :)

 

[fungus1487]

Also from this output I guess that the device is reachable?

Code:

smartctl -a /dev/ada3
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZAD113719
LU WWN Device Id: 5 0014ee 2067800a6
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Sep  6 23:03:24 2016 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (38760) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       50
  3 Spin_Up_Time            0x0027   220   159   021    Pre-fail  Always       -       3966
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       580
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       3462
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       573
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       43
193 Load_Cycle_Count        0x0032   189   189   000    Old_age   Always       -       35367
194 Temperature_Celsius     0x0022   090   076   000    Old_age   Always       -       60
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[toadman]

The good news is the drive is reachable. The bad news is it looks like no self-tests have been run? (or at least logged.) I would make sure you have self test tasks setup.

Re: ada3, there are a couple options I suppose. You can do the zpool clear and then a scrub. If you find any errors you gotta replace the drive obviously. You can also run a self-test at this point. smartctl -t short /dev/ada3

From the zfs stand point the drive obviously has errors. So I suspect you'll end up having to replace it.

 

[fungus1487]

Oops, took apart the NAS (this is post house move) and made sure all the sata connections were tight.

Suffice to say upon restarting the disk was found. Currently scrubbing with no issues.

 

[Bidule0hm]

60 °C?!

You have a serious cooling problem, solve it before all your drives dies.

They should never be allowed to go over 40 °C.

 

[fungus1487]

I will check when back but I am pretty sure it reports the wrong value I remember having a separate script to find this previously as the SATA controller card I am using doesn't play nice with reporting some values.

 

[fungus1487]

As suspected the actual temperature is roughly ~48. A little high but I can cope with it. I remember looking into this before and depending on who you talk to you will find different correlations for disk failure vs temperature. Suffice to say, current risk is acceptable. Thanks guys.