SOLVED SMART error (FailedOpenDevice) detected on host: FreeNAS????

[tmacka88]

Hi,

I just received a email notification:

Subject

SMART error (FailedOpenDevice) detected on host: FreeNAS

Body

This message was generated by the smartd daemon running on:

host name: FreeNAS
DNS domain: local

The following warning/error was logged by the smartd daemon:

Device: /dev/da3 [SAT], unable to open device

Device info:
WDC WD30EZRX-00DC0B0, S/N:WD-WCC1T1416374, WWN:5-0014ee-25e0bbcc2, FW:80.00A80, 3.00 TB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
No additional messages about this problem will be sent.

Just wondering what the best course of action is from here. How do I confirm that the disk has failed?

I am running 4 drives on my M1015 controller (that has been reflashed) and 2 drives on my MB. Setup includes 6 x 3TB RAIDz2.

$ zpool status

Code:

pool: Volume4
state: DEGRADED
status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
  scan: resilvered 4.36G in 0h2m with 0 errors on Sat Nov 16 16:37:37 2013
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    Volume4                                        DEGRADED    0    0    0
      raidz2-0                                      DEGRADED    0    0    0
        gptid/916ff50d-3151-11e3-92fc-f46d04de02ec  ONLINE      0    0    0
        gptid/92784bd8-3151-11e3-92fc-f46d04de02ec  ONLINE      0    0    0
        gptid/9372b04c-3151-11e3-92fc-f46d04de02ec  ONLINE      0    0    0
        gptid/94811c78-3151-11e3-92fc-f46d04de02ec  ONLINE      0    0    0
        gptid/958895ad-3151-11e3-92fc-f46d04de02ec  ONLINE      0    0    0
        8659309636289833160                        REMOVED      0    0    0  was /dev/gptid/96a49781-3151-11e3-92fc-f46d04de02ec
 
errors: No known data errors

Doing some searching on the forum. I have tried running some smart test again from shell.

$ smartctl -q noserial -a /dev/da3

Code:

smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
/dev/xpt0 control device couldn't opened: Permission denied
Unable to get CAM device list
/dev/da3: Unable to detect device type
Please specify device type with the -d option.
 
Use smartctl -h to get a usage summary

$ cat /usr/local/etc/smartd.conf

Code:

################################################
# smartd.conf generated by /etc/rc.d/ix-smartd
################################################
/dev/da0 -n never -W 0,55,60 -m email@gmail.com -m root
/dev/ada1 -n never -W 0,55,60 -m email@gmail.com -m root
/dev/da1 -n never -W 0,55,60 -m email@gmail.com -m root
/dev/ada0 -n never -W 0,55,60 -m email@gmail.com -m root
/dev/da2 -n never -W 0,55,60 -m email@gmail.com 
/dev/da3 -n never -W 0,55,60 -m email@gmail.com 

Any help would be great. Thanks

P.S please indicate the shel commands to run as I am not familiar with running these tests. Doesnt look like the ones I ran above have been working.

 

[cyberjock]

It appears that da3 has disconnected from the system. More than likely the disk just failed outright. The fact that its "REMOVED" tends to support that.

So you need to figure out which disk that is and pull it from the system and replace it with a new disk. That is, unless you were playing inside the server and accidentally bumped the cabling or something. You might be able to do a cold boot of the system and be able to query the smart data for da3, if it works long enough to let you.

 

[tmacka88]

Thanks for your quick reply.

Hmm. I did have the cover open playing inside for a bit, but didnt think I had touch any sata cables. Also the FreeNAS was on and working fine whilst I received the notification.

I will try rebooting it now. By querying da3 do you mean

Code:

smartctl -q noserial -a /dev/da3

When I run this command for all of my disk ada0, ada1, da0, da1, da2, da3 I always get this

Code:

smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
/dev/xpt0 control device couldn't opened: Permission denied
Unable to get CAM device list
/dev/ada0: Unable to detect device type
Please specify device type with the -d option.
 
Use smartctl -h to get a usage summary

So for some reason it's not giving me anything. Any ideas?

 

[tmacka88]

ran anothe zpool status after reboot

Code:

 pool: Volume4
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 4.36G in 0h2m with 0 errors on Sat Nov 16 16:37:37 2013
config:
 
NAME                                            STATE     READ WRITE CKSUM
Volume4                                         DEGRADED     0     0     0
 raidz2-0                                      DEGRADED     0     0     0
   gptid/916ff50d-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/92784bd8-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/9372b04c-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/94811c78-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/958895ad-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   8659309636289833160                         UNAVAIL      0     0     0  was /dev/gptid/96a49781-3151-11e3-92fc-f46d04de02ec

Also still not allowing me to run the SMART util and above.

 

[tmacka88]

I am now in the process of replacing the failed drive and am not sure how to proceed.

As per wiki http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive_or_SSD Step 1 I have navigated to Storage → Volumes → View Volumes → Volume Status and locate the failed device. When I select the disk however, I have only got 1 button "Replace". I am missing the "Offline" button.

I ran zpool status as below and the failed disk is unavailable and not offline.

Code:

pool: Volume4
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 4.36G in 0h2m with 0 errors on Sat Nov 16 16:37:37 2013
config:
 
NAME                                            STATE     READ WRITE CKSUM
Volume4                                         DEGRADED     0     0     0
 raidz2-0                                      DEGRADED     0     0     0
   gptid/916ff50d-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/92784bd8-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/9372b04c-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/94811c78-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   gptid/958895ad-3151-11e3-92fc-f46d04de02ec  ONLINE       0     0     0
   8659309636289833160                         UNAVAIL      0     0     0  was /dev/gptid/96a49781-3151-11e3-92fc-f46d04de02ec
 
errors: No known data errors

How do I proceed from here. Does this mean that maybe the sata cable has come loose or something (in the process of checking now) or can I change this to offline some how?

Thanks

 

[cyberjock]

The disk is already offline. You just have to click the "replace" button and continue on as the manual dictates. I just fixed it for future manuals, but the current manual won't reflect this change.

 

[tmacka88]

Thanks mate,

Replaced the drive fine. Appreciate the help.

 

[robdeep]

Did you ever figure out why the SMART commands wont return any info though? I'm having the same problem. I also have a M1015 cross flashed controller.

 

[tmacka88]

Did you ever figure out why the SMART commands wont return any info though? I'm having the same problem. I also have a M1015 cross flashed controller.

hey, sorry for the late reply.

No id didn't. Im sure if I spent more time on it I would but dont have time too at the moment.