SOLVED 1st warning ever from Freenas

[travalon]

Here is the 1st warning I've ever gotten from Freenas.

'OK: The volume media (ZFS) status is HEALTHY
WARNING: The volume Music (ZFS) status is UNKNOWN:
One or more devices could not be opened.
Sufficient replicas exist for the pool to continue functioning in a degraded state.
Attach the missing device and online it using 'zpool online'.'


I ran: zpool status -v and got ( I screwed up the copy and paste of this one)

Code:

config:                                                                         
                                                                                
        NAME                                            STATE     READ WRITE CKS
UM                                                                              
        media                                           ONLINE       0     0    
 0                                                                              
          mirror-0                                      ONLINE       0     0    
 0                                                                              
            gptid/d5f8be87-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0    
 0                                                                              
            gptid/d668abe7-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0    
 0                                                                              
                                                                                
errors: No known data errors                                                    
[root@TravNAS ~]#              

I then scrubbed all my volumes and the yellow light in the web GUI is still staring at me.
I tried the "online it using 'zpool online'." but didn't get it right. I was just brought to the man page.

Code:

[Travis@TravNAS] ~> zpool Music online
unrecognized command 'Music'
usage: zpool command args ...
where 'command' is one of the following:

        create [-fn] [-o property=value] ...
            [-O file-system-property=value] ...
            [-m mountpoint] [-R root] <pool> <vdev> ...
        destroy [-f] <pool>

        add [-fn] <pool> <vdev> ...
        remove <pool> <device> ...

        labelclear [-f] <vdev>

        list [-H] [-o property[,...]] [-T d|u] [pool] ... [interval [count]]
        iostat [-v] [-T d|u] [pool] ... [interval [count]]
        status [-vx] [-T d|u] [pool] ... [interval [count]]

        online [-e] <pool> <device> ...
        offline [-t] <pool> <device> ...
        clear [-nF] <pool> [device]

        attach [-f] <pool> <device> <new-device>
        detach <pool> <device>
        replace [-f] <pool> <device> [new-device]
        split [-n] [-R altroot] [-o mntopts]
            [-o property=value] <pool> <newpool> [<device> ...]

        scrub [-s] <pool> ...

        import [-d dir] [-D]
        import [-d dir | -c cachefile] [-F [-n]] <pool | id>
        import [-o mntopts] [-o property=value] ...
            [-d dir | -c cachefile] [-D] [-f] [-m] [-N] [-R root] [-F [-n]] -a
        import [-o mntopts] [-o property=value] ...
            [-d dir | -c cachefile] [-D] [-f] [-m] [-N] [-R root] [-F [-n]]
            <pool | id> [newpool]
        export [-f] <pool> ...
        upgrade [-v]
        upgrade [-V version] <-a | pool ...>
        reguid <pool>

        history [-il] [<pool>] ...
        get <"all" | property[,...]> <pool> ...
        set <property=value> <pool>

What does this mean?, How do I 'zpool online' correctly?, and, How can I identify the bad disk?

Googlin' and searchin' but thought I'd ask in the meantime.

Thanks

 

[cyberjock]

Have you tried a reboot of FreeNAS? Issues like this should be fixed with FreeNAS 9.1...

 

[paleoN]

'OK: The volume media (ZFS) status is HEALTHY
WARNING: The volume Music (ZFS) status is UNKNOWN:

The ZFS pool media is healthy and pool Music is unknown. From an SSH session, not the shell, as root output of the following:

Code:

zpool status -v Music

camcontrol devlist

glabel status

gpart show

What does this mean?, How do I 'zpool online' correctly?, and, How can I identify the bad disk?

It means you should read the actual man pages, man zpool. Blindly running commands is dangerous.

 

[travalon]

OK after checking the connections and rebooting, my volumes are reported as being healthy. However, my warning light now reports

Code:

    OK: The volume media (ZFS) status is HEALTHY
    WARNING: The volume Music (ZFS) status is UNKNOWN: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

Now my zpool status -v returns

Code:

[Travis@TravNAS] ~> zpool status -v
  pool: Music
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
  scan: scrub repaired 3.24M in 1h23m with 0 errors on Tue May 21 15:42:13 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        Music                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/822ba10c-8850-11e2-8f13-902b34e7c656  ONLINE       0     0   788
            gptid/82b1b732-8850-11e2-8f13-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

  pool: media
 state: ONLINE
  scan: scrub repaired 0 in 3h52m with 0 errors on Tue May 21 15:44:49 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        media                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/d5f8be87-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0
            gptid/d668abe7-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

camcontrol devlist

Code:

[Travis@TravNAS] /mnt/media# camcontrol devlist
<HL-DT-ST DVD-RAM GH22LP20 2.00>   at scbus0 target 0 lun 0 (pass0,cd0)
<WDC WD20EFRX-68AX9N0 80.00A80>    at scbus1 target 1 lun 0 (pass1,ada0)
<WDC WD20EFRX-68AX9N0 80.00A80>    at scbus2 target 1 lun 0 (pass2,ada1)
<Initio WD2500BEVE-00WZT 1.06>     at scbus3 target 0 lun 0 (pass3,da0)
<Initio WD2500BEVE-00WZT 1.06>     at scbus4 target 0 lun 0 (pass4,da1)
< Patriot Memory PMAP>             at scbus5 target 0 lun 0 (pass5,da2)

glabel status

Code:

[Travis@TravNAS] /mnt/media# glabel status
                                      Name  Status  Components
gptid/d5f8be87-67d1-11e2-b3d0-902b34e7c656     N/A  ada0p2
gptid/d668abe7-67d1-11e2-b3d0-902b34e7c656     N/A  ada1p2
gptid/822ba10c-8850-11e2-8f13-902b34e7c656     N/A  da0p2
gptid/82a42784-8850-11e2-8f13-902b34e7c656     N/A  da1p1
gptid/82b1b732-8850-11e2-8f13-902b34e7c656     N/A  da1p2
                             ufs/FreeNASs3     N/A  da2s3
                             ufs/FreeNASs4     N/A  da2s4
                            ufs/FreeNASs1a     N/A  da2s1a

gpart show

Code:

[Travis@TravNAS] /mnt/media# gpart show
=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>       34  488397101  da0  GPT  (232G)
         34         94       - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  484202703    2  freebsd-zfs  (230G)

=>       34  488397101  da1  GPT  (232G)
         34         94       - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  484202703    2  freebsd-zfs  (230G)

=>      63  15466433  da2  MBR  (7.4G)
        63   1930257    1  freebsd  [active]  (942M)
   1930320        63       - free -  (31k)
   1930383   1930257    2  freebsd  (942M)
   3860640      3024    3  freebsd  (1.5M)
   3863664     41328    4  freebsd  (20M)
   3904992  11561504       - free -  (5.5G)

=>      0  1930257  da2s1  BSD  (942M)
        0       16         - free -  (8.0k)
       16  1930241      1  !0  (942M)

Wow thanks for the quick response(s). The drives in question are a couple of lightly used IDE notebook drives in USB enclosures. I was happily surprised that they even worked with my cobbled together set up.
I'm taking note for future reference as well. As an after thought how long should a scrub on around 100GB of data take through a USB 2.0? The drives are 250GB though.

 

[paleoN]

I don't understand what this is telling me.

Several things:

1. Applications are unaffected.
   The actual data is correct and safe.
2. errors: No known data errors
   Self explanatory.
3. gptid/822ba10c-8850-11e2-8f13-902b34e7c656 ONLINE 0 0 788
   This is drive da0 which has a number of checksum errors. This is why it's flashing amber.

Smart information for da0:

Code:

smartctl -q noserial -a /dev/da0

 

[titan_rw]

If the drives are USB, and the original error was "cannot be opened", I assume the user suffered a usb disconnect issue. Rebooting the box brought the drives back online, and showed the checksum errors as the drives would no longer be in sync.

I think smartctl is going to need additional parameters in order to read smart info through the usb enclosure. If it supports them at all. I know some work, and some don't.

Kind of why usb drives are not recommended. USB is kind of flaky at the best of times.

 

[travalon]

I had read usb were not the interface of choice but IDE interfaces don't seem to be supported in Freenas. (Maybe they'll add it later) I figured what the hey I'll give it a shot. The connections did look a little loose so I gave them a shove and got the previously posted info. A strange thing I just noticed, when I view the disks,the HDD stand by and advanced power management settings I had set have reverted back to defaults. I'm wondering if this could have been caused by a power outage. (There seems to be more than average here)

Code:

[Travis@TravNAS] /mnt/media# smartctl -q noserial -a /dev/da0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

/dev/da0: Unknown USB bridge [0x13fd:0x1040 (0x106)]
Smartctl: please specify device type with the -d option.

Use smartctl -h to get a usage summary

So I started trying all the codes listed and this got me a response:

smartctl -q noserial -a /dev/da0 -d scsi

Code:

[Travis@TravNAS] /mnt/media# smartctl -q noserial -a /dev/da0 -d scsi
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               Initio
Product:              WD2500BEVE-00WZT
Revision:             1.06
User Capacity:        250,059,350,016 bytes [250 GB]
Logical block size:   512 bytes
Device type:          disk
Local Time is:        Wed May 22 09:12:57 2013 MST
Device does not support SMART

Error Counter logging not supported
Device does not support Self Test logging

all the usb codes returned this:

Code:

[Travis@TravNAS] /mnt/media# smartctl -q noserial -a /dev/da0 -d usbsunplus
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Smartctl: Device Read Identity Failed: scsi error unsupported scsi opcode

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

the codes with [] failed be cause I don't know what I'm doing.

atacam:

Code:

[Travis@TravNAS] /mnt/media# smartctl -q noserial -a /dev/da0 -d atacam
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

(pass3:umass-sim0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 01 00
(pass3:umass-sim0:0:0:0): CAM status: Function Not Available
(pass3:umass-sim0:0:0:0): ATAPI_IDENTIFY. ACB: a1 00 00 00 00 40 00 00 00 00 01 00
(pass3:umass-sim0:0:0:0): CAM status: Function Not Available
Smartctl: Device Read Identity Failed: Unknown error

Blind squirrel found a nut:

Code:

[Travis@TravNAS] /mnt/media# smartctl -c /dev/da0 -d scsi
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

SCSI device successfully opened

Use 'smartctl -a' (or '-x') to print SMART (and more) information

Then I lost it.
I went back and scrubbed the Music pool again and then ran the zpool status -v again and it says repairing:

Code:

[Travis@TravNAS] /mnt/media# zpool status -v
  pool: Music
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
  scan: scrub in progress since Wed May 22 09:06:16 2013
        29.7G scanned out of 105G at 18.3M/s, 1h9m to go
        2.08M repaired, 28.37% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        Music                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/822ba10c-8850-11e2-8f13-902b34e7c656  ONLINE       0     0 1.26K  (repairing)
            gptid/82b1b732-8850-11e2-8f13-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

  pool: media
 state: ONLINE
  scan: scrub repaired 0 in 3h52m with 0 errors on Tue May 21 15:44:49 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        media                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/d5f8be87-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0
            gptid/d668abe7-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

This look positive to me.

 

[travalon]

I guess it's not a good thing. the checksum went up I think.

Code:

[Travis@TravNAS] /mnt/media# zpool status -v
  pool: Music
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
  scan: scrub repaired 4.76M in 1h39m with 0 errors on Wed May 22 10:46:11 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        Music                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/822ba10c-8850-11e2-8f13-902b34e7c656  ONLINE       0     0 1.87K
            gptid/82b1b732-8850-11e2-8f13-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

  pool: media
 state: ONLINE
  scan: scrub repaired 0 in 3h52m with 0 errors on Tue May 21 15:44:49 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        media                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/d5f8be87-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0
            gptid/d668abe7-67d1-11e2-b3d0-902b34e7c656  ONLINE       0     0     0

errors: No known data errors

I guess I should unmount the drives until I can come up with a replacement.

My Disks are reported as healthy though.




Uploaded with ImageShack.us

 

[cyberjock]

Wow. All this pain just because of USB. Another point for USB and another fail for computer users. :(

USB just sucks for FreeNAS, which is why I tell people not to waste their time. When it works, it works. When it doesn't, getting your data back becomes very difficult.

Good luck!

 

[travalon]

I'm going to take your advice to heart and copy to the 2tb drives then pull the usb drives for some other use. Thank you guys for the help and the learning experience. I am marking this one as solved. If I can figure out how to +1 for all of you.

 

[cyberjock]

I have a friend that had to do this.... and it might work for you if you get desperate(aka don't have a backup of the data on your USB drives).

If you break the case open and pull the drive out you should be able to plug them into your computer(thereby making them internal drives) and you may have better luck. Just keep in mind to not use a hammer or something like that because you could damage the hard drive even more. Just pry it open(or unscrew it depending on the design) to get at the drive.

 

[titan_rw]

He said he's running them usb because freenas didn't support the IDE controller on the MB.

Honestly, IDE drives are old enough, and small enough, I don't really see the point in bothering. I have a bunch of 500 gig sata hard drives that I don't know what to do with. They're just too small to bother putting in a nas.

The pool seems to be working well enough for data recovery at this point. I'd copy everything to alternative storage, and ditch the IDE drives.

 

[gpsguy]

*** deleted *** since it wasn't relevant

 

[travalon]

SSHed in, simple cp -r command and every things on the other drives. Going to un-mount the usb volumes and pull them today. Thanks again to everyone who responded.