FreeNAS Degraded volumes and missing disks

[psionicsin]

Using FreeNAS 8.0.2

We have a server that has 4x2TB drives in it in a ZFS array in a mirrored array resulting in 4TB of storage. After being gone for about a week and starting it up again, we were presented with a flashing Red light telling us the volume was degraded. I found the offending disk (ada0) and pulled it in order to error check it in another computer before ruling the disk as the problem. The disk was 100% clean.

I totally wiped the disk and re-formatted it. After that I added it back into the array by creating a new volume and naming that volume the exact name as the volume we use. I added it to the array, but is now showing that the volume has 5 disks instead of 4 and that disk ada0 is still missing/degraded.

I attempted to remove the disk while shut down and restart it, it doesn't boot into the screen with the IP address. So I plugged the disk back up and now the server won't get pass a section of the start-up with grey text instead of white, and it's talking about:

Entropy Harvesting: kickstart
Starting file systen checks:
/dev/ufs/...

Mounting local file systems

What can/should I do?

 

[psionicsin]

Ok it's booted up, but now there's a yellow light flashing. It says "WARNING: The volume Comet (ZFS) status is " and that's it.

Under storage I'm getting "Comet /mnt/Comet None (Error) Error getting available space Error getting total space "

How can I remedy this? Is there a way to rebuild the array? Seems like the v8 GUI has less features than the old version as far as ways to rebuild/fix things.

 

[ProtoSD]

Can you post the output of "zpool status -v" and "gpart show"?

 

[psionicsin]

Ok so weirdly enough the problem seems to have corrected itself and I'm getting a green light. However when I go into disks, I'm still seeing 5 disks...the 5th obviously being one that was removed from the original 4 and then reinserted.

How can I REMOVE the listing of that disk so the volume doesn't think to try adding that to the pool all the time?

 

[psionicsin]

ZPOOL STATUS -V:

pool: Comet
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Comet ONLINE 0 0 0
mirror ONLINE 0 0 0
gptid/ff8eeea5-cc2f-11e1-8e7f-50e549534c96 ONLINE 0 0 3
gpt/ada1 ONLINE 0 0 0
mirror ONLINE 0 0 0
gpt/ada2 ONLINE 0 0 0
gpt/ada3 ONLINE 2 14.6K 2

errors: No known data errors



["ff8eeea5-cc2f-11e1-8e7f-50e549534c96" should be ada0]

 

[psionicsin]

GPART SHOW:

=> 63 3950478 da0 MBR (1.9G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 45549 - free - (22M)

=> 0 1930257 da0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)

=> 34 3907029101 ada0 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 ada1 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 ada2 GPT (1.8T) [CORRUPT]
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 ada3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

 

[psionicsin]

Here are image captures of the storage and disk sections of the GUI:

 

[peterh]

Ok so weirdly enough the problem seems to have corrected itself and I'm getting a green light. However when I go into disks, I'm still seeing 5 disks...the 5th obviously being one that was removed from the original 4 and then reinserted.

How can I REMOVE the listing of that disk so the volume doesn't think to try adding that to the pool all the time?

As you screwed up your pool ( hint : read the documentation about howto replace a failed disc) you have to
save your data on another media , destray your pool and rebuild it from scratch with the original ( 4-disk) configuration.

sorry to sound harsh.

 

[psionicsin]

As you screwed up your pool ( hint : read the documentation about howto replace a failed disc) you have to
save your data on another media , destroy your pool and rebuild it from scratch with the original ( 4-disk) configuration.

sorry to sound harsh.

That's the absolute ONLY way to get rid of that 5th disk?
If so, then I'll have to keep it there for now as we're in our busy season and do not have the space or time to backup 2.7 TB of information.

And I did read how to replace a failed disk, however when attempting to I kept receiving a error from both the GUI and the command line. My only option was to pull the disk.

 

[survive]

Hi psionicsin,

If I follow what happened correctly I think you might be at pretty significant risk right now. Best I can tell from the description in the original post you have 1 disk missing from your pool of mirrored drives, so if you lose the other disk in the mirror you lose it all and you have the drive that was reporting errors as a third vdev in your pool and if that drive goes you lose it all.

I would encourage you to reconsider letting this ride through your busy season.

-Will

 

[paleoN]

As both peterh & survive have already said you are still in trouble. What's more your GUI screenshot doesn't appear to match with the output of your zpool command. Perhaps that's old though?

Do me a favor and rerun, as root, both zpool status -v and gpart show. Throw the output inside of some [code][/code] tags as it will preserve the formatting and keep my eyes from crossing.

 

[psionicsin]

I would encourage you to reconsider letting this ride through your busy season.

Hey Will, and thanks for the concern. Since then I've thoroughly convinced her that if she didn't invest in a secondary backup solution that I would magically be unreachable when she inevitably has an issue come up. That changed her tune really quick. We've purchased adequate storage along with a controller card to place in one of our computers to do nightly/weekend backups when we're not in the office.

We're working in a Windows environment, so what's the best backup software that easy to use and stable?

 

[psionicsin]

As both peterh & survive have already said you are still in trouble. What's more your GUI screenshot doesn't appear to match with the output of your zpool command. Perhaps that's old though?

Do me a favor and rerun, as root, both zpool status -v and gpart show. Throw the output inside of some [code][/code] tags as it will preserve the formatting and keep my eyes from crossing.

Code:

[root@freenas] ~# zpool status -v
  pool: Comet
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: scrub completed after 9h48m with 0 errors on Sun Jul 22 12:51:12 2012
config:

        NAME                                            STATE     READ WRITE CKSUM
        Comet                                           DEGRADED     0     0     0
          mirror                                        DEGRADED     0     0     0
            gptid/ff8eeea5-cc2f-11e1-8e7f-50e549534c96  UNAVAIL      5 7.60M     0  cannot open
            gpt/ada1                                    ONLINE       0     0     0
          mirror                                        DEGRADED     0     0     0
            gpt/ada2                                    ONLINE       0     0     0
            gpt/ada3                                    UNAVAIL      0 15.1M     7  experienced I/O failures

errors: No known data errors
[root@freenas] ~# gpart show
=>     63  3950478  da0  MBR  (1.9G)
       63  1930257    1  freebsd  [active]  (943M)
  1930320       63       - free -  (32K)
  1930383  1930257    2  freebsd  (943M)
  3860640     3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992    45549       - free -  (22M)

=>      0  1930257  da0s1  BSD  (943M)
        0       16         - free -  (8.0K)
       16  1930241      1  !0  (943M)

Segmentation fault

 

[psionicsin]

Once these backups are complete, I can go about wiping and reconfiguring the server. Are there any precautions I should take this time when creating the array?

I know I'm going to be using the ZFS volume to make the 4x2TB HDDs into one 4TB volume. Any suggestions at all?

 

[survive]

Hi psionicsin,

I can't really tell you what's best for windows backup....but a successful backup using the built-in client is a thousand time better than nothing when it all goes pear-shaped, and just gets better from there! We really need more info to make a decent suggestion, for example do you want complete images of the disks, or do you just need the files? Personally I like "Cobain's Backup" for home use.

As to how do configure your disks I think there are really 2 choices: raidz2 or a pair of mirrors. It really gets down to what you need - performance or redundancy. A pair of mirrors will give you better performance at the cost of the ability to survive more combinations of double disk failure.

-Will