Drive went offline, Degraded ZPool, Back Online, Still Degraded?

[Fattrain]

So this is my first time super n00b post. Long story short I have a long history in IT but am really just starting to cut my teeth with FreeBSD / FreeNAS storage system.

I have a 5 x 3tb drive setup and my daily report came across my phone and said "Degraded"! Aghhh!!! So I logged in and took a look using ZPool status and sure enough one of my drives was "offline".

I rebooted, checked bios to make sure all 5 drives were showing and they were. I opened a CLI in FreeNAS and did "camcontrol devlist" and all 5 drives show there as well. I then did gpart show and found that "ada1" wasn't listed.

Following some threads I found online (hopefully not to a fault) I did a zpool online TrainData /gptid/efe98h893hj3 (just an example of the command)

Now my greenlight shows in the GUI like everything is OK but I don't trust it because when I do a ZPool status -v it still shows funky stuff (see 2 posts down)

Any help from you FreeNAS guru's would be MUCH appreciated. Thanks in advance!


EDIT: Posted originally in "quote" form (Doh!) and fixed it 2 posts down

 

[cyberjock]

You need to repost the output of zpool status -v in code tags. The formatting does matter. :P

 

[Fattrain]

Crap. Sorry guys. I apologize. Coming right up ;)

Code:

[root@freenas] ~# zpool status -v
  pool: TrainData
state: DEGRADED
  scan: scrub repaired 0 in 3h44m with 0 errors on Sat Jul 20 10:02:29 2013
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        TrainData                                      DEGRADED    0    0    0
          raidz1-0                                      DEGRADED    0    0    0
            gptid/e8bd8d19-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            replacing-1                                DEGRADED    0    0    0
              14473487319103256850                      UNAVAIL      0    0    0  was /dev/gptid/e920d7eb-8558-11e2-8829-08606ecff386
              ada1                                      ONLINE      0    0    0
            gptid/e9830e2e-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            gptid/e9ed7158-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            gptid/ea50fab4-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
 
errors: No known data errors

Code:

[root@freenas] ~# gpart show
=>        34  5860530988  ada0  GPT  (2.7T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856336584    2  freebsd-zfs  (2.7T)
  5860531016          6        - free -  (3.0k)
 
=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128          7        - free -  (3.5k)
 
=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128          7        - free -  (3.5k)
 
=>        34  5860533101  ada4  GPT  (2.7T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128          7        - free -  (3.5k)
 
=>      63  30941121  da0  MBR  (14G)
        63  1930257    1  freebsd  [active]  (942M)
  1930320        63      - free -  (31k)
  1930383  1930257    2  freebsd  (942M)
  3860640      3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  27036192      - free -  (12G)
 
=>      0  1930257  da0s1  BSD  (942M)
        0      16        - free -  (8.0k)
      16  1930241      1  !0  (942M)

 

[cyberjock]

It looks like you've replaced a disk at some point. You didn't complete the disk replacement procedure in accordance with the manual. That's why its showing the old and new disk and is saying degraded.

 

[paleoN]

Code:

[root@freenas] ~# zpool status -v
  pool: TrainData
state: DEGRADED
  scan: scrub repaired 0 in 3h44m with 0 errors on Sat Jul 20 10:02:29 2013
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        TrainData                                      DEGRADED    0    0    0
          raidz1-0                                      DEGRADED    0    0    0
            gptid/e8bd8d19-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            replacing-1                                DEGRADED    0    0    0
              14473487319103256850                      UNAVAIL      0    0    0  was /dev/gptid/e920d7eb-8558-11e2-8829-08606ecff386
              ada1                                      ONLINE      0    0    0
            gptid/e9830e2e-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            gptid/e9ed7158-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
            gptid/ea50fab4-8558-11e2-8829-08606ecff386  ONLINE      0    0    0
 
errors: No known data errors

Was this, scan: scrub repaired 0 in 3h44m with 0 errors on Sat Jul 20 10:02:29 2013, after you onlined whatever drive was offline? If not scrub the pool first. If it was after than zpool detach the old device via it's numeric ID. FYI, GUI drive replacement works quite well now.