messed up 2-disk raid1, please advise

[tseeling]

Hello,

I think I have messed up a so-far working 2-disk raid1, and before proceeding I'd like to ask for advice. I'd rather try to repair for educational reasons than start from scratch and do a restore then.

I had a freenas 9.1.1 setup with 2 WD SATA disks, suddenly ada1 had lots of parity errors and was unavailable; smart reported there's a firmware upgrade available for this particular device type ("drive not accessible after powerup"). I don't think there's a coincidence but before buying a new HD I wanted to try if the FW upgrade helps with the other problem.

I attached the disk to another PC, created a boot CD from WD's iso image, upgraded the firmware, formatted with NTFS, made a full disk check, and moved it back to the freenas server.

The raid volume showed something strange instead of the /dev/xxx output I expected so I detached and removed the drive with zfs commands, with gpart deleted all partitions, destroyed the GPT and planned to re-add it as if it was a completely new device.

But whatever I try, neither CLI nor GUI, it's not possible to add the 2nd HD to the raid device ;(

I completely admit detaching was obviously some stupid action, so before I do any more damage I'd like to ask for advice on how to properly add a 2nd device to my degraded raid1. I thought I could simply "attach" a new device but it asks for a device-old and device-new although I don't have a device-old any more.

Thanks in advance ;)

 

[Dusan]

Let's get first the facts straight:

1. What commands did you use to detach the drive?
2. What commands/GUI actions are you trying to do to "attach" the drive? (I guess you are using the GUI, as the zpool attach command doesn't need an "old device")?
3. What is the output of zpool status? (please paste the output using CODE tags)

 

[tseeling]

I used zpool detach and zpool remove commands for /dev/ada1.

I wanted to attach a new device but I'm stuck at what the params mean.

Code:

nas# zpool attach nas1 /dev/ada1
missing <new_device> specification

this is the output for the zfs status command:

Code:

nas# zpool status -v
  pool: nas1
 state: ONLINE
  scan: resilvered 676K in 0h19m with 0 errors on Sun Oct 13 16:34:56 2013
config:

        NAME        STATE     READ WRITE CKSUM
        nas1        ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

 

[Dusan]

I see that your pool does not use GPTIDs -- normally the pool references something like gptid/f2c218ae-fe13-11e2-9e64-002590aa3178 instead of the short disk/partition names (ada0p2). How did you create the pool?

 

[tseeling]

I created the pool with the GUI from freenas 8.0.4, then some weeks ago upgraded to 9.1.1.

 

[Dusan]

OK, I did not use 8.0.4, so it's possible it did not use GPTIDs back then.
Two more questions:

1. Please provide output of gpart show /dev/ada0
2. Are your drives identical? (same model & size)

EDIT: replaced ada1 with ada0

 

[tseeling]

Code:

nas# gpart show /dev/ada0
=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

yes, I have 2 identical WD Barracuda ST2000DM001, apart from some internal S/N and Microcode (one with CC4H, one with CC29 level).

I also tried to recreate identical partitions with gpart similar to ada0, but the zfs partition had a different sector count for the zfs at index 2 (...696 instead of ...703), so I stopped and removed that without trying further.

 

[Dusan]

OK, so to create the same partitions we can copy the partition schema from the existing drive (FreeNAS creates by default a 2GB swap partition):
gpart backup ada0 | gpart restore ada1
And then let's attach the drive to the pool:
zpool attach nas1 /dev/ada0p2 /dev/ada1p2

 

[tseeling]

I think it looks good now. Still can't believe it's been that simple.
Thanks very much, I feel so relieved that I have a 2nd disk back in my storage.

Code:

nas# zpool status
  pool: nas1
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Oct 13 23:25:28 2013
        1.90G scanned out of 834G at 23.7M/s, 9h58m to go
        36.8M resilvered, 0.23% done
config:

        NAME        STATE     READ WRITE CKSUM
        nas1        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0 17.7K     0  (resilvering)

 

[Dusan]

A suggestion, if I might, for future :). When something unexpected happens don't start "randomly" clicking in the GUI / running CLI commands . First read the documentation (http://doc.freenas.org, http://www.freebsd.org/cgi/man.cgi?query=zpool), then simulate your problem in VM, try your solution and make sure that the actions you want to do have the desired effect ;).

 

[tseeling]

No, doesn't look good, the drive really is faulty, it was removed from the pool again.

Code:

nas# zpool status
  pool: nas1
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub in progress since Sun Oct 13 23:29:59 2013
        7.21G scanned out of 834G at 74.6M/s, 3h9m to go
        0 repaired, 0.86% done
config:

        NAME                     STATE     READ WRITE CKSUM
        nas1                     DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            ada0p2               ONLINE       0     0     0
            1683787713533217836  REMOVED      0     0     0  was /dev/ada1p2

So I'll have to buy a new disk anyway, the FW upgrade didn't help.

Thanks anyway, you're still my freebsd hero!

 

[Dusan]

You are welcome.
When you get the new drive, please use the GUI to replace it: http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive_or_SSD
It will properly create the partitions even if the new drive is not exactly the same (of course the capacity needs to be at least the same or larger).

 

[cyberjock]

8.0.4 did use gptids. At least, the past p-release did as that's when I first started experimenting with FreeNAS.

 

[Nomad]

Dusan,

Great information in this thread. Now I know what NOT to do and what to do :)