SOLVED Did I just get 7 years of bad luck (offline disk)

[Naesstrom]

So I was going to replace a disk tonight in a mirror, tired after working overtime so I clicked offline instead of replace, added the new disk and couldn't import it.
Now my Pool status looks like this:

ada1 was the drive I turned offline apparently and ada0 is now alone and not mirrored? Storage says I have almost 2TB free and before offlining it I had 20Gb so have I screwed everything up?
What I've read it should get back up online after a reboot but that doesn't happen... I can see the disk and that it's Unused

zpool status:

Code:

  pool: Katrinelund_2
 state: ONLINE
  scan: scrub in progress since Wed Oct  7 19:39:13 2020
        168G scanned at 403M/s, 7.20G issued at 17.3M/s, 19.3T total
        0 repaired, 0.04% done, 13 days 13:08:21 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Katrinelund_2                                   ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/7add9e49-4d6b-11e3-b49d-18a90539fec5  ONLINE       0     0     0
            gptid/7b9d228b-4d6b-11e3-b49d-18a90539fec5  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/ff0d3747-6b54-11e4-ad25-18a90539fec5  ONLINE       0     0     0
            gptid/313898ef-9e88-11e6-806e-6466b30506f0  ONLINE       0     0     0
          gptid/ee705407-a502-11ea-b336-6466b30506f0    ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            gptid/186c1abb-f679-11e9-9f05-6466b30506f0  ONLINE       0     0     0
            gptid/d6745124-d79b-11e8-b0d4-6466b30506f0  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:04:12 with 0 errors on Mon Oct  5 03:49:12 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            da1p2   ONLINE       0     0     0

errors: No known data errors

 

[HoneyBadger]

You'll need to re-add the ada1 device back to the mirror. Allegedly there's support for this in the TN12 GUI but I don't think it's anywhere in the FreeNAS 11.x branches, even the latest 11.3-U5

There's commands for doing this from a shell though, including creating the 2G swap partition. Ensure that you use your own gptid or rawuuid values of course.

1. gpart create -s gpt /dev/ada1
2. gpart add -b 128 -t freebsd-swap -s 2G /dev/ada1
3. gpart add -t freebsd-zfs /dev/ada1
4. Run zpool status and note the gptid of the existing disk/partition you want to mirror (in your case this should be gptid/ee705407-a502-11ea-b336-6466b30506f0
5. Run gpart list and find the gptid of the newly created partition. It is the rawuuid field. In this example it would be rawuuid of ada1p2
6. zpool attach [yourpoolname] /dev/gptid/ee705407-a502-11ea-b336-6466b30506f0 /dev/gptid/[gptid_of_the_new_partition]

Bold warning: MAKE SURE you use "zpool attach" and NOT "zpool add".

Once it's completed and resilvered, running a manual scrub would not be a bad idea.

 

[Naesstrom]

5. Run gpart list and find the gptid of the newly created partition. It is the rawuuid field. In this example it would be rawuuid of ada1p1

Thanks a million, then there's still hope atleast! Going through your steps now, shouldn't it be ada1p2 that I attach. ada1p1 is the name of the one that was created for the swap space? Just want to be sure I do it right!
gpart show:

Code:

=>         40  15628053088  ada1  GPT  (7.3T)
           40           88        - free -  (44K)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  15623858696     2  freebsd-zfs  (7.3T)

 

[HoneyBadger]

Thanks a million, then there's still hope atleast! Going through your steps now, shouldn't it be ada1p2 that I attach. ada1p1 is the name of the one that was created for the swap space? Just want to be sure I do it right!

Oop. Yes, yes it should be. Too many times changing the ada0 in an example to ada1, I suppose. Good eye there - obviously you won't be able to use the 2G swap partition to mirror your previous drive.

This should hopefully cure what ails you, and turn your "seven years bad luck" (from a broken mirror) into "seven terabytes of resilvering" - less, if your drive isn't as full. :)

 

[Naesstrom]

Oop. Yes, yes it should be. Too many times changing the ada0 in an example to ada1, I suppose. Good eye there - obviously you won't be able to use the 2G swap partition to mirror your previous drive.

This should hopefully cure what ails you, and turn your "seven years bad luck" (from a broken mirror) into "seven terabytes of resilvering" - less, if your drive isn't as full. :)

OMG Yes... Can I buy you a beer or something!
Next time I'll skip the manual and just unplug the old drive! :D

Code:

  pool: Katrinelund_2
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Oct  8 15:06:18 2020
        741G scanned at 1.57G/s, 566G issued at 261M/s, 19.3T total
        0 resilvered, 2.86% done, 0 days 20:54:46 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Katrinelund_2                                   ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/7add9e49-4d6b-11e3-b49d-18a90539fec5  ONLINE       0     0     0
            gptid/7b9d228b-4d6b-11e3-b49d-18a90539fec5  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/ff0d3747-6b54-11e4-ad25-18a90539fec5  ONLINE       0     0     0
            gptid/313898ef-9e88-11e6-806e-6466b30506f0  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/ee705407-a502-11ea-b336-6466b30506f0  ONLINE       0     0     0
            gptid/76c318cd-0926-11eb-b2ba-6466b30506f0  ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            gptid/186c1abb-f679-11e9-9f05-6466b30506f0  ONLINE       0     0     0
            gptid/d6745124-d79b-11e8-b0d4-6466b30506f0  ONLINE       0     0     0

errors: No known data errors

 

[HoneyBadger]

Glad to see it's moving along! Let the resilver complete, run a manual pool scrub afterwards to be safe, and then you're back to full health.