SOLVED Unable to replace failed, UNAVAIL disk

[unca_NAS]

When coming home from work I my NAS was making a loud mechanical sound, from one of the HD's.

Disk Adap2 was failed, status UNAVAIL, so it couldnt be offlined.

Shut down NAS, replaced the disk, rebooted, attempted to replace from GUI:
"Replacing disk 8566134870468231396
Member disk: This field is required"

Zpool status

Code:

[root@freenas] ~# zpool status
  pool: vault
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 1.38T in 23h17m with 0 errors on Sat Nov 22 08:26:24 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        vault                                           DEGRADED     0     0     0
          raidz1-0                                      DEGRADED     0     0     0
            gptid/6743f360-6f49-11e4-9d79-984be10883dd  ONLINE       0     0     0
            gptid/3eb40b02-714d-11e4-8e54-984be10883dd  ONLINE       0     0     0
            8566134870468231396                         UNAVAIL      0     0     0  was /dev/gptid/308dcd88-748e-11e3-a493-984be10883dd
            gptid/87875728-ce05-11e2-ba15-984be10883dd  ONLINE       0     0     0

errors: No known data errors

From CLI attempted the following commands:

Code:

zpool replace vault /dev/gptid/308dcd88-748e-11e3-a493-984be10883dd ada2p2

Code:

zpool replace vault 8566134870468231396 ada2p2

Error:

Code:

cannot open 'ada2p2': no such GEOM provider
must be a full path or shorthand device name

Looked thru the manual and forum, couldnt find a solution - but had only three hours of sleep last night, so I may have missed it.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Shutted down server, put original failed disk back on.

Zpool Offline > Zpool online. Zpool status shows that disk status is UNAVAIL > OFFLINE. Great

Shutted the server, replaced the HD, rebooted. Attempted to replace the disk via GUI
MEMBER DISK MISSING

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Over my head here. Obviously the re-onlining wasnt really working.

What is my next step?

 

[Ericloewe]

And just why are you not doing what we tell everyone to do and stick to the GUI???

Try following the manual's instructions to the letter, first of all. From there, we can see where to go.

 

[unca_NAS]

Because GUI never had an option to offline.
Believe me kind sir, I have absolotely no problem with doing evrything in GUI - but hey, without the buttons needed you pretty much dont have other option, do you?

Having Freenas User Guide 9.2.1 right here. Searching with "unavail" cant find anything usable.

 

[Ericloewe]

Because GUI never had an option to offline.
Believe me kind sir, I have absolotely no problem with doing evrything in GUI - but hey, without the buttons needed you pretty much dont have other option, do you?

It does, but you do have to follow the manual to the letter. I was in a similar situation the other day, with someone who couldn't find it. So I looked and couldn't find it either!
Turns out, there are two very similar screens and only one of them has the offline button.

Section 6.3.12 has the following line:

Before physically removing the failed device, go to Storage → Volumes → View Volumes → Volume
Status and locate the failed disk. Once you have located the failed device in the GUI, perform the
following steps:

1. If the disk is formatted with ZFS, click the disk's entry then its “Offline” button in order to
change that disk's status to OFFLINE. This step is needed to properly remove the device from
the ZFS pool and to prevent swap issues. If your hardware supports hot-pluggable disks, click
the disk's “Offline” button, pull the disk, then skip to step 3. If there is no “Offline” button but
only a “Replace” button, then the disk is already offlined and you can safely skip this step.

Emphasis mine. If you don't go there, you won't see the button.

 

[unca_NAS]

Been there. No button. Pls see the screen cap:


Shutdown - Replaced the original faulted HD - startup
Volume status as presented in the capture.

Code:

zpool online vault /dev/gptid/308dcd88-748e-11e3-a493-984be10883dd
warning: device '/dev/gptid/308dcd88-748e-11e3-a493-984be10883dd' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present

 

[cyberjock]

Yes, because disk is already offline. See the status of the drive (unavail).

Now you just need to get a new disk, put it in, then follow the manual for disk replacement after the disk is offline. Straight out of the manual it says if the disk is already offline it may not show the offline button. ;)

 

[unca_NAS]

Cyberjock - already tried that.
After swap-reboot attempted to replace via GUI:

Code:

Replacing disk 8566134870468231396
Member disk: This field is required" 

After shutdown-replace hd-startup-Volume status-choose replacement drive-> Replace-dialog-window the drop-down-menu thats supposed to have replacement-HD's "id" is empty. Zip. Zilch. Nada.

The sole reason I was using CLI in the first plave with this process was because manual VS real world didnt meet, so to speak.

 

[unca_NAS]

EDIT:

After second time zpool offline -> zpool online the faulted disk the needed replacement-disk ID appeared to the "Replace"-window and replacement was possible.

Really dont understand what happened, but with the immortal words of late great Sheck Exley: "What works, works."

 

[Starpulkka]

Had nearly same case myself but i had freenas put hdd to Faulted state instead of unavail. What i had to do was two reboots until freenas managed to replace my faulted state hdd.

By the way did you check that did it "fully" resilver that replacement hdd..

 

[unca_NAS]

AFAIK resilver was completed, although it was not successfull, state of the pool is still degraded. With some data errors Im copying the stuff out of the pool and starting a new pool from scratch.

BTW is there a way to abort the resilvering? ZPOOL SCRUB -S (Poolname) triggered error "cannot cancel scrubbing (pool): currently resilvering" Reason Im asking that half of the data is now copied to "swap-HD". I can unmount it but in order to replace it with another HD I'd need to reboot the NAS - which propably isnt such a good idea since resilver is still in progres... Resilvering is still 13 hours from complete so waiting half a day before I can continue the data-rescue op seems a very loooong time to wait..

 

[Starpulkka]

Hah i knew it, that its in "fishy" state.
Perhaps a memtest is in order.. like a.s.a.p. What i really would do is throw that piece of hp in a ser trashcan and do proper hardware and raidz2..
Do you have replacement hdd same size as old?

 

[unca_NAS]

Have both 2tb and 3tb spares. How come?

Planning to gradually add storage to 4 X 3tb. Flashing the BIOS and updagrading the memory to 16 GB ECC is already in to-do list.

 

[Starpulkka]

Hope for best, i tried replace 500GB to 3TB and i had not good results. Yours results actually might differ by having newer ZFS.