How to replace a HDD in a raidz1 in 8.0.1 Beta 4

[globus999]

Here we go again! Beware that it is NOT possible to do a straight raidz1 in-place hdd replacement in Beta 4. The *only* procedure that works is the one I'll describe below, but it is a pain in the neck. Nothing else works (you will just corrupt your data):

Steps:

1 - Determine which hdd you will replace (let's say ada7) in a raidz1 (let's call it tank1)
2 - Id the physical disk
3 - Shut down the system
4 - Pull ada7 out of the system and replace it with a new disk in the same port
5 - Power-on the system (tank1 will be in a degrade state and /dev/ada7 unavailable)
6 - Issue a zpool replace tank1 ada7
7 - Let resilvering finish
8 - Issue a zpool detach tank1 /dev/ada7/old
9 - Issue a zpool export tank1
10 - Re-install FreeNAS from scratch - YES - you heard me right, from scratch.
11 - Do *NOT* restore your existing configuration. It will contain the info from the old hdd and will screw-up during boot time. You *must* re-input all the variables / configurations by hand.
12 - Do and Autoimport from the GUI to import tank1

Now, what kills me is that there is no freaking way to do this in *any* other way.

Few things that do not work:

1 - Use HDD Replace In-Place from the GUI: error message - no action - new bug ticket created.
2 - Export from GUI and re-import from CLI: no mount point - import does not survive reboot - further import will corrupt your data since raidz1 is in degraded state.

 

[justhiggy]

I had the same problem with my raidz2 volume. A spare kicked in so I didn't have to do steps 2-5, but the rest applied to me also. Thanks for the post. It would be nice if the GUI would update from live zpool data rather than from a database store. Or at least you could have the option to re-query the pools for update info.

Also, recovering from a failed drive, with or without a spare, should work better from the GUI.

 

[billm]

It's not often I get my question answered before I ask it! Thank you globus999!

I've posted the output of "zpool status Data1" from my system at the end of this post.
Based on that I think I'm ready for step 8 in your list. I did the other steps BEFORE reading your post, so I'm looking for confirmation that I'll be o.k.
Should my step 8 be:
zpool detach Data1 /dev/ada3p2/old
and 9:
zpool export Data1
?
And I'm very ready to do a from scratch install since the config is messed up from upgrading to the beta series from 8.0 release, so your list of steps is very topical.

Thanks!

*****************
freenas# zpool status Data1
pool: Data1
state: DEGRADED
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Data1 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2p2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
12596499309562694375 UNAVAIL 0 0 0 was /dev/ada3p2/old
ada3p2 ONLINE 0 0 0

errors: No known data errors

 

[stratege]

6 - Issue a zpool replace tank1 ada7
7 - Let resilvering finish
8 - Issue a zpool detach tank1 /dev/ada7/old
9 - Issue a zpool export tank1
10 - Re-install FreeNAS from scratch - YES - you heard me right, from scratch.
11 - Do *NOT* restore your existing configuration. It will contain the info from the old hdd and will screw-up during boot time. You *must* re-input all the variables / configurations by hand.
12 - Do and Autoimport from the GUI to import tank1

WHOAAAA !!!!
dumb mode on !!!
after reading from this post, i have a real simple question here:
- what is the use of freenas ???
So far so bad: if you have to redo a from scratch install at each failure (yes, failure do occur !!! ) i prefer to go back, stop using a nas i revert to a double backup using usb hdd thing

This will save my time and money, and save time for your headheck answering dumb post like mine !!!!
dumb mode off

Ok, let's be serious again:

Step 6: how to do it ?? seem the gui is not explicit !!! What do you call a zpool replace ....
Same for step 8 and 9

STEP 10 ARE YOU REALLY SERIOUS HERE !!!! There is no other way ???

 

[billm]

I think what globus999 did not mention in his list of steps is between steps 5 & 6 logging into the freenas box via ssh and using the su command to switch to the root account. That takes some basic familiarity with unix shells, what many here call the "CLI" (command line interface). I don't know if there is a tutorial on accessing shell accounts floating around freenas.org, but that would be the thing to find and study.

Freenas is tricky because most of the configuration files are replaced on boot from a database that only the GUI can modify. So, (I'm guessing here) you can fix the raidz from the shell, but you need to import it via the GUI so those settings will survive a reboot.

And, finally, since this is a beta release (B4) there are bugs in the GUI implementation and the only safe way to get the raidz back is to start from a fresh config.

 

[Tekkie]

Actually the beta release has nothing to do with this particular problem, the official 8.0 release cannot replace a drive either... lets just say the dev team let this one slip through their fingers. ;)

 

[ProtoSD]

For a more reasonable STEP 10 alternative, go back to the GUI, System -> Settings -> Config, restore factory settings.

If you want to ID the disk and get the serial number so you can verify you've pulled the right one, do:

smartctl -i /dev/name-of-failed-disk for example 'smartctl -i /dev/ada0'

After you boot back up with the new drive installed, do like globus999 says (from the command line with ssh) 'zpool replace your-pool name-of-original-bad-disk for example 'zpool replace tank /dev/ada3'

This step can take HOURS if you have a lot of data. It needs to 'resilver' and you can check the status with a 'zpool status -v'

When it's finished, the status may still say DEGRADED, don't freak out. It thinks there's a ghost disk that's bad still in the arrray, so from the command line again, like globus999 says, do a 'detach' on the out of place device that says 'UNAVAILABLE'.

Check your status again with the 'zpool status -v' and that ghost device will disappear and your status should change from DEGRADED to ONLINE.

Hope that helps to dissipate some anxiety... ;)

 

[stratege]

welllllll, i love step 13:

- step 13: bang your head on a wall if you get strange bahaviors !!!


Tried to follow the step by step, (yes, i am a IT geek) but something went wrong.
So far so bad: my raidz2 simply died, all data lost.

Hopefully, a full data backup was done.

Now, i would love to have a freenas offering the same "GUI-easy-of-use-without-bug-and-all-functionality-for-dumb-user-like-me-and-my-family" as openfiler ( or like freenas v7)
Of course, i did a full reinstall, a full format of disk-array and spend a few hours to retransfer my data back to freenas v8 ... pfeuuuu


Anyway, a message to the devell team: Keep up the good work. If you have a priority list, i really think that disk management should be more explicit, more easy to use. The old freenas disk management from v7 was a very good example of what should be done.

Actually, my disk array is a 5 X 1TO. I am thinking of "upgrading" to a 5 X 3TO, not restarting from scratch !!! So i would love a little GUI-disk-management-flexibility ... Thanks

 

[globus999]

Man! so sorry... I feel you. I tried all kinds of combinations before developing the list above.
I don't use SSH since I am on-site so I can log into TTY directly, but SSH will do nicely too.
One thing that *may* work (although in my case was a hit and miss) is to do an export/import after step 10 - But it may not work. This is, once the zpool is "clean" do an Export from GUI (cross your fingers and make sure you have a lemon hanging over the freenas box) and then an Automatic Import. Sometimes it works.
Wrt reset to factory defaults, I had some nasty experiences. Apparently there are some leftovers in the freenas SQL DB. It may or may not work.
Last note, it is not advisable to store and restore the config file since it will have the old HDD config settings in it.

 

[audix]

I am happy to report that in 7226 it works by doing a "replace" and "detach" from the GUI to replace a broken disk!

The one thing is that when doing a "zpool status" the new disk is shown by its device name (ada6p2) and not the longer "gptid/***xx" as all the others are.

 

[Tekkie]

As I've had yet another Samsung F4 fail on me I need to replace it, I am running RAIDZ2 which means I am still good but the question is should I wait with the replacement until Beta5 comes out or grab a nightly build?

 

[Tekkie]

RC1 allowed me to replace the drive, however zpool status shows the drive with its device name and not the GUID, should I worry?

 

[creepwood]

I'm not sure if this is the thread to ask this in, but my RAIDZ1 volume has accidently picked up another drive ADA4 (250gb) and replaced my ADA3 (2TB) in my 4 x 2TB RAIDZ. I don't know what to do, or how to do it to get my 2TB ADA3 drive back into the mix instead of my 250GB ADA4 drive, that isn't even connected right now, still my volume in the gui says everything is fine.

Can I do a replace with a drive on another "channel" the same way as this?

 

[Tekkie]

Start a new thread this sounds funky, accidently means you pressed a button right? :)

 

[dalex]

11 - Do *NOT* restore your existing configuration. It will contain the info from the old hdd and will screw-up during boot time. You *must* re-input all the variables / configurations by hand.

Now, what kills me is that there is no freaking way to do this in *any* other way.

What if you EDIT your backuyp file nas*.db with SQLITEBROWSER (http://sqlitebrowser.sourceforge.net/):

0 - Take a configuration backup if you don't have any recent.

1 - 5 your steps

6 - Edit db file and replace the correct (new) disk serial number.

7 - Restore the new db from gui and resilver (auto?).


This is how the field looks:

Table: storage_disk, Field: disk_identifier: {serial}WD-WMAY02272363

And this is an extract from the SQL export of the DB file:

CREATE TABLE "storage_disk" ("disk_acousticlevel" varchar(120), "disk_hddstandby" varchar(120), "disk_identifier" varchar(42), "disk_togglesmart" bool, "disk_advpowermgmt" varchar(120), "disk_transfermode" varchar(120), "disk_description" varchar(120), "disk_group_id" integer, "disk_smartoptions" varchar(120), "id" integer PRIMARY KEY, "disk_name" varchar(120));
INSERT INTO storage_disk VALUES('Disabled','Always On','{serial}WD-WMAY01907303',1,'Disabled','Auto','Member of big raidz2',1,'',1,'da4');
INSERT INTO storage_disk VALUES('Disabled','Always On','{serial}WD-WMAY02272224',1,'Disabled','Auto','Member of big raidz2',1,'',2,'da5');


In FreeNAS 7 i did it many times since it's a simple XML you can easily edit.

 

[ProtoSD]

This shouldn't be necessary anymore with 8.02, but there still are some situations where it's still useful, like renaming your pool, or when you have more than one pool and want to delete it and reuse the disks. The database seems to hang on to the old disk info and you need to go in and delete the old disk info or it won't let you reuse the disks.

 

[samfarkus]

not worth the hassle

all this raid/zfs stuff i randomly get 'need to fsck' and lost my data so many times i asked myself 'whats the point of a backup'

then i just created a ufs file system and copied the files over with rsync.. i can see them there and theres no stripes or raids or whatever to worry about

And i can sleep at night knowing if my hard drive crashes i have a real backup that doesnt require a 50 step process to recover.

yes it is inefficient drive wise but it works.

 

[dalex]

all this raid/zfs stuff i randomly get 'need to fsck' and lost my data so many times i asked myself 'whats the point of a backup'

I don't agree.

I also have my backups there, but i never had to use them. Only to retrieve some files accidentally erased by users...

My UFS raid 1 disks and my RAIDZ1/2 pools never lost a byte, although i had 3 disks failed (in a total of 18 disks) in a 4 years period.