Replacing a drive in FreeNAS 8.0.3

[zombieapocalypse42]

Hello all,

I have several questions regarding FreeNAS. Currently I have FreeNAS 8.0.3 (64bit) running on a HP N40L Microserver with 8GB of RAM and a Intel EXPI9301CTBLK NIC. Currently I'm testing it with 3 2TB WD drives (one EARX and two EARS without the jumpers). The biggest issue I have is trying to figure out what the "proper" method is for replacing a disk.

Originally I had two WD drives and one Hitatchi drive installed to discover the Hitatchi drive is not advance format so I wanted to remove that. I figured simple enough. Remove the drive and replace it according to the 8.0.3 user guide. What happened was not a simple experience and ended with results I can't explain. So the steps in the user guide are either not correct or for some reason did not work for me. Somehow in the process the installation on the USB disk got hosed. Anyhow after all was said and done I managed to get the disk "replaced" but somehow I ended up with ZFS datasets that did not match the space. So for instance I had:

/mnt/tank-Size 2.1TB (Was 3.6 before the replacement attempt. I have no idea why or how it magically shrunk. I'm guessing something with the replacing of the disk didn't work correctly. Although the resilvering and such reported no issues.)
/mnt/tank/Movies - size 3.3TB (How is the dataset somehow magically bigger than the pool?)

I'm glad I tested this before I had completely switched over from Windows Home Server v1 so no data was lost. Just some time. I've tried some other options like Nexenta and Openfiler but neither really worked well and I like FreeNAS. I just need to better understand. I'm OK with a "process" so long as I know what it is.

So some other questions:
-When replacing a disk do you have to preformat it to ZFS to get a certain layout such as 4K sectors or swap space? Maybe that was the issue before I just added it to the pool but didn't format it similar to the other disks (FreeNAS formatted the entire drive as ZFS without any swap. If someone could tell me how to properly align all the partitions that would be appreciated.)
-Any idea on how long a "long" SMART test takes on a 2TB drive? Recommended frequency for running one?

I hope that all makes sense and someone can help out. I'd just prefer to have everything ironed out before I transfer all the data to the box. However, regardless there will be backups :)

 

[ProtoSD]

Hi Zombie,

I can't answer all of your questions because I recently had my own "experience" replacing a disk in 8.03-p1 and it also failed. I knew enough to complete it manually from the command line, but...

I can't answer the auto-magically shrinking datasets / size differences.

You don't have to preformat your disk for ZFS, BUT...

If you created your volume/pool from the GUI with the default swap partitions, you'll need to pre-partition your disk.

One way to see how your existing disks are partitioned is by doing "gpart show". Somewhere here in the forums (under the bugs section) should be the steps I used to do the pre-partitioning (I think) and I can find it out if my memory is wrong.

After the pre-partitioning you just basically do a zpool replace, again the steps I used are in that post. When you're done you may not end up with a "pretty" disk name when you do a zpool status. Let me find my post and update this post with a link.

Hopefully someone else can answer your other questions.

EDIT: Here's the link- http://forums.freenas.org/showthread.php?5810-Replace-Disk-fails-from-GUI-in-8-0-3-p1

I was wrong about the steps I used being in that post, but if you look CLOSELY at the output from the log file you can see how it gets done from the GUI

 

[zombieapocalypse42]

Hi Zombie,

I can't answer all of your questions because I recently had my own "experience" replacing a disk in 8.03-p1 and it also failed. I knew enough to complete it manually from the command line, but...

I can't answer the auto-magically shrinking datasets / size differences.

You don't have to preformat your disk for ZFS, BUT...

If you created your volume/pool from the GUI with the default swap partitions, you'll need to pre-partition your disk.

One way to see how your existing disks are partitioned is by doing "gpart show". Somewhere here in the forums (under the bugs section) should be the steps I used to do the pre-partitioning (I think) and I can find it out if my memory is wrong.

After the pre-partitioning you just basically do a zpool replace, again the steps I used are in that post. When you're done you may not end up with a "pretty" disk name when you do a zpool status. Let me find my post and update this post with a link.

Hopefully someone else can answer your other questions.

EDIT: Here's the link- http://forums.freenas.org/showthread.php?5810-Replace-Disk-fails-from-GUI-in-8-0-3-p1

I was wrong about the steps I used being in that post, but if you look CLOSELY at the output from the log file you can see how it gets done from the GUI

Protosd,

Thanks for the response. Your link made me think just to recreate it and copy the commands out. Just so it's exactly the same in case I ever need to do it. That's the closest thing I can think of with the dataset size being wrong is that it's because the partitioning is different. Which I would have thought would be handled by ZFS because the documentation doesn't specify anything about it but that's OK. Live and learn.

I guess besides the SMART questions I'll just have to test out the drive replacement then I'll be good to go. I'll post back once I get a chance to try it to see if it happens again.

 

[zombieapocalypse42]

Well I want to say I successfully completed a test replacement of a disk. I put some files (few gigs) into two different ZFS datasets under a main ZFS volume and compared the sizes before and after and they matched what they should have been. The only thing I can conclude is that since the partitioning did not match exactly the other drives things just seemed to have gone horribly wrong. That or maybe some other combination. Anyhow here's what I did to replace the disk in case it helps some other person. Note these commands were for my situation and may need to be tweaked depending on the situation. This is for a RAID-Z with 2TB Western Digital Drives using a "Forced" 4,096 sector size and a 2GB ZFS swap partition on each drive. Oh also note in case it wasn't obvious the drives are all advanced format.

First shutdown the server and swap out the disks. Then either SSH in and su to root or use the local console. Whichever works for you. Then in my case I ran:

swapoff -a (Disabled SWAP. It was used when creating the pool initially so can't hurt)
dd if=/dev/zero of=/dev/ada2 bs=1m count=1 (Overwrites first megabyte of disk ada2. I assume to remove any potential residual metadata)
dd if=/dev/zero of=/dev/ada2 bs=1m oseek=`diskinfo ada0 | awk '{print int($3 / (1024*1024)) - 4;}'` (I won't lie. I don't know what this is for. I assume this is to properly align the partitions.)
gpart create -s gpt /dev/ada2 (Create main GPT partition)
gpart add -b 128 -t freebsd-swap -s 4194304 ada2 (Create second GPT partition of 2GB for zfs-swap. The 4194304 works out to "sectors". Haven't determined how to convert that back to a human number like 2GB. Then again didn't look to terribly hard but regardless it's needed)
gpart add -t freebsd-zfs ada2 (Adds the second partition for the actual data)
gpart bootcode -b /boot/pmbr-datadisk /dev/ada2 (Not sure what the boot part of this does)
/sbin/swapon /dev/ada2p1 (Enable swap partition)
/sbin/zpool replace tank 13980078937846798318 ada2p2 (Replace a disk with the identifier "13980078937846798318" in the ZPOOL "tank" with the newly created ada2p2)

After that wait for resilvering to complete. Do a zpool status -v and that will show you the process.

Detach the old disk:
zpool detach tank 13980078937846798318

Reboot and it should show up correctly. Hopefully all that helps and anyone that could enlighten what a few of those commands do that would be appreciated. Again some assumptions are made but I'm sure the general principle is all there. Of course the more experienced ZFS people can chime in and say whether or not that's correct.

 

[ProtoSD]

Zombie,

This looks similar to what I had to do recently. The problem I had was because I originally created my pool in verision 8.0 and the default swap size was 1GB. Now it's 2GB and the replace failed because after the new swap partition was created there wasn't enough room to create the ZFS partition that followed the swap partition.

Glad you figured it out.


I just posted a new tutorial in my blog with a bunch of useful tips for diagnosing/manipulating disks in FreeNAS, it's nice little reference guide.

http://protosd.blogspot.com/2011/12/useful-commands-for-diagnosingmanaging.html

 

[dstwins]

Umm.. interesting..

Well, I'm not sure if this is a bug in Beta3, an oversight, or what...

But I found the source of my problem with replacing the drive in the GUI.

First the Specs:
Originally I had 8GB of RAm in my system, but then I had to dumb it down to 4 (different memory) because the system ran very unstable on the 8GB sticks I was using.. (high voltage requirements). So I dumbed it down to 4GB (4 x 1GB DDR2 sticks) then upgraded to (just recieved today) 8GB (2 x 4GB sticks) plus (2 x 1GB) sticks for a total of 10GB Ram.. this is running 10 Drives in two ZFS pools. and I had two bad drives.. (WD EARS suck)..
Pool 1: Thing1 (6 x 1TB Drives)
Pool 2: Thing2 (4 x 2TB Drives)

I replaced one of the drives in Thing2 with no problem through the gui, it creates the swap partition and everything and does the repartitioning..

However when doing one of the drives from Thing2, I keep getting "MiddlewareError: Disk Replacement Failed, DEvice too small".

What the heck..

So after a lot of work, it seems that the GUI wants to create another SWAP partition (even though none of the 1TB drives have it) and then use the remaining space for the ZFS partition, which puts me 2GB shy of the required amount. Walking through the CLI I can do the same tasks.. (had to manually create the GPT and everything).

My suspicion is when I originally created this system (on 8.0.2), I had 8GB, so it created 4 x 2GB partitions on Thing2 (since that was the first set made), and nothing on Thing1.. but since then I added 2 more GB of RAM, so it wants to expand the swap space (equal to the RAM in the system) and the replacement GUI function is just crazy to want to do the "right thing" and create a new partition map with the additional SWAP space.

Is there a way to override this creation..

And also since I have essentially done a lot of work on the same drive (replaced of course).. I can't do a detach on the drive and then reattach it can i?

Because my drives look like this:

[root@storage] /etc# zpool status -v Thing1
pool: Thing1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h0m, 0.12% done, 11h38m to go
config:

NAME STATE READ WRITE CKSUM
Thing1 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ada2p1 ONLINE 0 0 0
ada3p1 ONLINE 0 0 0
ada1p1 ONLINE 0 0 0
ada0p1 ONLINE 0 0 0
ada7p1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
12304366338284876637 OFFLINE 0 0 0 was /dev/ada6
ada6p1 ONLINE 0 0 0 740M resilvered

And this: 12304366338284876637 is really this: ada6.

So I think I can just issue a zpool detach Thing1 12304366338284876637 at the end of my resilvering process to clean everything up and then reboot to make sure it all comes back normal correct?

 

[William Grzybowski]

Correct, detach should be safe.

 

[Trianian]

Has any tested the latest 8.2 beta to see if drive replacement can be done entirely through the GUI?

This feature is really needed.

 

[DaCeige]

Since this is the only page that still seems to reference how I have to replace disks... I'd like to make a couple updates to the above script... Comments removed for easier copy paste... reference the post above for them.

Here's my variant of the script which I just used to replace a disk with a usb based one. My disk identifiers needed gptid infront of them, and when creating the swap, the sectors created a 16gb partition instead of a 2gb one. Fixed by specifying 2gb instead of 4194304.

Suggest you copy paste this to notepad... find/replace:
gptid/6220719b-ea4e-11e1-955d-0001080115cd -> your failed disk
da1 -> your new disk
tank -> your zfs volume name

swapoff -a
dd if=/dev/zero of=/dev/da1 bs=1m count=1
dd if=/dev/zero of=/dev/da1 bs=1m oseek=`diskinfo da1 | awk '{print int($3 / (1024*1024)) - 4;}'`
gpart create -s gpt /dev/da1
gpart add -b 128 -t freebsd-swap -s 2gb da1
gpart add -t freebsd-zfs da1
gpart bootcode -b /boot/pmbr-datadisk /dev/da1
<Validate your work> gpart show da1
swapon /dev/da1p1
zpool replace tank gptid/6220719b-ea4e-11e1-955d-0001080115cd da1p2
After that wait for resilvering to complete. Do a zpool status -v and that will show you the process.
Detach the old disk:
zpool detach tank gptid/6220719b-ea4e-11e1-955d-0001080115cd

PS. Protosd, we miss your blog. I used it as a reference all the time.

 

[cyberjock]

ProtoSD is no longer a member of the forum. He quit about 2 months ago. :(

Also, you just responded to a thread from April, 2012. I could be mistaken but I believe that things have changed since FreeNAS 8.0.3 is out, and I believe things are changing again with FreeNAS 9.1.

 

[DaCeige]

Things have changed... I'm currently running FreeNAS8.3.1 p2... however I created the zfs dataset back in the 8.0.2 days and to date the gui replace still tells me the replacement disk is too small (even though they are the same size/manafacturer/model number). This is still the only way to date I have found to replace disks...

I'd rebuild it, but I don't have the free disks to make it happen... this is currently a 6x3tb zfs pool.

 

[paleoN]

Things have changed... I'm currently running FreeNAS8.3.1 p2... however I created the zfs dataset back in the 8.0.2 days and to date the gui replace still tells me the replacement disk is too small (even though they are the same size/manafacturer/model number). This is still the only way to date I have found to replace disks...

Fix your swap size.