How long to rebuild 16GB USB boot drive Mirrored with ZFS

[NorCalTechSupport]

I have a pair of 16GB USB thumb drives Mirrored with ZFS as my boot drive. My SAN is a FreeNAS-9.3-STABLE-201506292130
Everything has been working well for two years and all of a sudden I start getting messages that " The boot volume state is ONLINE: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state."

Over the last 10 days this has happened 3 times so last night I pulled the USB thumb drive I suspect as being bad and replaced it with a brand new straight out of the cellophane wrapper identical USB thumb drive (I put aside a 3rd thumb drive when I built my SAN). I have never had a problem with my FreeNAS SAN or any drives so I am not sure how long it will take to replicate the data onto the new drive? Does anyone have an idea how long it should take to get the 2nd drive to become a full fledged member of the mirrored pair?

Is simply replacing one USB drive with another all I need to do? If not what is the procedure to get everything working again?

Here is an email the SAN sent me this morning:

Checking status of zfs pools:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Athene 36.2T 6.35T 29.9T - 21% 17% 1.00x ONLINE /mnt
Mercury 1.81T 311G 1.51T - 18% 16% 1.00x ONLINE /mnt
freenas-boot 15.5G 1.12G 14.4G - - 7% 1.00x DEGRADED -
pool: freenas-boot
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 450K in 0h0m with 0 errors on Mon May 1 02:43:05 2017
config:
NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
17767100420177258852 REMOVED 0 0 0 was /dev/gptid/87250026-0ae4-11e5-8fca-0cc47a18b26c
gptid/88e0a3ba-0ae4-11e5-8fca-0cc47a18b26c ONLINE 0 0 0
errors: No known data errors
-- End of daily output --

Thanks in advance all

 

[Vito Reiter]

My boot drive resilvers usually take less than a minute, given they are very small like 500M. Just click the replace button, select the new USB (Make sure it's the one you intend to use) and allow it to resilver. There will be a status % at the top of System > Boot > Status but it shouldn't take more than a few minutes. Flash drives fail all the time, that's why we mirror them :)

Edit: This stuff is very easy to do through the GUI, if you can reach that I would recommend using it over the CLI in this situation unless you're a die-hard terminal fan.

 

[NorCalTechSupport]

Thanks Vito Reiter. From the GUI pic included can you tell me which line to click on? I presume the bottom line that says Removed and then click on "Replace"?

 

[Vito Reiter]

Make sure your new drive is in and the old one is out, click the bottom one that's labeled 'Removed', and at the bottom left you'll see a blue 'Replace' button. Once you click that it will ask you to select the replacement drive (chose the new drive, should be the only listed or easy to tell it's a USB due to size) and confirm it, resilver will start and finish within a few minutes.

 

[NorCalTechSupport]

Thanks Vito Reiter. My SAN is now a happy camper as am I.

 

[Vito Reiter]

Thanks Vito Reiter. My SAN is now a happy camper as am I. :)

That's great to hear man, ZFS is awesome like that and replacing drives in your main storage array pretty much works the same way just in a different part of the GUI. This PDF HERE is what really got me going when I was new to FreeNAS I recommend it to everyone.

 

[ATechGuy]

I had a similar situation. I thought the two drives were FULLY mirrored, but alas, it appears only the 'p2' partition is mirrored, and but bootblock isn't. Is that what I should expect?

In my case, I had two drives, and the "second one" (not the one booting) died so I replaced it and reslivered it. Things were good until a few weeks ago when we had a nasty power hit. As a result, my bootable thumbdrive completely (well, for this forum, lets say completely) died. Trying to boot from the mirror fails as there is no boot block.

It turned out, I had an OLD thumbdrive that was essentially a copy of the bootable one (perhaps I had made it from an earlier freenas build, and so I tried that one. It booted, but many of the settings were understandably incorrect.

So, (this gets good). Let's just say, da0 (lower slot, and the one BIOS was set to boot first) was the bootable thumbdrive, and da1 (upper slot) was the mirror.

I then discovered that this older, but bootable thumbdrive had been built where it expected da1 to boot from the bootblock, not da0. So, by putting this old, bootable drive into my da0 slot, it was happy to boot my da1 mirror and my freenas came back alive. Before wasting any time, I rebuilt the mirror so the old, but bootable drive p2 was now mirrored with the correct p2 partition of the original mirror.

But I really don't trust either drive, and would like to make backups of both. This system doesn't have any spare USB slots, so I'm limited there, but I figure if I can just put a boot block on the working original thumbrive, I can then know how to build a bootblock on a new drive, and then replace the mirror members one at a time. I just don't know how to mirror, or build the bootblock, other than using some "ISOtoUSB" tool and specifying grub for the boot loader, but then I'm not sure about the details.

I am a techie, but not super great with unix, but could boot a Linux or FreeBSD live CD if needed, and then, would DD work to copy the bootblock?

Thanks for your help.