Mirrored Boot: One of two USB drives failed - How to replace?

[adamgoldberg]

I have a SuperMicro machine with two Sandisk Ultrafit 32g USB drives as boot devices, mirrored. After installing the latest update, the machine too a very long time to reboot and sent me an email that the boot volume state is degraded:

Code:

The boot volume state is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

If I look at System -> Boot -> Status, I see (first & last columns only):

Code:

freenas-boot ... DEGRADED
-- mirror-0 ... DEGRADED
---- da0p2 .... ONLINE
10046630494804285423 .... UNAVAIL

One of the mirrored boot drives has failed, the one that's still working is at da0p2, and I need to replace the other one.

In an abundance of caution, I've downloaded the config (System->General->Save Config), just in case it all goes to hell. (Is that sufficient to recover from "all goes to hell"?)

Questions:

1. How can I tell which one is the failed one? Shutdown, take one out, and see what happens? Or something else?
2. Should I order identical replacement(s) (they're still available)?
3. How do I replace the failed one? Do I put a new one in (for a total of 2 good, 1 bad) and then do something? Or do I remove the failed one, reboot, and tell FreeNAS to replace the now missing one with the new one? Something else?
4. Is the process here in the documentation somewhere? I can't find it.

 

[JohnDigital]

Best way to figure this out is to remove them one at a time and boot up. If the machine boots, thats probably the good one, if it doesnt, thats probably the bad one. Depending on settings you may need to adjust the bios. Id take note of the gptid and make sure when it does boot, it is in fact the good one, and not the bad one playing tricks on you and coming up for one last breath of air. When you have figured out the one that is bad (be sure), smash it so there is no future confusion. Then put a new USB drive in and tell freenas to replace the missing one with the new one.

 

[adamgoldberg]

Well, I got lucky. Shutdown, removed one USB drive, turned it on and it booted right up. Stuck a replacement USB drive in, clicked on the bad one (under boot status), hit replace, and presto. A few minutes later, it was resilvered and working just fine.

Thanks.

 

[JohnDigital]

You got it!

 

[tfran1990]

this thread is old...but my question relates.
i have 2x 8 gig (there old)sandisk 2.0 flash drives with my os in a mirror. i want to replace them with some new 32g patriot usb 3.0 drives.(same exact usb drives)
is the process as easy as described above? yes i have my config backed up.
i assume that i can take out one of the 8Gig and insert a 32 then resilver then do the same to the last remaining 8Gig stick
i have seen the current issues with 11.3 SMB and i want to avoid using a new install, where you import a saved config.

 

[garm]

Don’t take them out, just plug in the new stick and replace one of the existing ones using the GUI. Then repeate for the other one

 

[tfran1990]

When you replace the usb (NEW), does it delete what was on the previous(OLD)? like a security measure?
My idea was to keep the old 8gig flash drives just in case something happen during the resilvering.

Also I noticed that FreeNAS says my boot size is 7.3G, I am thinking it must have not extended to the full capacity of the new 32G USB. Does FreeNAS detect more space by its self or do I need to extend it with a command?

 

[nojohnny101]

From my understanding, now it will not delete the old USB stick once it puts the new USB stick in service.

Also note that you can't go back to smaller size USB drives without destroying the installation and starting fresh by creating a new mirror from scratch. Not a huge deal as you can save your config, but I believe the "replace" function will not work in the GUI if you try replacing an 16GB USB drive with an 8GB drive.

 

[tfran1990]

ok i used to have 2 8g usb for boot. but i noticed one update took almost 20% of the space. whats going to happen down the road after a few updates are applied and the 32gig flashdrives have close to 8g of data on them? will i run into errors because the GUI thinks the flash drives are only 8gigs, or will it let me go beyond 8gigs of data?

 

[wblock]

When you add a mirror device to an existing boot device, there is an option to "use all space on the drive". If you check that, the partition on the new device is created to fill all the space. The boot mirror is still limited in size by the original device, so it can't grow to the larger size. But if the original boot device is removed, the mirror will grow to use all the space on the second drive. The catch is that now new mirror devices have to be large enough to fit the new size.

An example:

Original boot device is 8G. Boot space is approximately 7G after overhead and drive manufacturer "math" is done.
New 32G device is added to create a mirror, "use all space" is checked, so a (roughly) 30G partition is created.
Original 8G device is removed, mirror size automatically expands to 30G.
New devices added to the mirror must be at least 32G, because the mirror is now that size.

 

[tfran1990]

i still have the 8gig flash drives. is there anyway i can "redo" this process to have the ui report correctly? i used the replace button not the add.

 

[wblock]

The only way to do that is to save the FreeNAS configuration, then install FreeNAS again to one of those 8G USB drives. (Or to several at the same time, to create a mirror.) Then restore the configuration data.

 

[tfran1990]

S*@!, i wanted to avoid re installing FreeNAS at all cost. can i put the old 8g FD back in and roll back to the last config used then do it the right way. (Adding) not replacing.
Could i go buy 2 more of the exact same flash drives and add them to the mirror and select "use all space on the drive"? Then remove the first 2 or will it still only use 8g not the full32.

I don't want to upgrade due to stability reasons. Trying to re import the pool on a new install has me worried about data loss.
I was trying to avoid having any bumps in the road with this process.

Is there any way to force the pool to use all the space per flash drive?

 

[garm]

Just download the version you’re on?

 

[Ericloewe]

download.freenas.org has all the old versions available for download.

 

[tfran1990]

After shutting down my FreeNAS (I did a proper shutdown from GUI) to prepare for some bad weather. Today I go to turn it back on and it boots into the UEFI shell. This has never been a problem for me. I have looked around the forum and tried to see if I could solve the issue myself.

I found CSM support and used variation of CSM disable/enable with UEFI and LAGECY modes as mentioned in another thread.

I am using a HP Proliant ML10 if it matters.

EDIT. So I put the 2x8G older flash drives back in my FreeNAS with the 2x32 (new ones) and everything boots up.... can someone tell me what is going on?
Here are some links to a screenshot of what I am seeing.
https://gyazo.com/d024743a96951d80b4b54be80cd371d8
https://gyazo.com/4ba3ade56626dd83b2c30f2b37b4ba49
https://gyazo.com/d74bbe1cda22067b160b94209dd43794
What is wrong with dev/da1?

Just to recap. I had my initial install of FreeNAS on the 2x8gig flash drives. I wanted to start using 2x32 GIG flash drives for my boot mirror. I went wrong when I chose to "replace" the 8gig FD with the new 32GIG.

 

[tfran1990]

Is it possible to solve this by creating a new boot? I have more of the same flash drives. As of right now my FreeNAS will only boot up with my original 2x8G FD AND 2x32G FD.

What is on the 2X8G flash drives that it needs in order to boot?

I am using 11.1u1. Will an update to 11.1u4 solve this?

 

[wblock]

As of right now my FreeNAS will only boot up with my original 2x8G FD AND 2x32G FD.

Please do not post the same problem to multiple threads. It only makes things more confusing and difficult to solve. The best method is to stick to a single thread with a thread title that clearly describes the problem.

 

[Ben Waywood]

I just opened up a new thread https://www.ixsystems.com/community...s-replacing-and-new-device-wont-attach.77234/ I hope you don't mind my linking it my issue is slightly different to the one mentioned, essentially I have got my boot pool stuck in "replacing" I'm hoping some of the steps you took might have led you to a solution that may help me. Thanks in advance.