How to recover from UFS drives degraded error

[anika200]

Situation Is that the power got pulled on the Freenas box and now I have all four drives in "degraded" state and the red glowing alert indicator in the webgui.

What are the standard set of instructions to recover from this problem?
I have no raid, no zfs, plain old disks (4 in total) with a jail on one of them.

I already dropped to single user mode at the console and ran fsch_ufs -yf at the command prompt which is how I ended up with a usable webgui again but now the system refuses to use the drives.
I have already ran smartctl from webgui shell and all the drives pass and seem to be alive and accessible.

What should be the next steps to get back to normal?

 

[cyberjock]

I'm not a UFS master so I can't help you with your exact issue. I replied because running a file server without an UPS installed and setup to shutdown the server automatically is a recipe for disaster.. hopefully your disaster won't be too painful. Someone else just spent over a week trying to get his files back after a power outage without an UPS. He got his most important files, but many of them were unrecoverable.

 

[anika200]

I'm not a UFS master so I can't help you with your exact issue. I replied because running a file server without an UPS installed and setup to shutdown the server automatically is a recipe for disaster.. hopefully your disaster won't be too painful. Someone else just spent over a week trying to get his files back after a power outage without an UPS. He got his most important files, but many of them were unrecoverable.

Certainly good advise and if I set this up at the office it will have a proper UPS. I was just putting it though some testing to see how robust it was and if it was a viable solution for the office network storage device and backup solution.

I do not want to get off topic here because I really want to know if there is documented procedure for this kind of recovery. I found some info on google from another user but I figured I would check here first, seemed to make sense to me. The advise from my google find was basically to detach and re-import the drives but I wanted to heahr from at least one other source before trying it.

 

[anika200]

Incredible, the machine is working fine now. :)
After sitting powered off for a day waiting for some advise from someone I fired it up to try more repair and it ran its own fsck routine corrected a bunch of errors and came back to life.
I am wondering now what the heck happened, maybe something was overheated???

 

[cyberjock]

No, it was probably something with the software. Maybe fsck won't run automatically on its own the first time or something. The red alert light is somewhat a liar. If I pull a drive from my zpool I'll get the yellow warning light after a while because something is wrong. If I shutdown the server and then add the drive back in and boot up the light will be green. But for full redundancy I should be doing a scrub because the drives will be out of sync.

I'm not a UFS god, but I'd guess that there was a problem justifying the red light(temps aren't monitored in FreeNAS itself but via SMART if you have emails setup) and by doing a reboot you corrected the cause of the red light. Normally I'd recommend you do a fsck, but you said it already did one and fixed a bunch of stuff. I've only seen the yellow or red lights if some of the drives weren't detected during bootup or zpools couldn't be mounted.