FreeNAS won't boot after power outage

[stoooo]

Hello.

I recently bought a lightweight VMware environment consisting of a pair of HP N40L servers coupled to a pair of HP ML110 G7 servers. The ML110s are diskless, and using the N40Ls for their data storage. One N40L has 4x2TB drives and an extra dual port NIC and connects to the ML110 vSphere hosts via iSCSI. The second N40L has 4x250GB drives shared via NFS, and is destined to be the backup repository. The iSCSI box contains all of the VMware datastores for the virtual machines.

This has all been working great for the last six weeks, but it has all gone horribly wrong today...

We had a faulty light in the bathroom, so we had an electrician over to take a look at it. While he was here, we also needed a couple of wall sockets fixed. Unfortunately, the fuse box in our rented accommodation was incorrectly labelled, and so my office power was brought down hard, and all of my servers with it. Yes, yes, yes, UPS, blah blah, yada yada. If the fuses had been properly labelled, things would have been fine.

Anyway, the upshot is that one of my physical servers has suffered a terminal failure. I have put the drives into another box so I can get to the data. This server was destined to be moved to the virtual environment in the next couple of weeks, so this is just bringing forward the plan. However, the VMware datastore N40L now won't boot up. The NFS one has come back up just fine, but I really need my VMware datastores back online so I can get on with the migration.

So, my FreeNAS problem is this: When the N40L boots, it gets as far as the boot menu (F1, F2 etc), appears to start the bootstrap loader, but hangs at the message

Loading /boot/defaults/loader.conf
| <--This normally spins while things load up

This is booting from a 2GB USB memory stick, so I am pretty confident that my data is still OK and sitting on the drives. But how do I go about fixing FreeNAS so it boots again ? I am toying with just trying to install it again but telling it to keep the config, which I seem to remember was an option. I would really prefer not to have to build the config from scratch again, and rely on importing the volume.

Oh, I did try choosing F2 from the boot menu, but it just sat and filled the screen with # signs for about 30mins before I got bored and power cycled it.

Thanks in advance,
Stuart.

 

[stoooo]

Doh ! It might help to mention the version number I'm running - FreeNAS-8.0.4-RELEASE-p3-x64

 

[paleoN]

Yes, yes, yes, UPS, blah blah, yada yada. If the fuses had been properly labelled, things would have been fine.

Oh, please. Things simply could have gone wrong when you had a power outage instead. You chose to run without one so don't wine about it.

This is booting from a 2GB USB memory stick, so I am pretty confident that my data is still OK and sitting on the drives

FreeNAS-8.0.4-RELEASE-p3-x64 requires a 4 GB stick unless you have an elusive 2GB stick that's actually a full 2GB. The data is fine as long as the drives are fine.

You should have a backup of your config. Since you don't, you can stick it into the NFS N40L, or any FreeBSD system, and mount the 4th partition, I believe, and copy freenas-v1.db to a safe place. Then a fresh install to a larger memory stick and import your old database.

If that doesn't work you can export the pool on the NFS N40L, swap the disks with iSCSI N40L and import the iSCSI pool.

 

[stoooo]

Oh, please. Things simply could have gone wrong when you had a power outage instead. You chose to run without one so don't wine about it.

I'm not whining about it, just trying to fend off an avenue of discussion that would divert from the main topic. Once we get it working again, I shall don my asbestos undies and let you all grill me to your hearts content :D

FreeNAS-8.0.4-RELEASE-p3-x64 requires a 4 GB stick unless you have an elusive 2GB stick that's actually a full 2GB. The data is fine as long as the drives are fine.

Interesting. I'm sure the docs said 2GB was fine, and it has been running great until the plug was pulled. The NFS box is also running from an identical 2GB stick. The servers were bought with drives and a USB stick specifically marketed as a FreeNAS bundle.

I'm not really a FreeBSD user. Would I be able to do your suggestions on one of my SuSE boxes ?

Thanks,
Stuart.

 

[paleoN]

I'm not whining about it, just trying to fend off an avenue of discussion that would divert from the main topic.

You're one who brought it up. You are whining period. If you choose to run without an UPS, fine except the consequences. If you hadn't said anything about it then I wouldn't have mentioned it again.

Interesting. I'm sure the docs said 2GB was fine, and it has been running great until the plug was pulled. The NFS box is also running from an identical 2GB stick. The servers were bought with drives and a USB stick specifically marketed as a FreeNAS bundle.

The image sized increased in later builds. It will still work fine as long as it's a full 2GB. I haven't seen one that is though. The one stick may be flaking out on you now anyway.

I'm not really a FreeBSD user. Would I be able to do your suggestions on one of my SuSE boxes ?

Sure, as long as your SuSE box supports mounting FreeBSD UFS slices. Otherwise trying it from a FreeBSD based system will be easier.

 

[survive]

Hi stoooo,

Basically any time there is some funk involving booting & a 2GB usb key the first thing "we" will tell you to do is to install to a 4GB key & see if that sorts things out. I know the docs say 2GB key, but the install image is 2000MB & every single 2GB key I've seen usually comes out at ~1.9GB.

Here's what I would do....go get a couple of 4GB keys, and pick up a 3-pack of CDRW's while you are at the office supply\computer store. Grab the FreeNAS .iso of your choice & burn it to the CDRW then do the install on any PC you have around that can boot the CD. When it prompts you to reboot after the image is written to the key power off the computer & move the new 4GB key over (be sure to save your config!). Once the system is booted just upload your config.

To get to the config on the key that won't boot you can download a FreeBSD based live CD...something like PCBSD's live DVD or the mfsbsd boot disk. All you need to do is get the freenas-v1.db file from /data on the key. It might really just be easier to configure the box from scratch.

-Will

 

[stoooo]

The image sized increased in later builds. It will still work fine as long as it's a full 2GB. I haven't seen one that is though. The one stick may be flaking out on you now anyway.

Sure, as long as your SuSE box supports mounting FreeBSD UFS slices. Otherwise trying it from a FreeBSD based system will be easier.

I've just ordered up a couple of bigger flash drives to upgrade to. At least we can eliminate that as a potential issue.

I guess the only way to know about the UFS slices is to plug it in and try it.

Thanks,
Stuart.

 

[cyberjock]

I just want to point out to anyone who ever reads this thread to look at the PITA the OP is going through because he didn't use an UPS.

Just as sad of a story as the people that don't do backups but know they should.

 

[stoooo]

Oh, yes, I do know better. Probably better than most. We moved from Cyprus to the UK last year after this happened to the island's main power station...

http://robotpig.net/extras/cyprus_zygi/

The trouble is, a suitable UPS that will cover all of my equipment would probably cost as much as the equipment it covers. And there just isn't enough money around for that right now :(

I'm trying to be pragmatic. As much of a PITA as this is, it is an opportunity for me to learn more about FreeNAS, and to start to develop a DR plan. I already have the config for the NFS box backed up ;)

 

[cyberjock]

The best DR plan I can think of is "have backups with an UPS on the backup server".

UPSes are known for protecting from power surges very well. I had lightning strike the transformer on the pole behind my house. I had everything unplugged except my server(oops), my router, an 8 port switch and my cable modem. All toasted except my server. The router's insides were interesting. Smelled like burned electronics and one of the chips had physically exploded. Switch was the same, one of the chips had burst. I was lucky in that I somehow accidentally unplugged my server's LAN port. I always unplugged all of my computers from the wall and the LAN when storms come.

UPSes also protect from data corruption due to a loss of power(the OP has a keen understanding of this unfortunately). Also, since my switch burst like it did and the chip that burst was for the port my router was connected to I've bought one of these to connect my router to my switch. I don't connect anything directly to my router anymore, and my switch(and everything that's plugged into it) should hopefully be protected in the future by this little surge protector. I've heard of power surges going through network cables and destroying entire LANs, but it would suck particularly bad to have a power surge through the network cable burn out my server. I'm not taking any chances after being about 50 feet away when lightning struck. It's the second loudest noise I've ever heard. The first was in the military and if I told you I'd have to kill you...

Of course, your mileage may vary for storms. I haven't seen thunderstorms like what is in my area before in my life. Hopefully when I move I'll be somewhere that has less of them.

 

[NASA]

It seems to me that the surge while tripping the breaker did the damage--not the power outage itself. What else is on the same circuit as your computer equipment? Many surge protectors are worthless. PolyPhaser makes the absolute best network/telephone line/cable line surge protectors, but they are very expensive. A close lightning strike will induce surge current in all of your network wiring--the longer the wire the greater the risk. It will also induce surge current into your house ground system. If the lightning actually strikes your telephone or cable line and it is attached to a modem (attached to your router) the best surge protector in the world probably won't protect your equipment--it is best to unplug the telephone or cable line from your modem during a bad storm.

 

[stoooo]

First of all, thanks to everyone who offered useful advice. Much of it was good, and I was able to have some measure of success.

I was never able to get the USB flash drives mounted under SuSE. It looks like I should have been able to, and it is something I may continue to pursue out of curiosity, but I gave up on that, opting instead to take another suggested approach. I downloaded and created a PCBSD 9 Live DVD. It was troublesome on the first two PCs I tried it on, but the third worked like a champ. While that was booting up (since the PC had 4GB of memory it chose to copy itself entirely to RAM, which took some time) I installed FreeNAS on the two new USB thumb drives. Then when the PCBSD box came up, it was a trivial matter to copy the config DB file from the old stick to the new one.

The good news is that the NFS box came straight back up using this method. Yay ! The bad news is that the iSCSI config was apparently corrupt, so that one didn't come as hoped. Booooo ! So I have spent the morning rebuilding the iSCSI configuration, which seems to have worked out well, and my ESXi hosts are again able to access their respective datastores.

So, important lessons learned:
1. Backup the FreeNAS configs somewhere safe, i.e. not *on* the data stores on the FreeNAS box. (I didn't know this could be done, as it isn't mentioned anywhere (obvious) in the docs.)
2. Buy a UPS. Dear Santa...

I've been doing backups religiously every week for years, so the data itself was always recoverable. However, I really need to get up to speed on backing up in a VMware environment as I move from a physical setup to a virtual one. Veeam looks the most promising.

Oh, and those of you worrying about storms have valid concerns for those areas that get violent electrical storms. I know we had some epic storms when I lived in MN. However, here in the UK they are few and far between. So I make do with a couple of Belkin surge protectors.

Cheers,
Stuart.

P.S. The file server that failed is an old HP/Compaq ML370 G3 that was already having power issues. The PSUs seem to be fine, but I think the mobo finally gave up. Some of the LEDs still light up, which leads me to believe the 5v line was still working, but I think the 12v line broke down so it wouldn't boot up.

 

[paleoN]

Glad you got it straightened out.

You can save your config to your zpool. IMO, it's a bit easier if you have it saved to your computer though.

 

[noob598]

I;m glad I read this thread. If freenas is this bad at error recovery I think i'll stick with windows server.

 

[stoooo]

I;m glad I read this thread. If freenas is this bad at error recovery I think i'll stick with windows server.

I think that's a bit uncharitable. I was relying on a vendor supplied solution which included a 2GB memory stick. Reading a bit more on here it seems that a 4GB or larger is a more reliable solution. Also, my data wasn't impacted, just the OS. I've been in IT for many, many years now, and I've seen Windows machines do some crazy things to data for no apparent reason, so that's not exactly a totally reliable solution either.

While not related to FreeNAS specifically, I think the most impressive data recovery I've ever seen was in an old IBM server I had that was running SuSE Linux. It had four Pentium Pro CPUs in it (that's how far back we're going) and 12 internal drives. It also had an external drive array attached with a further 9 drives in it. These were all pooled together into a giant (for its day) array. While away for the weekend, we had something happen that caused the external array to switch off. My only clue that anything was wrong was when I connected remotely and some of the files seemed unable to open. They still showed up in the directory list, and I could still open many other files, and save new ones. Of course, blind panic set in when I got home and found out what had happened. So I ran everything down, then brought it all back up gracefully. There was, of course, an fsck, but when that completed I had full access to everything again. Amazing. I really thought the array would be toast, but the system just seemed to work around it as best it could.

 

[survive]

Hi guys,

noob598, it's not that FreeNAS is bad at error recovery, it's just different and if you don't know your way around a FreeBSD\Linux\UNIX environment it can be daunting at best. OP had a couple of problems that compounded his problems....the biggest was probably the USB key size that caused problems booting after the power fault was corrected.

On with my impressive power outage story...I run a SAN at work that pretty much everything important runs off of. We had a particularly bad night where there was a power event that knocked out power to the whole campus compounded by a double controller crash in the SAN. The whole SAN was down for over 9 hours, however I had 1 HP blade chassis that stayed up that included 2 of my Domain Controllers. Since most all of my infrastructure was down as well there were was next to no traffic to the blade system so even though they were running without any disks attached at all they all stayed up, presumably caching any writes they wanted to do. Got the SAN back up and they continued on like nothing had happened.

-Will

 

[BobCochran]

So, important lessons learned:
1. Backup the FreeNAS configs somewhere safe, i.e. not *on* the data stores on the FreeNAS box. (I didn't know this could be done, as it isn't mentioned anywhere (obvious) in the docs.)
2. Buy a UPS. Dear Santa...

I've been doing backups religiously every week for years, so the data itself was always recoverable. However, I really need to get up to speed on backing up in a VMware environment as I move from a physical setup to a virtual one. Veeam looks the most promising.

I know we had some epic storms when I lived in MN. However, here in the UK they are few and far between. So I make do with a couple of Belkin surge protectors.

Cheers,
Stuart.

This is really interesting reading. You did a remarkable recovery job and I agree you had some great advice. I'm going to make a note of this thread for future use.

I fully sympathize with your cost concerns about a UPS. A useful UPS solution, inside the UK, is going to be extremely expensive after VAT. Everything seems to cost vastly more over there. I know electrical construction codes are different over there as well, line voltages are a lot higher, and the actual electric plugs used are of different construction and need to handle higher voltages than standard American plugs.

I wonder if you can find an electrician with steady ongoing data center experience.

Does your site have really good earth grounds?

I agree that Minnesota has epic storms. And that the UK itself does not seem to, at least not in the southern parts. There is a lot of rain but I've never noticed much lightning.

Bob

 

[JaimieV]

I've no connection except as a happy customer: upstrader on eBay.co.uk has been a splendid source of surprisingly inexpensive previously-loved UPS hardware for me. http://stores.ebay.co.uk/upstrader

(sorry if this looks like spam, it's a genuine suggestion!)

 

[CRYPTR]

I know this is an old post but there still seems to be confusion so i will enlighten.
my server crashed and when freenas went to boot it stayed on the f1/f2 screen.

****HOW TO FIX****
1. remove your usb stick from the device.
2 place usb stick into a laptop or desktop and boot off usb.
3 wait for it to build....
4 once complete, shut down using option 11
5 remove usb and place into origonal device
6 boot from the usb.



hope this clears up the confusion for newbs.