Critical Alert Email

[djdwosk97]

I just received an email from my server:

Code:

The boot volume state is ONLINE: One or more devices are faulted in response to IO failures.

Does this mean that there's a problem with my boot USB?


The server still appears to be running as I can stream from it, but I can't access the web GUI.
Is there a way to shutdown the server without just forcing a shutdown? Would the 'Graceful shutdown' option from Supermicro's IPMI be my best option?

 

[SweetAndLow]

You can log in via ssh and shutdown or you can probably use the ipmi shutdown method.

You should look into replacing your boot drive though. Do you have them configured in a mirror?

 

[djdwosk97]

You can log in via ssh and shutdown or you can probably use the ipmi shutdown method.

I tried interacting with the system directly over IPMI's KVM console, but I can't do anything I'm stuck with a screen that has a whole bunch of CAM errors -- so I assume I would run into the same issues if I tried to SSH in.

Also, do you think it's an issue with my USB? http://imgur.com/AWcozUa

 

[djdwosk97]

You can log in via ssh and shutdown or you can probably use the ipmi shutdown method.

You should look into replacing your boot drive though. Do you have them configured in a mirror?

I have a single USB flash drive I'm using as a boot drive, but I have the config saved.

Would it be a good idea to use mirror'd USB drives? I have a single USB 2.0 to internal header adapter and the onboard USB 2.0 female connector that's built into my X9SCM-F that I could use to connect two USB flash drives.

 

[Bidule0hm]

Use the IPMI viewer to shutdown the server directly.

Sounds like the boot drive has died but as the system loaded in RAM is ok the server is still up but as soon as you try to access something on the boot drive it fails of course.

I recommend to use a SSD (possibly a cheap used one on ebay) for the boot drive, far better than even 2 USB drives mirrored ;)

 

[djdwosk97]

Use the IPMI viewer to shutdown the server directly.

Sounds like the boot drive has died but as the system loaded in RAM is ok the server is still up but as soon as you try to access something on the boot drive it fails of course.

So the graceful shutdown option in the IPMI viewer?

And would it be worthwhile to mirror two USBs across the internal header (using an internal header to female USB adapter) and the female adapter that's built into my X9SCM-F? I'll probably run out and pick up one/two 8gb Corsair/Sandisk/PNY/Lexar flash drive in a few minutes.

Would a single SSD be better? Also, an SSD would run me like $40 (I would just buy new since used is like $30) whereas I could just get two USBs for <$10.

 

[Bidule0hm]

So the graceful shutdown option in the IPMI viewer?

Yes ;)

And would it be worthwhile to mirror two USBs across the internal header (using an internal header to female USB adapter) and the female adapter that's built into my X9SCM-F? I'll probably run out and pick up one/two 8gb Corsair/Sandisk/PNY/Lexar flash drive in a few minutes.

Yeah, always better than the current single failing USB drive :D

Would a single SSD be better? Also, an SSD would run me like $40 (I would just buy new since used is like $30) whereas I could just get two USBs for <$10.

Yes, I know, it's more expensive; in the short term... because when you'll have replaced the USB drives a few times you'll see it's not that expensive plus you'll avoid the hassle to replace the drives. I made the jump a few weeks ago (for a bit less than 40 € for a brand new 120 GB SanDisk SSD) after I replaced the boot (USB) drive twice in less than two years and I'm more than happy to have done that ;)

 

[djdwosk97]

Yes ;)



Yeah, always better than the current single failing USB drive :D


Yes, I know, it's more expensive; in the short term... because when you'll have replaced the USB drives a few times you'll see it's not that expensive plus you'll avoid the hassle to replace the drives. I made the jump a few weeks ago (for a bit less than 40 € for a brand new 120 GB SanDisk SSD) after I replaced the boot (USB) drive twice in less than two years and I'm more than happy to have done that ;)

Well, by the time I'll have needed to replace the USB drives multiple times SSDs will come down in price again quite a bit (a 120gb SSD will likely be $20-30 by the time these flash drives will fail), so at best it will be cheaper, at worst it will cost the same :p

Alright, I guess I'll run out to staples now and get a couple 8gb Sandisk drives since they seem to be the cheapest right now.

 

[Bidule0hm]

You should be able to already find 32-64 GB SSDs (you don't need 120 GB, it's just that it was not much more expensive to take the 120 GB one... yeah, sometimes you can't resist... :D) for less than $30 if you search well enough ;)

But it's your choice, no problem with that ;)

 

[Mirfster]

I just bought four (4) of these yesterday...have to feed my hardware addiction... :)

http://www.ebay.com/itm/161933540349?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT

 

[djdwosk97]

You should be able to already find 32-64 GB SSDs (you don't need 120 GB, it's just that it was not much more expensive to take the 120 GB one... yeah, sometimes you can't resist... :D) for less than $30 if you search well enough ;)

But it's your choice, no problem with that ;)

Would it be ill-advised to run the system in its current state? I won't be getting the replacement boot drive for a day or two probably.

 

[Mirfster]

At the very least, ensure you have a copy of the configuration.

 

[djdwosk97]

At the very least, ensure you have a copy of the configuration.

Yep, I've got one.

 

[Robert Trevellyan]

Is there a way to shutdown the server without just forcing a shutdown?

With any reasonably modern hardware, you just push the power button (don't hold it in, that's for forced power off) and that initiates a graceful shutdown, assuming the OS supports it (any *nix does).

 

[djdwosk97]

With any reasonably modern hardware, you just push the power button (don't hold it in, that's for forced power off) and that initiates a graceful shutdown, assuming the OS supports it (any *nix does).

Yes ;)

You can log in via ssh and shutdown or you can probably use the ipmi shutdown method.

You should look into replacing your boot drive though. Do you have them configured in a mirror?

I just noticed that my server is still running even after doing the Graceful shutdown. I logged back into IPMI and it says the system is off, but when I went into the room it's in I can clearly tell that it's still on.

Any advice on what I should do to shut down the system? I have it on a UPS, would unplugging the UPS from the wall and letting the server shut itself down be my next best approach?

 

[djdwosk97]

At the very least, ensure you have a copy of the configuration.

I pulled up the IPMI KVM and noticed that the system was waiting on some process and then cancelled the shutdown, but I have no way of terminating the process nor do I have any way of determining what the process is...

Would unplugging my UPS from the wall and letting that trigger the server to shutdown be my next best option?

 

[Mirfster]

Would unplugging my UPS from the wall and letting that trigger the server to shutdown be my next best option?

Not thinking so, since it ignored your command from IPMI. It may just sit there until all power is lost and then go down (not gracefully). So it is not taking any keystrokes in the IPMI KVM session? What about via SSH (Putty)?

 

[Robert Trevellyan]

Would unplugging my UPS from the wall and letting that trigger the server to shutdown be my next best option?

I wouldn't do that.

Can you see the console, either with a display attached or via IPMI? If so, try pushing the power button. If the output indicates the system doing most of the proper shutdown i.e. stopping jails, taking the network down, syncing filesystems, then I would take the chance on a forced shutdown at the end.

 

[djdwosk97]

Not thinking so, since it ignored your command from IPMI. It may just sit there until all power is lost and then go down (not gracefully). So it is not taking any keystrokes in the IPMI KVM session? What about via SSH (Putty)?

I can type into it, but it's not accepting anything I type.

I wouldn't do that.

Can you see the console, either with a display attached or via IPMI? If so, try pushing the power button. If the output indicates the system doing most of the proper shutdown i.e. stopping jails, taking the network down, syncing filesystems, then I would take the chance on a forced shutdown at the end.

I pressed the power button and afterwards I noticed that the shutdown was halted by a process on the system (although I don't if that cancelled shutdown was caused by me pressing 'graceful shutdown' through IPMI a few hours before or from pressing the power button on the case (I don't remember what the time stamp was).

Either way, I just did power off shutdown through IPMI and I'll be reinstalling FreeNAS later so hopefully there are no issues.

 

[Chris Moore]

I had the very same thing happen to me. My USB boot drive in NAS1 failed without warning. There is always a possibility of data loss with a forced shutdown, but I don't think I lost anything.
Were you able to get back up and running?
What did you decide to do for a replacement boot device?
I run a mirrored pair of boot drives in the backup NAS, NAS2, and have not had any problems on it.