Unable to access GUI after a disk failed

Status
Not open for further replies.

Goatcrasher

Cadet
Joined
Dec 28, 2016
Messages
5
Hello all!

Hope we had a nice Christmas?

Mine was going great until i got a email from my box stating a disk had failed! Doh! These things happen and i have a raidz2 so no big deal.

However the issue came when i tried to access the web gui (192.168.20.245 Static) and i get page cannot be displayed. Trying to Telnet to port 80 does not work either. Its like the web demon has crashed on the box?

Unfortunately this kit is in a datacentre about 150 miles away so i don't have any easy access to the console and i never got round to setting up SSH...

Does anyone have any idea how i can get the gui back remotely? The iSCSI connections are still working fine.

Thanks in Advance

Matt

My Setup
Dell R710 with 1 4 Core Xeon (cant remember the exact model!)
36gb RAM
Boot is for 2x 16gb Kingston Flash Drives in a Mirror
2x 250gb Samsung SSD (I was using for ZIL\L2ARC but found it was actually slower than without!, so now redundant)
6x 500gb 2.5in NL-SAS drives connected to H700(Each disk is setup as a Raid0 with Write back turned off (I know not the bets but i didn't really have a choice with the budget i had))
6x 2tb 3.5in Sata in a MD1000 DAS. Connected by 2 3gbps SAS connections to a PERC 6 SAS
DRAC Enterprise (but this is not plugged in!)
All Connections are via iSCSI to 2 Windows 2016 Hosts
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
Have you tried turning it off and on again?

Plug in a monitor to see what's happening.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Unfortunately this kit is in a datacentre about 150 miles away so i don't have any easy access to the console and i never got round to setting up SSH...
I see a road trip in your future. Since your machine is not locally accessible, I would have the boot device be something else besides USB, they are just not reliable enough even in a mirror IMHO
 

pirateghost

Unintelligible Geek
Joined
Feb 29, 2012
Messages
4,219
I see a road trip in your future. Since your machine is not locally accessible, I would have the boot device be something else besides USB, they are just not reliable enough even in a mirror IMHO

I highly doubt that it was a failed boot device.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Road trip time. You should also figure out how to get remote console access in the future... cable up that DRAC, but please do NOT directly connect it to the Interwebs.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
I highly doubt that it was a failed boot device.
I did not mean to suggest it was. The comment was just more or less an aside to the fact that the server was so far away...
 

pirateghost

Unintelligible Geek
Joined
Feb 29, 2012
Messages
4,219
I'm really surprised that nobody has mentioned the horrible choice in drive controller. Along with the raid0 disks...
 

Goatcrasher

Cadet
Joined
Dec 28, 2016
Messages
5
Thanks all. @BigDave: Thanks - What would you say is a better device? Dedicating 2 of the Internal to be Boot? I'm sure I read in the manual that that’s not advised?


@m0nkey_ I think this may be the ultimately "fix" for this issue, I don’t have the luxury of being near it to be able to reboot it, and the DRAC is unplugged :(.


As @pirateghost says, I don’t think this is boot drive failure, The OS must be operational as the iSCSI targets are still functional. My question is a long shot. I don’t think there is a answer apart from getting on the console. But thought I would put it out there anyway, someone else must have been in my position before, right?
 

Goatcrasher

Cadet
Joined
Dec 28, 2016
Messages
5
I'm really surprised that nobody has mentioned the horrible choice in drive controller. Along with the raid0 disks...
i know, this is really bad and i want to replace it soon. The ironic thing is, the failed disk is not on that controller! Its in the DAS!
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
What would you say is a better device? Dedicating 2 of the Internal to be Boot? I'm sure I read in the manual that that’s not advised?
The manual is telling you NOT to use data drives for the OS, primarily due to the capacity loss. The installer partitions the
drive you choose to a size < 4GiB and the rest of the space is not used and inaccessible for storage use.

Myself and many others use a small capacity SSD to install the OS on to get away from the unreliability of a cheap USB
flash drives (see my signature for the SSD make and model I prefer). When using an SSD, I don't think a mirrored
configuration is needed, but my server is not located 150miles away either;)
 

Goatcrasher

Cadet
Joined
Dec 28, 2016
Messages
5
Road trip time. You should also figure out how to get remote console access in the future... cable up that DRAC, but please do NOT directly connect it to the Interwebs.

Reading you loud and clear. I'm going to have to visit to replace the drive anyway, I stupidy thought "FreeNAS is solid as a rock, i dont need the DRAC for now". DOH!
 

Goatcrasher

Cadet
Joined
Dec 28, 2016
Messages
5
The manual is telling you NOT to use data drives for the OS, primarily due to the capacity loss. The installer partitions the
drive you choose to a size < 4GiB and the rest of the space is not used and inaccessible for storage use.

Myself and many others use a small capacity SSD to install the OS on to get away from the unreliability of a cheap USB
flash drives (see my signature for the SSD make and model I prefer). When using an SSD, I don't think a mirrored
configuration is needed, but my server is not located 150miles away either;)

Ah, I see :) ill look at that for future! Looks like ill be taking a road trip anyway.

I'd love the Kit to be right next to me but a Tier4 Datacentre is just too tempting! ;)
 

Slavik

Dabbler
Joined
Jun 6, 2016
Messages
39
I reported similar issue earlier:
https://forums.freenas.org/index.php?threads/freenas-stops-responding-after-running-badblock.44451/

I think this is clearly bad design on FreeNAS part: if you have disk issue in one of data volume (not boot) - the whole system becomes unstable.
I think the system should continue to work, regardless of any issue on DATA volumes, as long, as BOOT volume is ok.

I was told:
>>Freenas store most of its configuration files on your zfs pool. So when ... your drive started to fail everything on that drive was ruined. That is what slowed down your system.

This kind of responses, found often on this forum, makes me believe, that FreeNAS is `fragile` system - every time someone reported issue, they are told, that they did this wrong and that wrong. How about just saying, that FreeNAS is the system, which can't reliably handle any kind of equipment failures, even not-critical (failure of disk in RAID is not critical), and only works on "ideal" systems.
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
I reported similar issue earlier:
This kind of responses, found often on this forum, makes me believe, that FreeNAS is `fragile` system - every time someone reported issue, they are told, that they did this wrong and that wrong. How about just saying, that FreeNAS is the system, which can't reliably handle any kind of equipment failures, even not-critical (failure of disk in RAID is not critical), and only works on "ideal" systems.
I wouldn't say it's any more fragile than any other OS. I've seen the same on Windows and Linux systems when a hardware failure occurs.

However, there are things that can be done to preempt any such failure, such as running regular SMART tests, scrubs, etc.
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211
Myself and many others use a small capacity SSD to install the OS on to get away from the unreliability of a cheap USB
flash drives (see my signature for the SSD make and model I prefer).
Holy cow! Why is that thing so expensive? I'm seeing $340 and up on Amazon, not available on Newegg.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
I reported similar issue earlier:
https://forums.freenas.org/index.php?threads/freenas-stops-responding-after-running-badblock.44451/

I think this is clearly bad design on FreeNAS part: if you have disk issue in one of data volume (not boot) - the whole system becomes unstable.
I think the system should continue to work, regardless of any issue on DATA volumes, as long, as BOOT volume is ok.
This isn't a FreeNAS issue, this is a FreeBSD issue... and largely, simply a factor of PC design in general. The system requests some data from the drive... it replies with "hold on, working on it." Usually the data is returned quickly (instantly in our eyes, a short eternity for a modern CPU) but sometimes it isn't, as the drive churns on a potentially bad sector and tries to recover the data. After enough retries, it finally gives up... but the system waits for awhile.

I've had drives fail in my system... performance on the impacted array was horrid (as it kept waiting for the drive to give up) but the system was quite stable otherwise.

I was told:
>>Freenas store most of its configuration files on your zfs pool. So when ... your drive started to fail everything on that drive was ruined. That is what slowed down your system.

This kind of responses, found often on this forum, makes me believe, that FreeNAS is `fragile` system - every time someone reported issue, they are told, that they did this wrong and that wrong. How about just saying, that FreeNAS is the system, which can't reliably handle any kind of equipment failures, even not-critical (failure of disk in RAID is not critical), and only works on "ideal" systems.
FreeNAS stores its configuration data on the boot pool... specifically, in /data/freenas-v1.db. If the boot pool is what's failing, then yes, you'll see issues. This is why many people don't recommend USB flash drives as boot devices... they have a high failure rate and things get wonky when they do.

FreeNAS isn't any more fragile than any other enterprise-class storage array. Keep in mind, there are people that work full time doing nothing but managing storage. It's not an easy thing, especially when you get into advanced configurations. FreeNAS does expose a lot of the inner workings to the user (and assumes the user is smart enough to know what they're doing with that info, or to ask for help)... often in cases where a cheap hardware controller would have thrown up its hands and said "sorry, your array is trash, recover from backup." FreeNAS also lets you make a lot more decisions about hardware - many of which are often bad.

Let's compare a Mac to a typical Windows/Linux/etc. PC. The Mac is like the hardware controller... the manufacturer controls the hardware, you're expected to operate within a certain set of boundaries, and things go pretty smoothly. The Windows PC is different - you can buy your own hardware, configure it your way... and you're free to make bad decisions as well. If you expect huge performance and buy a bunch of second-rate hardware, you can expect second rate performance and issues. If you want seamless, high performance operation, you need to buy hardware that makes that possible.

FreeNAS is no different... if you buy dedicated server-class hardware, higher-tier drives, etc., you'll be impressed. If you buy crap hardware, or try to virtualize (something that is quite doable, but even the experts will tell you it adds a whole new level of potential problems to the mix), then you'll get crap performance and issues.

Personally, I've been running it for about a year on enterprise-class hardware and I've experienced very few issues. In fact, most of the issues came from trying to use hand-me-down drives pulled out of a SAN, that had already been run hard for quite a long time and failed somewhat frequently... I finally said "screw it" and bought new drives, and things are much better.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Holy cow! Why is that thing so expensive? I'm seeing $340 and up on Amazon, not available on Newegg.
Suggest you try used on ebay, if you want buying tips PM me;)
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Suggest you try used on ebay, if you want buying tips PM me;)
There's no kill like overkill, but... enterprise-grade SSDs (especially those that have been cancelled) are serious overkill for boot devices. I picked up a handful of Intel 320 40GB drives off eBay for $20/ea. and use them in pairs in my FreeNAS and my 3 VM nodes. They work great, and the price is right.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
There's no kill like overkill, but... enterprise-grade SSDs (especially those that have been cancelled) are serious overkill for boot devices. I picked up a handful of Intel 320 40GB drives off eBay for $20/ea. and use them in pairs in my FreeNAS and my 3 VM nodes. They work great, and the price is right.
Back when I bought mine, I paid $18-$25, they go for about $35-$45 now because there's more demand
or less supply dunno which. The drives have decent reliabilty, check this out...
Reliability
— Bit Error Rate (BER): 1 sector per 10(to the 15th power) bits read
— Mean Time Between Failures (MTBF) 2,000,000 Hours
— Write Endurance: 1 petabyte of random writes (32 GB)
2 petabyte of random writes (64 GB)
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
That kind of reliability is great... your FreeNAS boot pool will still be happily churning along when the rest of us have moved on to isolinear chips :D
 
Status
Not open for further replies.
Top