HDD Fail Notification...?

[SilverJS]

Guys,

This might seem n00b-ish (that's cuz I am one! =)), but I haven't really been able to track down a solid answer to : what happens when a drive fails in FreeNAS? Will I get an e-mail notification? (Now that I've learned to configure the e-mail notifications correctly...=)))

Also, is there a way in the webGUI to run actual commands, like zpool status or some such? (This is for when I'm away from home and can't do it on the server itself.)

Thanks!

 

[ProtoSD]

Well that is what is supposed to happen, but I don't think they've completely added support for it yet. FreeNAS 8.01 should have it for sure. Supposedly in FreeNAS 8.0 if you had your email setup and add '-m root' as "Extra SMART options" for EACH disk under Storage -> View Disks -> Edit (each disk) (without the ' marks), it should email you.... BUT

I had that setup and the test email worked correctly, but one of my drives started failing and I got NO email.... There was an error in the log that some error prevented sending the message, but no real details. So definitely be cautious until you know for sure more than the "Test Email" button works...

 

[SilverJS]

Awesome. Just finished configuring my disks as per above, will test in a few minutes. That being said - again, from the web GUI (assuming I'm away from home), how do you view that log you talked about? (Heck - what's the command to view it in shell when at the server??)

 

[ProtoSD]

The error message about the failed email was in /var/log/messages, so either 'more /var/log/messages' or 'tail /var/log/messages' or whatever your favorite way to look at files is. ;-)

That was in version 8.0, so if you're using 8.01 beta it could be different, or it may actually work without any errors!

As for viewing the error from the GUI while you're away, you could enable console messages at the bottom of the web GUI, but if the message scrolls off the bottom, you'll miss it. I'm not sure, but I think 8.1 might let you do shell commands from the GUI when that comes out.

 

[SilverJS]

Well, this isn't too good. Pulled the SATA data cable from one of my drives (have a RAIDZ2 with 6 3TB drives, + 1 hotspare). The server had a message onscreen about losing a device. No e-mail. (I put '-m root', without apostrophes, in all of my SMART options like you said - and, test e-mails work fine). zpool status showed all drives as being online, however. Hmm...when I reconnected, no message about the device being re-acquired or anything. Then, pulled that same drive, and another one. Two 'lost device' messages, but all drives still showing as online with zpool status. Weird...?

Then, when I reconnected, I got an error about it being unable to reconnect. I forget exactly what....and, when I tried transfering a file from my zpool to my Windows computer, everything froze (including the Windows box!). It now shows me an error (in the bottom of the GUI) as zpool I/O error 28, and a vdev I/O failure.

This was just a test with garbage files, but that's not too reassuring...?

Tried rebooting my server from the webGUI, it just kind of hung...I had to go in there and manually press the reset button.

 

[ProtoSD]

What version of FreeNAS are you using? They may still be working on it, I'm pretty certain it's *supposed to work* in 8.01. I'm not sure, but I think there's a background / cron job that checks the disks for errors and I don't know how often it runs, so it may take say 15 minutes for that to run and *notice* that there's an error. It sounds like you put the option correct '-m root' without the quotes. You can also use another account other than root if you have an external email address setup for it. My guess is they haven't finished working on it yet, or the script didn't have time to detect the error.

EDIT: Just to confirm, you also went to Account -> Users -> View All Users and changed the email address for root? This could be the same email address you setup on the "Test Email" page. Also, like I mentioned above you could use another account like Admin, it doesn't matter as long as you also change the email address for that account to what you have setup on the email test page.

 

[SilverJS]

Ah! I wasn't aware of this script and the 15 minute delay (assuming it's that). I'll unplug one now, wait and see. =) Thanks for the help so far! I REALLY want to keep FreeNAS! =)

 

[SilverJS]

Uh-oh - not good. FreeNAS just totally froze on me. No e-mails, of course. I just waited 20 minutes, and nothing. I went to the server, and it wouldn't respond to my keyboard inputs - it was simply frozen. (I suspected something was amiss after I had no connection with the webGUI.) Re-connected, re-booted, and re-disconnected (a different drive). We'll see what happens.

 

[iustinig]

Just my 2 cents on the problem - i am no expert;

I upgraded to BETA3 and I actually had a lost device event that froze the box. After the event, the drive completely disappeared, no command (smart or ls /dev) knew about it, and the gui had problems showing the drive page. The zfs pool still showed the drive but was reporting an error as well (not on the drive but on the entire pool).

After a reboot (with a new sata cable) the drive came back, and i did some snooping around and I found that when you create a volume with new drives the OS makes a 2G partition for swap on every hard drive and uses the rest for your volume. In my case that means that 4 partitions (one each drive) are used for raidz1, and the 4 2G swap are used for a 8 gig swap space. The catch here is that the swap is not raidz but 4 pieces glued together and when you take one drive off (or loose it in my case) the swap gets corrupted. In my case the system froze when i tried to reboot. I think it depends on what data the system puts in the swap.

I am not sure if this 2G swap strategy was present in the freenas 8 release (didn't look for it when i had it installed) but is there in the BETA3. I hope at some point there will be an option to assign your swap (somewhere else other than the zfs drives)

Also I am not sure about the 15 minutes script. It could be there..., but if you want to speed up the process, I would start a transfer to or from the box and while that happens I would pull the drive. The detection should be instant. I believe that in order for zfs to detect a problem it need to access data on the drive. Since you have the option to turn off drives after x amount of time, i believe that the 15 minutes script does not exist. If it would, the turn off drives option would be useless.

Again ... a noob's opinion; don't jump at me if I'm wrong :)

 

[ProtoSD]

I just posted a How-To in the hacking section that includes a script as a work around to check the disks and email you. Hope this helps until it gets fixed (could be soon, but just in case).

Here's the link in case you have trouble searching ;)

Setup-Email-notifications-for-disk-failures

 

[cpotter638]

Just got my FreeNAS box setup. 1st time user. Trying to learn now. Has email notification with HDD failure been corrected in 8.0.2? Thanks.