3ware JBOD - RAIDz pool fails to degrade upon drive failure

[trininox]

Greetings,

I'm testing FreeNAS as a replacement for several NAS servers we have that previously ran another product (Open-E).
The hardware will be intact except an upgrade to the memory. However a test has not gone as expected.

Hardware,
SuperMicro X7DBN
Dual Xeon 5130 @2Ghz
2GB RAM - 4x512mb FB-DIMM DDR2 (currently) (16GB is on order)
3ware 9650SE-16ML
FreeNAS-8.3.1-RELEASE

I have 16 500GB Drives connected,
I created a pool of 5 data, 1 log, 1 cache, 1 spare.
I extended that pool with 5 data, 1 log, 1 cache, 1 spare.
I connected to Active Directory.
I setup a CIFS share or two.

So far great,
Lets try causing trouble.

I pulled drive #16 (da15) to see what would happen, nothing.

I couldn't get it to acknowledge a drive failure/missing.
I could probe using the shell commands and see it was missing, smartctl for example.
It still listed the drive online, the pool healthy, I could still change settings for the drive.

I did a reboot, and sure enough the drive was found missing and a spare was put into action, I think.
One spare>stripe is da7p2 the other is a string of numbers.

I've been searching around the forum and the web but nothing recent,
I did find this from a previous version, 2011. Is this still and issue???

Removed-disk-does-not-cause-ZFS-to-degrade-without-restart

Solution? I saw somewhere scrubs may aid in detecting the fault, but I also say that can be counterproductive to performance if they are ran often.

 

[ProtoSD]

Sorry you didn't search very well, yours is the 2nd post this week about this topic.

 

[jgreco]

For an HBA-attached drive:

FreeNAS 8 has poor support for detecting drives that have been removed. ZFS will eventually decide that the device isn't acting properly and will figure out there's a problem with the pool, but that will not happen instantly. In theory, since the OS sees the drive removal event, it ought to be able to do better. A framework is being developed in FreeBSD 9, called zfsd, to handle these sorts of meta-issues that kind of straddle the line between userland and kernel.

For a RAID-attached drive (your situation, because on the 3Ware controllers, disks are always virtualized into devices):

FreeNAS 8 has poor support for detecting drives that have been removed. ZFS will eventually decide that the device isn't acting properly and will figure out there's a problem with the pool, but that will not happen instantly. However, since the RAID controller probably masks the drive removal event from the OS, detecting and handling this sort of event will vary based on the type of RAID controller, and as such, probably won't happen for some time due to the complexity.

In both cases, ZFS should eventually realize that there's a problem when the missing disk hasn't responded correctly. I haven't tried this lately to see what actually happens, though. Generally speaking, most people are working around this by monitoring SMART and for SMART warnings and failures.

 

[bernardc]

Make a sticky

A comment in the other thread on this persistent flaw this week: "it's probably one of FreeNAS's biggest weak spots." True. So why not advertise it? Has any of you heard of "Failure to Warn?" Why not write a "Best Practices" manual on how to work around the trouble?

I actually don't think it's FreeNAS's biggest weak spot. The biggest weak spot are the snippy, snarling, snotty attitudes in the forum. Every time I make the mistake of reading something here I see questions asked in good faith and with care answered with information-lacking, belittling, and tiresome snubs. If you can't provide useful information, just put a cork in it and save your venom. A positive example that should be followed is that of jgreco, from whom I learn something from nearly all of his posts.

Thanks.

 

[jgreco]

There's something to be said for n00b-failure-to-(re)search leading to forum participant burn-out, because the number of active experienced posters on the forums is disappointingly low.

But really I can't have you holding me up as a positive example on these forums. I'll have to go take some more grinch pills, or maybe move all my posting activities to early morning pre-coffee.

 

[ProtoSD]

Re: Make a sticky

A comment in the other thread on this persistent flaw this week: "it's probably one of FreeNAS's biggest weak spots." True. So why not advertise it? Has any of you heard of "Failure to Warn?" Why not write a "Best Practices" manual on how to work around the trouble?

Perhaps one of the reasons I'm so "grinchy" is because I've been holding my breath for so long waiting for permission to edit the bloody FAQ that was ripped from my blog and left with no way to update it. Last time there was an uprising here, I was told to hang on for some new software to manage/edit the FAQ. The disk failure notification has been an ongoing gripe from people right from the beginning, so for almost 3 years now? I suppose it should have been in the FAQ, I haven't looked at it for so long now I don't remember if it's there in some form. If there is ever a new FAQ, the questions are going to need to be phrased 15 different ways just so people can find it in a phrasing that tickles their elmo, if they take the time to look at it.

I've been meaning to ask how your power supply disaster turned out, PM me if you want since we've kinda hijacked @trininox thread. Sorry Triminox!

There's something to be said for n00b-failure-to-(re)search leading to forum participant burn-out, because the number of active experienced posters on the forums is disappointingly low.

So true.

But really I can't have you holding me up as a positive example on these forums. I'll have to go take some more grinch pills, or maybe move all my posting activities to early morning pre-coffee.

Been meaning to say I like your new avatar :) Though you and I have had our differences, you do whip out some good stuff ;)

 

[jgreco]

So not to poke at an already sore point with you, but am I to assume what you're referring to is the forum FAQ at http://forums.freenas.org/faq.php rather than the FreeNAS FAQ at http://doc.freenas.org/index.php/FAQs?

I don't really have an idea what's transpired or why ... so I'm just mostly clueless w.r.t. this.

From my perspective, the forum software has seen improvements, apparently at the hands of HolyKiller, so that forum FAQ might be correctable now that someone with forumware experience is involved. Or am I wrong?

As for the doc site (MediaWiki) FAQ, it's editable and you appear to have access to it. Further, the FAQ seems to be in mostly an abandoned state. Is there some reason this can't be claimed?

I also note that the stuff that I can identify as having originated with your FAQ has somewhat rotted as it appears to have been written in 8.0.x days. That is not meant as anything other than an observation of the current state of affairs, and that we could view it as an opportunity to think about how best to tackle the FAQ issue while also bringing it all up to date.

Having an FAQ in the forums and a different FAQ elsewhere strikes me as somewhat confusing, because people refer to "the FAQ" but there's two. My thinking: eliminate one by merging/updating.

The MediaWiki FAQ is probably the more maintainable because iXsystems appears to have Dru Lavigne maintaining the doc site, so I suspect any outages or problems there will tend to get more attention because it involves iXsystems more directly. MediaWiki has a fairly rich set of features for documentation use. I have no idea what the forumware FAQ module can provide.

Some of the stuff such as http://forums.freenas.org/showthrea...rdware-should-I-Buy-Get-Use-or-is-recommended is kind of a confusing mishmash of two-year-old unobtainable stuff plus some small amount of contemporary content, and the way I see it, it'd be better not to maintain multiple lists of hardware in multiple places. I found that particular link sufficiently outdated that I posted a different list in the Hardware forum. This is saying to me, "unified FAQ with subpages and encourage lots of linking". That way, multiple people can edit and maintain, but it doesn't get to be unwieldy in the manner of a single massive forum Sticky post with lots of semi-meaningless followup responses tacked on.

My goal would be to be able to redirect end users who are asking a frequently asked question to the FAQ with a terse response and a link, and to have that FAQ answer their question more comprehensively than any one-on-one forum response would have been likely to have conveyed.

 

[ProtoSD]

So not to poke at an already sore point with you, but am I to assume what you're referring to is the forum FAQ at http://forums.freenas.org/faq.php rather than the FreeNAS FAQ at http://doc.freenas.org/index.php/FAQs?

I don't really have an idea what's transpired or why ... so I'm just mostly clueless w.r.t. this.

This post gives a brief overview of what's transpired, not including the nomination fiasco that Alfred put us through without any followup.
FAQ-Mounting-NTFS-Drive-to-Copy-Files-From#post52644

From my perspective, the forum software has seen improvements, apparently at the hands of HolyKiller, so that forum FAQ might be correctable now that someone with forumware experience is involved. Or am I wrong?

Holykiller was able to make some changes, but nothing like was discussed, and none of the stuff that was further requested in followup discussions. My last contact with Holykiller he said his messages to Alfred went unanswered. Typically this is how things have gone when there's been an uprising here, full of good intentions and promises and very little if any follow through.

As for the doc site (MediaWiki) FAQ, it's editable and you appear to have access to it. Further, the FAQ seems to be in mostly an abandoned state. Is there some reason this can't be claimed?

It's been too long since I played with the wiki version, but I seem to recall it had some editing limitations, I could be wrong. The current Forum FAQ from my blog requires admin privileges to edit and I'm not sure how well it supports screenshots.

I also note that the stuff that I can identify as having originated with your FAQ has somewhat rotted as it appears to have been written in 8.0.x days. That is not meant as anything other than an observation of the current state of affairs, and that we could view it as an opportunity to think about how best to tackle the FAQ issue while also bringing it all up to date.

Yeah, it's pretty outdated, I gave up after it was taken and I couldn't edit it, that was about a year ago.

Having an FAQ in the forums and a different FAQ elsewhere strikes me as somewhat confusing, because people refer to "the FAQ" but there's two. My thinking: eliminate one by merging/updating.

That's why I used to have a link for it in my signature and most of the tutorials I wrote.

The MediaWiki FAQ is probably the more maintainable because iXsystems appears to have Dru Lavigne maintaining the doc site, so I suspect any outages or problems there will tend to get more attention because it involves iXsystems more directly. MediaWiki has a fairly rich set of features for documentation use. I have no idea what the forumware FAQ module can provide.

Don't get me wrong, I think Dru does an outstanding job with the documentation and all the stuff she has on her plate. When I first added stuff to the wiki FAQ, it had like 2 whole questions. I added about 20 or so, and over half got deleted.

Some of the stuff such as http://forums.freenas.org/showthrea...rdware-should-I-Buy-Get-Use-or-is-recommended is kind of a confusing mishmash of two-year-old unobtainable stuff plus some small amount of contemporary content, and the way I see it, it'd be better not to maintain multiple lists of hardware in multiple places. I found that particular link sufficiently outdated that I posted a different list in the Hardware forum. This is saying to me, "unified FAQ with subpages and encourage lots of linking". That way, multiple people can edit and maintain, but it doesn't get to be unwieldy in the manner of a single massive forum Sticky post with lots of semi-meaningless followup responses tacked on.

I think that post was me trying to get people to post what they were using so others would stop posting the "review my hardware" stuff. I think I ended up signature surfing to add most of that, but it's too much to maintain without any help, especially when it just gets disregared.

My goal would be to be able to redirect end users who are asking a frequently asked question to the FAQ with a terse response and a link, and to have that FAQ answer their question more comprehensively than any one-on-one forum response would have been likely to have conveyed.

You have some good suggestions, I think I'm pretty burned out with all the BS I've been through trying to help iX when they can't take the time to help the people that are trying to help them. Back during the good intentions phase in the last uprising, I was told they were thinking about implementing THIS, but it's been a couple of months and I was under the impression that this was going to be part of the new website, but again, no updates. I might be able to be re-inspired, but it's going to take more than unfulfilled good intentions.

And yes, I haven't had my anti-grinch coffee, so I'm sure you can sense my warm and fuzzy attitude ;)

 

[cyberjock]

I've been closing the tabs on my web browser more and more. Just tired of reanswering "that" question from yesterday. Today while I was driving home I was asking myself if I'm really doing any good by even sticking around. It feels like I'm wasting more and more time every day here. :(

 

[James]

This post gives a brief overview of what's transpired, not including the nomination fiasco that Alfred put us through without any followup.
FAQ-Mounting-NTFS-Drive-to-Copy-Files-From#post52644

In less words, I emailed Proto, told him I planned to migrate the FAQs to the forums so that the burden was distributed. This meant that I painstakingly copy/pasted everything for you so that life would be easier for everyone. Boy was I wrong. I'm not sure how I fat fingered the permissions, but at one point you were a super mod, and it was expected that you'd let me know if there was trouble. You seemed pretty optimistic about the whole thing at the time, then dropped off the map. Or maybe I did. I feel as if this one mistep on my part caused the falling out we're currently experiencing with you Proto. I have a screenshot of our initial conversation if you'd like to see where you stood that day. I promise you I had no intention of "stealing" your content, credit, or credibility.

I was not very good at the vBulletins. Let's not take this personally, please.

We can easily start a FAQ in Xenforo buy adding yet another add-on, as long as it does what you need it to do. If it's traffic you're concerned with, then I am sure we can put your blog on blast should you happen to author a new FAQs for 9.1. FreeNAS.org gets enough traffic for everyone. Let's play nice.

 

[trininox]

Apart from the meltdown, the original case for this thread and the constructive portions toward it did allow me the "feel good" about going forward with FreeNAS for our Archive, 5.5TB out of 30TB later. I did go with two 8 disc vdevs using Z2.

Look forward to a complete FAQ and contributing what experience I can. Thanks.