Repeatedly dropping disks from striped RAIDZ1 pool -- Can it cause data loss?

[Apple Eater]

Hey all,
TL;DR: I have 4 of 12 disks dropping out of my RaidZ1+RaidZ1 striped volume simultaneously. Will I lose data if I keep mounting it and dropping disks? Any thoughts why they're dropping (disks are external via E-sata with independent power supply)?

First of all, thanks for taking the time to read my post. Unfortunately the detail I provide will be limited as my system is currently offline out of fear of data loss. It will remain offline until I am certain that continued attempts to diagnose will not result in data loss.

I have a FreeNAS system that has run rock solid for years. I started with a single RaidZ1 and later added another to add capacity and currently have 12 disks in a striped RaidZ1 configuration. Due to space within my case, 4 of these drives are in an external enclosure (via E-sata). Herein lies my problem.

Recently I moved across country. I powered my system up and it has been running without complaint for about a week (apart from a quick resilver of a single drive). However, starting last night at around midnight, the 4 drives from my external storage enclosure disappeared. Given my configuration, this is obviously an unrecoverable fault. This morning, I rebooted things as safely as possible, remounted, entered my encryption passphrase and hey presto! Everything is back in place. Then about 5 minutes pass, the drives disappear, and I have painfully dismounted my volume AGAIN. Hoping the third time would be the charm, I just recently tried again and experienced the same result.

My first concern: Is my data likely to be at risk from repeated "hard dismounts" of the volume? Intuitively I would say "No", but I doubt this is a use-case that is often tested. Are other members of the forum worried about continually unplugging drives, failing a volume, and adding them back? I want to be able to continue troubleshooting the issue, but if I am taking even a small risk of data loss, I'll figure out something else.

Second: Diagnosing the fault. The biggest change (in my mind) from the move is that I am currently running without my UPS (still in transit). The power supply for the external enclosure is separate from the system's main power supply (it has it's own power brick). I worry that slight disruptions in the local grid that are not significant enough to shut down the FreeNAS server but will drop power to the drives may be to blame (I know this sounds far-fetched, but I'm having trouble eliminating other components). Everything else has remained the same. I have no reason to believe my E-sata card or cable have failed. All drives seem to fail at once so drive problems are unlikely. Then again, things worked fine for about a week after arriving to my new home. I am absolutely open to other suggestions -- particularly things that are easy to eliminate!

Sorry for the large wall of text, and the lack of any useful details about system version, configuration, etc. Once I am certain I won't make the the problem worse, I'll happily collect any data that is requested for troubleshooting purposes.

EDIT: I have some error messages related to the "dropped" drives that I screenshot from the server console. Please see attached file -- sorry about the limited data.

 

[SweetAndLow]

This isn't go to go very well for you... Storage devices attached in raid encloses are not going to work very good. This is not a configuration you should be using. Not to mention that your pool is encrypted, lol.

Sent from my Nexus 5X using Tapatalk

 

[Apple Eater]

This isn't go to go very well for you... Storage devices attached in raid encloses are not going to work very good. This is not a configuration you should be using. Not to mention that your pool is encrypted, lol.

Sent from my Nexus 5X using Tapatalk

Firstly, I want to thank you for reading my post and apologize for not including much in the way of details.

Fortunately, I am not using any sort of raid enclosure; I don't know a ton about FreeNAS but I know that I want ZFS to manage my disks rather than some shoddy raid controller. The enclosure is acting only as an e-sata port multiplier and power supply for the drives. Previously the compatibility with this enclosure has been excellent, FreeNAS detects all disks independently, full S.M.A.R.T support for all drives, and not a single hiccup in terms of reliability (frankly I am fairly surprised given how cheap it looks).

Secondly, I recognize this configuration (even as I have described) is far from the optimal FreeNAS deployment. This is purely a home setup that I use to store large amounts of not-terribly-sensitive data. I may have overstated how upset I would be if data loss were to occur. It would be a significant annoyance, but it's happened before and I'm sure it'll happen again. Everything important is backed up off site.

Finally, the encryption point. I wasn't aware that the encryption employed by FreeNAS had significant reliability implications. I'll be the first to admit that I have suffered data loss by using FreeNAS encryption (lost a whole 20TB volume due to my own stupidity and failure to read instructions) but I didn't realize that it could be dangerous even if used properly. What sorts of things should I be looking out for when using encrypted volumes? Is the advice to avoid them entirely?

Again, thanks for taking the time to read my post and respond. I appreciate this may not go well for me -- if that's the case, I'll take my licks, upgrade to a proper SAS enclosure next time, and move on with life.

 

[Mirfster]

By chance, is the JBOD at least connected to a Surge Protector? I know you are awaiting the arrival of your UPS, but am trying to eliminate the "low hanging fruit".

You mentioned that it is connected via eSATA, if only the disks are dropping from that enclosure then you may want to double check the cable as well.

The enclosure is acting only as an e-sata port multiplier and power supply for the drives.

Can you elaborate on this a bit more? Normally "Port Multipliers" are not recommended and again since you are mentioning that it is these 4 drives that are dropping (if I understand you correctly), I would first focus around all items pertaining to the JBOD.

P.S. Never hurts to double check all connections, especially after the equipment has been moved recently.

 

[rs225]

You are wise to not touch this system until you can fix the problem.

If ZFS notices the drives are dropped immediately, the risk of loss is very low. If ZFS doesn't notice immediately, the risk of loss is very high.

As long as ZFS notices the drives are gone, it will freeze the pool and you are ok. But that does not mean you should take any chances. Fix the problem, whatever it takes. A possibility is that your power brick's output has degraded enough to cause this. Replace it.

 

[SweetAndLow]

Check all your cables and make sure you have a clean source of power. Your enclosure could be going bad also. Do you have a way to take the disks out and plug them directly into your main system?

 

[m0nkey_]

You don't mention what SATA controller you're using. It's very well known to this forum that Intel controllers have a hard time with port multipliers.

 

[philhu]

Also, make sure the external box and freenas system are on the same outlet with the same ground.

Your ground differential cannot be guaranteed if you plug into two different outlets

 

[Apple Eater]

Wow, incredible thanks to all who have responded. Great ideas all around! I never expected such a great response. I'll update with my current situation:
I've checked and re-checked all my connections and swapped the enclosure power supply to the same power strip as the main chassis (never would have thought of this!). I swapped E-sata cables and ports on the controller.

The good news is, the volume was mounted and has remained stable overnight for a volume scrub. I'm currently 50% thru scrubbing the volume and have found no permanent corruption yet (great news!). I haven't yet restored any of my workloads (jails, external systems, etc) so time will tell how it goes under load.

For those with questions about my hardware setup:
Enclosure: https://www.amazon.com/gp/product/B003X26VV4/?tag=ozlp-20
E-Sata Card: https://www.amazon.com/gp/product/B003GSGMPU/?tag=ozlp-20
Mainboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813157475
Case: http://www.newegg.com/Product/Product.aspx?Item=N82E16811163255

Only issues I've had from the setup (other than the current issues) are needing to update one of the onboard controller firmwares (can't remember which) to achieve full SMART support within FreeNAS.

Thanks so much for the feedback, and I hope the system remains stable for the rest of the scrub and then I'll start my normal workloads back up. The UPS should be arriving soon to completely eliminate and power quality issues (though I am investigating running the enclosure off the internal power supply -- if the enclosure drives lose power, I want the whole system to die!). I have not yet replaced the enclosure power supply but that is next if I detect even a hiccup.

Very impressed at my reception here and the helpfulness of this community. I'll try to post an update to this thread in a few days if the system is still stable. My best guess for the culprit is power (possible ground differential -- what a good catch) but if it happens again I'll update for sure.

 

[Apple Eater]

Hey all,
Another update. Unfortunately the drives continue to drop randomly, the volume scrub did not complete. I've finally got everything hooked up to a UPS, no change, issue persists. I've now ordered a proper SAS card (https://www.amazon.com/gp/product/B0085FT292/?tag=ozlp-20), SAS enclosure (https://www.amazon.com/gp/product/B00OHJY6AA/?tag=ozlp-20) and SAS cable. I think e-sata expansion was a mistake, as was purchasing such a cheap e-sata card and enclosure. I'll migrate the drives to the SAS enclosure when it arrives and hopefully be able to bring my volume back online.

Given that enclosures in general are a bit unreliable, what is recommended for storage expansion beyond what my case can hold? Should I just get a bigger case? Are some enclosures reliable? Unfortunately the 8 bays provided by my case simply do not scale to my current storage needs. Is it time to upgrade my whole system?

 

[Mirfster]

Not to sound harsh, but for what you spent (~$417.00), I would have just grabbed a 2U 12 Bay Server off of eBay instead. The Silverstone Tek 4-Bay Storage looks cute and all, but doesn't have dual PSU. Also, now you have two systems to work with for just 12 bays.

BTW, did you ever post your system specs? Would help us provide info. Like if you could easily just buy a case and swap parts instead.

 

[Apple Eater]

Not to sound harsh, but for what you spent (~$417.00), I would have just grabbed a 2U 12 Bay Server off of eBay instead. The Silverstone Tek 4-Bay Storage looks cute and all, but doesn't have dual PSU. Also, now you have two systems to work with for just 12 bays.

BTW, did you ever post your system specs? Would help us provide info. Like if you could easily just buy a case and swap parts instead.

Wow, I had no idea how right you were. Took a look at Ebay and found a 4U 36 bay system that should suit very nicely for $450. I think maybe it is time I retire my current setup and move to something a bit more professional grade.

I posted some of my specs above (specific links to parts). I don't think many of my parts will be useful in any other applications, and with the very low cost of a complete system on Ebay, I think an upgrade might be the better approach. I can easily just wait a few days for the 36 bay system to arrive and import my volume on that system at that time. I'd like to get up and running before then, but I just don't have the space in my case to fit the drives, nor the internal SATA ports to support that many, and frankly I'd rather not risk my volume by continuing to mess with this flaky enclosure setup.

I may end up repurposing my current setup as an offsite backup solution for a better NAS on site. I know that the offsite backup capabilities for ZFS systems are pretty awesome.

Thanks again for the advice. It sounds like I probably should've created a forum thread when I first start building out my system, I could've avoided some costly mistakes.

 

[Mirfster]

I posted some of my specs above (specific links to parts). I don't think many of my parts will be useful in any other applications, and with the very low cost of a complete system on Ebay, I think an upgrade might be the better approach.

I would think so. However, you may be able to use the "ASRock C2750D4I Mini ITX Server Motherboard FCBGA1283" in a better case and save some money. I haven't checked out the Motherboard specs, so I am not 100% sure.

If you are going this route, for sure start a "Build" thread on it. To get the best feedback make sure to:

• State your "Use-Case"
• Budget
• Link(s) to any item(s) you are considering for purchase
• Concerns (say like Space or Noise... thinking a 4U is going to get pretty loud...)
  
• Preferences (like IPMI is needed or I want SuperMicro)