Pool is offline. i/o failure. don't use sata to pci 1x cards.. Drives are fine.

[AlexGG]

likely caused from a cable issue,

Another good candidate to blame is PCI-E x1 SATA controller. Some models do not do well under load.

 

[joeschmuck]

Did you not have any warning signs of data issues before this occurred? Do you know when the last SCRUB was completed and if it have any error messages? On your other systems that are operational, I'd see when the last SCRUB was completed and if it's been a while, run a SCRUB. The default is monthly but you can change that. I leave mine on the default. Do you have email notification setup?

I'm trying to make sure that you do not suffer this again on a different system. It's a terrible feeling to lose your data.

 

[Whattteva]

No judging, I was broke and this all I had to work with.. Not my best work.. I use these female connectors to male all the hard drives. I only plug into the main power supply with them. No daisy chains. 500 watt power supply. There are 3 chains for all the drives.
12 Drives total.

View attachment 59305
View attachment 59306
View attachment 59307
View attachment 59308

OMG, this looks like it came straight out of my nightmare. I suspect all your power cycle issues is at least, in part, due to this crazy card & cabling, which in turn, causes all the data corruptions.

I'm not trying to judge, but you're a brave soul for doing this on 16 TB drives on data you actually care about losing with no backup.
ZFS is rock solid with proper setup, but you can't blame it when you set it up for failure though, all bets are off.

Let this be a lesson that when it comes to data you care about losing, it really isn't worth playing with fire. Better to wait when you're not broke and build, then play with fire while you're broke and cry and weep 1 year later after it all crashes and burns spectacularly.

If I were you, I'd start the process of migrating the remaining drives of your other pools to a more sane setup before you encounter another fate like this one.

 

[CookieMonster1]

OMG, this looks like it came straight out of my nightmare. I suspect all your power cycle issues is at least, in part, due to this crazy card & cabling, which in turn, causes all the data corruptions.

I'm not trying to judge, but you're a brave soul for doing this on 16 TB drives on data you actually care about losing with no backup.
ZFS is rock solid with proper setup, but you can't blame it when you set it up for failure though, all bets are off.

Let this be a lesson that when it comes to data you care about losing, it really isn't worth playing with fire. Better to wait when you're not broke and build, then play with fire while you're broke and cry and weep 1 year later after it all crashes and burns spectacularly.

If I were you, I'd start the process of migrating the remaining drives of your other pools to a more sane setup before you encounter another fate like this one.

If this was you what parts would you buy? Brand new build for only ZFS. Please try to keep it under $550. Its needs to hold all 12 drives. Keep the same power supply and sata cables that are new?

 

[CookieMonster1]

Did you not have any warning signs of data issues before this occurred? Do you know when the last SCRUB was completed and if it have any error messages? On your other systems that are operational, I'd see when the last SCRUB was completed and if it's been a while, run a SCRUB. The default is monthly but you can change that. I leave mine on the default. Do you have email notification setup?

I'm trying to make sure that you do not suffer this again on a different system. It's a terrible feeling to lose your data.

My storage server is offline my next resilver is in a few days, not letting that happen this month.

 

[Cosmix]

If this was you what parts would you buy? Brand new build for only ZFS. Please try to keep it under $550. Its needs to hold all 12 drives. Keep the same power supply and sata cables that are new?

I’d get at least 1 if not 2 ($50 each) LSI 9200 / 9211 cards with 2 or 4 mini sas to 4 sata splitters ($10 each) and a 650w seasonic psu ($150-200). Keep your case motherboard cpu ram for now but look to upgrade them when you have more budget. Also toss all those jank splitter cables you’re using at the moment.

 

[Whattteva]

If this was you what parts would you buy? Brand new build for only ZFS. Please try to keep it under $550. Its needs to hold all 12 drives. Keep the same power supply and sata cables that are new?

You can start here.

 

[CookieMonster1]

I’d get at least 1 if not 2 ($50 each) LSI 9200 / 9211 cards with 2 or 4 mini sas to 4 sata splitters ($10 each) and a 650w seasonic psu ($150-200). Keep your case motherboard cpu ram for now but look to upgrade them when you have more budget. Also toss all those jank splitter cables you’re using at the moment.

I am going to buy 3 9211 cards. Link:

LSI 9211-8i 6G SAS HBA FW:P20 IT Mode ZFS FreeNAS unRAID 2* SFF-8087 SATA US | eBay

LSI 9211-8i IT Mode Raid + 2 SAS SFF-8087 to 4x SATA Cable.

www.ebay.com

INSPUR 9211-8i 6Gbps SAS LSI 2008 HBA IT Mode ZFS FreeNAS unRAID+2*SFF-8087 SATA | eBay

Find many great new & used options and get the best deals for INSPUR 9211-8i 6Gbps SAS LSI 2008 HBA IT Mode ZFS FreeNAS unRAID+2*SFF-8087 SATA at the best online prices at eBay! Free shipping for many products!

www.ebay.com

What's the difference? Prefer not to buy from china. Only USA. Liking the first one $49.

Just to make sure, does everyone agree that will stop all future problems with the i/o failure? I have 1 last pool with all my data on it, 1 year old copy. Needed the newer one but at least it has most of my pictures from growing up.. Really scared when I start copying it fails.. Please make sure this is the right move. Thank You for all your help! Ready to buy it!

 

[CookieMonster1]

I do have another nas I never talked about, this all on ssd's but z0. So no backup drives. Would always copy to that storage pool for a backup monthly. Replacing that sata to pci 1x card. 4x 1TB ssd's.

The data recover hasn't gone to plan, it fails to recover anything after 23% on my 3rd try. Still running 24/7. Will get an update in another day or soon. At 23% right now.

 

[CookieMonster1]

Should get rid of the power spitters for all the drives? I won't be changing my power supply for the time being. Is it a good idea?

 

[jgreco]

500 watt power supply. There are 3 chains for all the drives.
12 Drives total.

500W? This is catastrophic. It should be in the 750 watt range, according to the precalculations made in the Proper Power Supply Sizing guide.

Should get rid of the power spitters for all the drives?

Absolutely. I'm ashamed that no one climbed all over that one. The "SATA power" connector is sucky. Each pin is rated to handle only 1.5 amps, and three are typically ganged, so the SATA power connector hardwired to your PSU is only able to deliver 4.5 amps for each of 5v and 12v. We know some drives to require spinup currents north of 2.2 amps on the 12V. If you have a three-way splitter, then you might be trying to draw 6.6 amps through a connector only capable of handling 4.5. This can (and does) cause SATA power connectors to melt and arc.

 

[jgreco]

Ah, with picture.

BUILD - Please review my Intel Skylake FreeNAS build

If you do want the SATA adapters, get crimped not moulded ones. GOOD BAD

www.truenas.com

 

[CookieMonster1]

Ah, with picture.

BUILD - Please review my Intel Skylake FreeNAS build

If you do want the SATA adapters, get crimped not moulded ones. GOOD BAD

www.truenas.com

Ok, ok ok, I'll buy a power supply.. I still don't want to spend money.. What about the power connectors. What should I buy? Just show me what to buy on amazon, ebay, newegg for the cables.

 

[CookieMonster1]

Buy 3 cards below:

LSI 9211-8i 6G SAS HBA FW:P20 IT Mode ZFS FreeNAS unRAID 2* SFF-8087 SATA US | eBay

LSI 9211-8i IT Mode Raid + 2 SAS SFF-8087 to 4x SATA Cable.

www.ebay.com

1 850Watt power supply. Has 12 power connectors for all 12 drives.

Thermaltake Toughpower GF1 850W 80+ Gold SLI/Crossfire Ready Ultra Quiet 140mm Hydraulic Bearing Smart Zero Fan Full Modular Power Supply 10 Year Warranty - PS-TPD-0850FNFAGU-1 - Newegg.com

Buy Thermaltake Toughpower GF1 850W 80+ Gold SLI/Crossfire Ready Ultra Quiet 140mm Hydraulic Bearing Smart Zero Fan Full Modular Power Supply 10 Year Warranty - PS-TPD-0850FNFAGU-1 with fast shipping and top-rated customer service. Once you know, you Newegg!

www.newegg.com

Is that all I need?


For a 6900xt I have a 850watt power supply should I need a 1000w? I could buy a 1000watt and replace it with my 850watt for the storage server.

 

[CookieMonster1]

Bought the 1000w power supply, will do a swap on them.

 

[joeschmuck]

I'm ashamed that no one climbed all over that one.

I mentioned it in one of the earlier posts but did not force the issue and did not reference the guide. I suggested that the power supply may be faulty or undersized.

 

[jgreco]

I mentioned it in one of the earlier posts but did not force the issue and did not reference the guide. I suggested that the power supply may be faulty or undersized.

I was referring to the power splitter cables. These are almost never a good idea.

There's a good possibility, given the power cycle counts, that there were a bunch of brownouts due to overdrawing on the 12V. If that wire on the PSU side is 18ga, it should be able to handle 10-15 amps. If you have multiple SATA power connectors on the PSU cable, and you hook up two power splitter cables, you also get near the current carrying capacity of that PSU cable as well. In the meantime, both the power splitter's input connector and the PSU's SATA power connector that it is connected to are both experiencing occasional power overloads as the drives spin up. At best, this can cause brownouts. At worst, the plastic can start melting.

 

[joeschmuck]

I was referring to the power splitter cables. These are almost never a good idea.

Agreed. I've never seen drives with so many power cycles. One with basically double the power cycles than hours, crazy.

It very well could be the power splitters, that makes more sense since there was no report of the computer crashing. I didn't catch that. Good thing you did.

 

[jgreco]

It very well could be the power splitters, that makes more sense since there was no report of the computer crashing.

Yeah, a brownout in the SATA power cabling due to overheating/arcing/melting is less likely to affect the host system.

I didn't catch that. Good thing you did.

Many eyes make for easier troubleshooting, or something like that.

 

[CookieMonster1]

The power just got here. Switching out my 850w to 1000w and putting the 850w in the storage server. Sata cards will be here later today. Will have time on Sunday to build it.