SOLVED Constant errors freshly formated disks

[meisnick]

I am running 9.10 Stable
ASRock E3C224-4L latest bios
i3-4170
16gb DDR3 1600 ECC memory
2 Dell Perc H200 Cards flashed to Avago p20 IT firware
14 WD red 3TB drives

The system was running fine up to about a week ago where it dropped several hard drives
I rebuilt the Raidz3 array 2 times and pulled the erroring drives and ran a full destructive surface scan and found no errors with the disks.

I ran a weekly backup to a second array and towards the end of the backup the web interface crashed files stopped copying and the system dropped its network connection while IPMI continued to work on next IP address

I rebooted the system everything came back up finished copy but again it failed the same way near the last files

I pulled several of the disks and ran SMART and surface checks in another machine which all came back good

Rebuilt the array on a fresh install of 9.10 array failed several drives suffered Error 22 and timed out
different drives every attempt

Destroyed the array ran DBAN across all drives to reset everything back to factory fresh

Built the array again and it was successful write errors immediately started happening on multiple drives from both Dell cards

Shut system down and ran memtest86 and viewed system diagnostics nothing is wrong

I'mat a loss I've been running some form of freenas since version 7 and I've never had a system spin into instability like this


after 23 mins of up time

 

[Bidule0hm]

Check the SATA cables connexions, check the power connexions, try to change the SATA cables, ...

What PSU are you using?

 

[BigDave]

After reading your post, I notice you did not mention having done badblocks testing on these drives.
My suggestion is to divide the drives into two pools (one on each controller) and badblock test
all the hard disks and see what shows...

 

[meisnick]

I appreciate the suggestions on dividing up the drives into smaller groups

I unplugged everything except a single 4 sata cable off one Perc card and tested drives in sets of 4

Everything checked out this eliminated the drives from the equation

I checked a set of drives with different sas cables in rest of the ports on the Perc cards

Located the port on the Perc card causing errors, no physical damage, cleaned port changed cable no change

Guess the $30 used Perc passed its last bits

 

[BigDave]

Located the port on the Perc card causing errors, no physical damage, cleaned port changed cable no change

Guess the $30 used Perc passed its last bits

Nice process of elimination
Glad you were able to find the problem :)