Does camcontrol debug change controller operation? SCSI reset errors driving me mad

[Bezerker]

Hi all,

I'm troubleshooting an issue with a new hardware order from mrrackables specifically designed for freenas. It's a 24U 2.5" bay supermicro, 9211-8i SAS2008 controller, with a BPN-SAS2-216EL1 backplane.

I've thrown 6 750g wd 7200rpm spinning disks in it, as well as 8 new 4tb seagate 5400 rpm spinners.

I'm running into issues where no matter what I do, if I run >4 of the 750g's at once (stripe, raidz, mirror, anything) under heavy IO load (writes especially) I get tons of errors: http://pastebin.com/raw/JhSFNYx8 . Running less than 5 of the 750s I have no problems at all but if I run the 5400 seagates at the same time i get errors again.

It will occur on random disks and kill IO. The 5400 rpm disks alone do the same, but much much more rarely.

This screams power to me, but to be safe, I've run both powersupplies shipped with the device and even run them in redundant mode (both in the machine at the same time.) Barring that (It may still be power but I have no easier way of testing that for now), I have replaced the SFF8087 cable going from the controller to the backplane expander. I've replaced the controller witha 9240-4i I had laying around just to check as well, and had similar issues. This leaves me the backplane/expander to replace, which rarely fail in my experience in this method but I'll confirm soon when I the replacement I found for 90 bucks arrives. Still swapping a backplane out is annoying.

I have updated the firmware/bios of the controller but not the expander (I cannot find an easy way to do this or acquire the firmware.) I have ONLY tested this on freenas 9.10 as I don't have a simple way of testing another OS like linux at the moment, but what I HAVE noticed is that throwing the disks into debug mode with camcontrol debug -c all seems to mitgiate the issue. I do still notice the LEDs lighting up and the disks "Freezing" but for a significantly less time (not enough to cause IO performance issues).

I'm hoping you guys can check my sanity here with the following:

1> power sounds like a sane thing to be concerned with though its an 800w power supply or something along those lines. It should be fine and 6 disks shouldn't be insane.
2> Backplane/expander is worth replacing given the current fact I've replaced everything else. Could firmware on the expander cause this? I really have no idea how to update this or where to get it.
3> Could this be an OS level issue? Is 9.10 known to have some kind of issues with certain hardware? I hope not.

Thanks all I appreciate any help.

Thanks,
Bez

 

[mjt5282]

what motherboard do you have installed? Which PCI slot is your LSI card installed in?

 

[DrKK]

I mean, everything you're saying, literally everything you're saying, suggests brownouts on the power rails from load. The power supply may be rated for 800w (obviously sufficient), but it still may be defective. I would positively rule out the power supply before I started poking at backplanes et al.

 

[Bezerker]

what motherboard do you have installed? Which PCI slot is your LSI card installed in?

Running it on a tyan 7012s. Have tried multiple pci-e 8x slots to rule out a bad slot. Even took out the 10g card thinking it may be interfering.

 

[Bezerker]

I mean, everything you're saying, literally everything you're saying, suggests brownouts on the power rails from load. The power supply may be rated for 800w (obviously sufficient), but it still may be defective. I would positively rule out the power supply before I started poking at backplanes et al.

Hrm. Realistically, I was thinking the same thing. It does seem odd however that both power supplies included with the device would be bad. I do want to take a closer look at the connectors and be sure all 4 power plugs for the backplane are not all on the same rail just split, that could easily do it.

Thank you for confirming this sounds something like a brownout on the power rails. I wish I had a decent method of testing the voltages. At this point I'm tearing my hair out because I'm 99.9% there to finishing my home system rebuild, and this storage box is a key part of it hah.