So my next question is how much time is passing before you consider it "working" or "not working", and how are you determining this? Badblocks or dd reads? Could it be that it is always doing it, just ironically more (not working) or less (working) frequently? What are the constants, what are variables. It seems you have tried different variables, now try changing some constants (if you can), boards, powers, rams, etc. Maybe try booting into a Linux environment and seeing if the errors persist. Not sure what else to add here. Some unseen compatability issues maybe? Its a mystery when these show up sometimes.
Im pulling for you. You will figure it out eventually, and when you do please let us all know.
I am testing with dd primarily because it's simple and fast to use. The results are virtually instant in all cases. The instant I start a dd write (via SSH) command errors spew out on the FreeNAS screen (video output). Badblocks is how I initially noticed this problem as many drives were going excessively slowly.
I seriously doubt it's doing it at all when connected through the backplane to the motherboard. I ran a full badblocks scan of 4 drives recently and it seemed to go full speed the whole time. While I can't confirm there were no errors (I didn't think to check since they were working at a rate that seemed full speed), I seriously doubt it was still erroring.
I have tried running the same test in on a Debian Live CD similar issues, except debian doesn't outright say what FreeNAS does. I'll post an image of the output of one of the log files below (I don't remember the specific one m0nkey_ told me to check).
I also found this, where basically its stated, that no you cant use a 9211 on that model Dell. Claim only H200. Its
HERE, im sure you have seen it.
Fire up all disks without backplane and see how it goes yet? Backplane is the biggest variable for me now. Can you use the backplane to power all the disks in place and use SFF8087s right off the HBA's/MB SATA ports to see if the errors continue?
That is talking about the expander backplane so I'm unsure if it still applies. Also the creator of the Ode to C2100 thread has his working with H700s and crossflashed mezzanine H200s like I'm using.
There isn't a way to use the backplane to power disks and to connect the drives data using other methods.
There aren't any errors when I use extra power lines running off the PSU to power 2 drives and connect the drives to the H200 via reverse breakout cable.
Also my seller confirmed that the replacement backplane they sent me was tested before sending it and based on how I asked, it should have been tested with a 9211.
I'm not opposed to this being all some really strange compatibility issue but it feels so strange to get this particular error across multiple devices and cables. I'm starting to wonder if the issue lies somehow in the motherboard or something weird. Since I have had some weird incompatibility with my DDR3L ram on only a specific processor and only when the ram was running at low voltage instead of standard 1.5V.
Thank you for all the suggestions. I hope you don't feel discouraged by me shooting any of them down. I really appreciate the help and support.