2X - LSI 9211-8I - Upgrade to 9201-16I - Advice and Please Help

[webdawg]

I upgraded a system from OpenIndiana to FreeNAS a few months ago. At the same time, I purchased 5 more WD red drives, and went from 5x drives to 10x drives. The supermicro motherboard that I have has three slots. This is the motherboard that I have:

http://www.supermicro.com/products/motherboard/Xeon3000/3400/X8SIL.cfm?IPMI=N

The motherboard has these pci-e slots:
2 (x8) PCI-E 2.0 slot
1 (x4) PCI-E (using x8 slot)

I have a 10gbit fiber card in one of the (x8) PCI-E 2.0 slot's, and the other two slots are filled with 92118i's. The last (x8) PCI-E 2.0 slot, and the other (x4) PCI-E (using x8 slot).

The system has been working good for the last few months. Today a scheduled scrub ran, and I am getting all sorts of checksum/read/write errors.

I had a situation like this before, it was when I upgraded to the latest (2 months ago) P20 firmware. I know there was a bug in this firmware previously, but there was an update to the P20 firmware that fixed this bug (w/ freenas). Everything should have been fine, but I downgraded to P19 anyways, and I did not have an issue after that.

Like I said, today I am. No changes in FreeNAS software (I have not upgraded).

Now, I did read something about having two of the same cards in two different slot types (x8 vs x4), and I am wondering if this is the issue. I could be that I have a bad 9201-8i two.

To make a long story short, I just purchased a 9201-16I to remove any slot issue (I do not even know if this exists), and also check for bad cards.

What firmware do I need to use for the 9201-16I?

What firmware do I need to use for the 9211-8i?

Have you heard of issues with different speed slots being used like I am?

 

[Chris Moore]

To make a long story short, I just purchased a 9201-16I to remove any slot issue (I do not even know if this exists), and also check for bad cards.

Sounds like you made a short story long. You should have asked before buying instead of asking after buying. Research first, not after.

Looking at the docs for that card: https://dustinweb.azureedge.net/media/111827/sas-9201-16i.pdf
I don't think it needs anything more than a firmware update. It appears to be HBA only

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Detailed newcomers' guide to crossflashing LSI 9211 HBA and variants
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

 

[webdawg]

Sounds like you made a short story long. You should have asked before buying instead of asking after buying. Research first, not after.

Looking at the docs for that card: https://dustinweb.azureedge.net/media/111827/sas-9201-16i.pdf
I don't think it needs anything more than a firmware update. It appears to be HBA only

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Detailed newcomers' guide to crossflashing LSI 9211 HBA and variants
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

The 9201 is on the HCL.

Just for the heck of it, I read those guides, thinking there was something I did not know/see...

The crossflash guide has firmware files at the end, with all of the minor versions of the firmware undocumented.

*I knew the 9201-16i was on the HCL list (I am asking about firmware)
*I know what SAS is and how it works (are you saying to examine my cables, and my drive bays here?)
*I know how to crossflash (These cards flashed just fine)

What I need to know is:

The Broadcom phase 20 firmware labeled "Phase20-20.00.07.00"...Is it stable?

Is it stable with the 9211-8i?

Is it stable with the 9201-16i?

It looks like both cards latest firmware is: Phase20-20.00.07.00

I remember reading about bad p20 firmwares, and It was supposed to be corrected a few minor versions in (I think 4?)...but when I used it on my 9211-8i's I had issues. I thought it was just a fluke. So now I am reexamining the situation and I wonder if "Phase20-20.00.07.00" broadcom firmware is stable before I move forward again.

The only other question I have is, does anyone else remember reading about running two of the same cards in different speed slots (same array) and having issues? Does anyone do it now and not have an issue? I was reading about this issue when I was installing my dual port 10gbit card...after many hours of research I determined that the 4x slot was slowing the card down (duh), and I had to move things around. I remember reading that someone had an issue with 2x 9211-8i's in slots with different speeds...

To conclude, I think something is faulty with one of these cards, the cables, or the drive bays...but I wanted to get this information first.

 

[Chris Moore]

I remember reading about bad p20 firmwares,

Never heard of such a thing.

after many hours of research I determined that the 4x slot was slowing the card down

That is the reason why I never suggest using more than one drive controller card. One SAS controller with SAS expanders to add additional connectivity. My main server has two SAS expanders connected to a single SAS controller and I have had 48 drives in it, but only have 32 drives in it right now. There is no reason to have more than one controller unless you have more than 256 drives.

 

[Johnnie Black]

I remember reading about bad p20 firmwares,

p20.00.00.00 had several issues
p20.00.04.00 had some issues
p20.00.07.00 is fine

 

[Scharbag]

Never heard of such a thing.

That is the reason why I never suggest using more than one drive controller card. One SAS controller with SAS expanders to add additional connectivity. My main server has two SAS expanders connected to a single SAS controller and I have had 48 drives in it, but only have 32 drives in it right now. There is no reason to have more than one controller unless you have more than 256 drives.

Good advice. I run 2 controllers only because I have a 20 bay server enclosure and a 24 bay JBOD enclosure. I am lucky enough that my SuperMicro MB has multiple 8x pipes on it.

I wish I would have bought the RES2CV360 instead of the RES2SV240 when I only had a single chassis but it was more $$ and I did not think ahead. Life would be much easier with single controller. For home use, there is no way that I will ever saturate a single controller or PCI-e 8x pipe.

Cheers,

 

[webdawg]

At this point I am thinking it is the enclosure. I replaced the 2x 9211-8I's w/ the one 9201-16I, and everything seemed to hum along just fine. I ran a scrub with no errors.

Yesterday, as the scrub was called, a few hours in, it looks like 5x of the drives went offline, and one of the CSE-M35TQB had red drive fail led's across the board. I do not know what the deal is.

I have a new in box CSE-M35TQB that I replaced it with, and ran another scrub, and so far so good. I guess the next step is the sas cable, or power supply. I doubt it was the PSU, and I think I would see more errors with the cable. We shall see.

 

[Chris Moore]

CSE-M35TQB had red drive fail led's across the board

Those things shouldn't be able to go bad. It is basically just a wiring harness on a PCB. No real smarts in it.

I guess the next step is the sas cable, or power supply.

I quickly looked back at the thread and I don't see any mention of your power supply. What do you have and how old is it. It is entirely possible that under the current demand of a scrub, the supply might be passing too much ripple or voltage may be sagging. Power problems can absolutely cause drives to go offline. That is a real possibility. We just had a user here last week that solved their problems by replacing an old supply with a new one.

If you would, please look at the guidance here and give us a full rundown on your hardware:
https://forums.freenas.org/index.php?threads/updated-forum-rules-8-3-16.45124/

 

[webdawg]

The power supply I have now is:
EVGA 600 B1, 80+ BRONZE 600W

It is a year old. I may try that next.

As an update, the issue reared its head again about a month ago. With the new card, and new enclosure. This time I replaced some cables.

I also did this: I have two 5 bay enclosures, and 4 ports with 4 sata ports each. I put 1 enclosure on 2 of the SFF-8087 ports, and the other on 2 SFF-8087 ports. So the enclosures are on 'different' ports. (how could this be different then 'different cards, heh)

I do not know what to do next, if it fails. I could have a bad cable. They are highpoint cables.

If it happens again, I am ordering a new power supply, and also new cables.

two scrubs so far, and everything seems to be good.