HBA Card Firmware

[Fastline]

Not strictly PLX (they're fine, but they were bough by Broadcom/Avago around the same time LSI was and PCIe switch prices shot up up up following that), but yeah, PCIe switches are the way to go.

I mean Wow. Patrick, Etorix and dan introduced me to PLX thingy and i was not aware that Broadcom bought it. xD

 

[Ericloewe]

Fun fact:* "Monopolize the storage attachment market" is an anagram of "Broadcom".


* Not guaranteed to be factual.

 

[nabsltd]

I can't actually think of a compelling reason to flash the legacy BIOS OpROM.

Unless you completely erase the flash, it's already there in whatever version it shipped with. Might as well flash it.

Also, the BIOS UI is often better than the UI inside the UEFI config provided by the motherboard vendor.

 

[nabsltd]

What for? A single -8i card can easily handle a whole chassis full of disks, and more.

This is not actually true for the 943x and 953x cards. They are artificially limited to 32 drives per card, and there are single chassis with more than 32 drives.

Yet another reason to stick with the 93xx cards, as their limit is 256 drives.

 

[Ericloewe]

This is not actually true for the 943x and 953x cards.

Those are low-end RAID controllers, so it's a bit of a moot point.

 

[Fastline]

Unless you completely erase the flash, it's already there in whatever version it shipped with. Might as well flash it.

Also, the BIOS UI is often better than the UI inside the UEFI config provided by the motherboard vendor.

Makes sense!

 

[Fastline]

This is not actually true for the 943x and 953x cards. They are artificially limited to 32 drives per card, and there are single chassis with more than 32 drives.

Yet another reason to stick with the 93xx cards, as their limit is 256 drives.

Wow.

 

[Fastline]

Those are low-end RAID controllers, so it's a bit of a moot point.

What do you mean? Is 94XX not good like 92XX or 93XX?

 

[nabsltd]

What do you mean? Is 94XX not good like 92XX or 93XX?

He means they aren't HBAs, so they aren't good with TrueNAS.

But, the 94xx series are "TriMode" (SATA/SAS/NVMe), and it turns out this means they aren't really good at any of them. If you only want to use SATA or SAS in large numbers, then the 93xx are a far better value. If you want to use more than 8x NVMe drives and can find a backplane with PCIe switches that specifically works with the Broadcom 94xx and 95xx cards, then they are OK at that.

 

[Fastline]

He means they aren't HBAs, so they aren't good with TrueNAS.

Umm, the 93XX or 94XX series?

But, the 94xx series are "TriMode" (SATA/SAS/NVMe), and it turns out this means they aren't really good at any of them.

Wait, is this for real? Not even good at SATA?

Above, you mentioned that the 93XX have a limit up to 256 drives. So my question is are those in regard of SAS or SATA HDDs?

I just checked the 9400-16i datasheet and it says up to 1024 SAS/SATA devices.

If you only want to use SATA or SAS in large numbers, then the 93xx are a far better value.

If someone wants to use like 4 drives per port on a card that has 4 ports, for e.g. 9400-16i, is that good when compared to 93XX cards? Considering the factor that one does not have a need to connect SATA/SAS expanders to connect tons of drives?

I'm worried if i bought the wrong card ;(

 

[Etorix]

Umm, the 93XX or 94XX series?

@Ericloewe specifically commented on the 943x and 953x.
But the whole point of the 9400 and later series is for Broadcom to maintain its lucrative, and captive, storage market by throwing NVMe drives under the SCSI bus. And the implementation involved merging HBA functions with RAID controller functions to have one unified driver, as well as inventing U.3 backplanes which, in order to spare a pair of wires, have created an incompatibility with U.2 drives. Bad moves!

Wait, is this for real? Not even good at SATA?

Maybe not, but the new driver does not have the same degree of validation with ZFS as good old mpr. Hard-core ZFS users are very cautious and very conservative (storage technology-wise). Proud paranoids!

So the advice is to stick with 9300 HBAs for SAS/SATA drives, and to go for genuine PCIe switches and/or U.2 (not U.3!) backplanes for NVMe and to NOT enter the "TriMode" ecosystem.

Since you have the 9400, you may try and use it as a SAS HBA. But you'd have been just as happy with a 9300-8i (and expander) or a 9305-16i/24i. Expanders are cheap compared with -16i HBAs or piling up HBAs.

 

[Fastline]

But the whole point of the 9400 and later series is for Broadcom to maintain its lucrative, and captive, storage market by throwing NVMe drives under the SCSI bus.

Yeah, that's sad ;(

And the implementation involved merging HBA functions with RAID controller functions to have one unified driver, as well as inventing U.3 backplanes which, in order to spare a pair of wires, have created an incompatibility with U.2 drives. Bad moves!

Wow. I was not aware that it also has such incompatibility issues.

Maybe not, but the new driver does not have the same degree of validation with ZFS as good old mpr Hard-core ZFS users are very cautious and very conservative (storage technology-wise). Proud paranoids!

Holy shit. What did i just read. I went for 9400-16i cause i got to know that 9305 has two chips which consume more power and generate lot of heat. Is that same with the 9300-16i too? Just checked the LSI 9300 and as well as 9305 and both have dual controllers and if we talk about the 16i version, both requires an external power.

However, i did noticed that none of these 93XX Cards have the Tri Mode function. Were 94XX were the first card from Broadcom to have the TriMode feature?

So the advice is to stick with 9300 HBAs for SAS/SATA drives, and to go for genuine PCIe switches and/or U.2 (not U.3!) backplanes for NVMe and to NOT enter the "TriMode" ecosystem.

I do get that fully and like i have always said, i prefer the PLX Cards over these fancy Tri-mode Adapters which adds an extra latency as compared to the PLX ones. But, what if one has to not use the NVMe with the newer cards like 9400-16i?

Also, I'm quite sure that you're already aware of this fact, but would like to bring your attention to the fact that the newer cards like 9400-16i has two firmware available which are "Mixed Profile" which is for SAS/SATA (HDD/SSD) + NVMe (U.2) and the "SAS_SATA Profile" which is for SAS/SATA (HDD/SSD) device only. Is this implementation/workaround still fine with SAS/SATA HDD? Or still one needs to use the LSI 9300 HBAs to have the validation thingy you mentioned above?

Also, on a side note, i would like to ask whether it is safe and practical to use SATA SSDs with the 9300/9305 and 9400 HBAs.

Since you have the 9400, you may try and use it as a SAS HBA. But you'd have been just as happy with a 9300-8i (and expander) or a 9305-16i/24i. Expanders are cheap compared with -16i HBAs or piling up HBAs.

Umm, i know. But i still have the option to return the card. What do you suggest at this point? Return my 9400-16i and get 9300-16i/9305-16i? The Card is as it was delivered. I have just checked the package contents as I'm still collecting the parts and researching on things.

 

[Etorix]

I went for 9400-16i cause i got to know that 9305 has two chips which consume more power and generate lot of heat. Is that same with the 9300-16i too? Just checked the LSI 9300 and as well as 9305 and both have dual controllers and if we talk about the 16i version, both requires an external power.

9300-16i is two LSI3008 controllers (9300-8i) and a switch; the whole lot might draw a little over what the PCIe slot can deliver.
9305-16i is a single LSI3216 controller and feeds off the PCIe slot.

Also, on a side note, i would like to ask whether it is safe and practical to use SATA SSDs with the 9300/9305 and 9400 HBAs.

I hope it is safe, but "practical" is a lost cause. SATA SSDs are a dying breed while you can buy 30 TB NVMe drives (if you have deep pockets…).
I have a SSD pool of second-hand 3.84 TB enterprise drives. It's working to satisfaction—the first point being that it's totally silent—but it has little future. I might, or not, find an opportunity to upgrade to 7.68 TB drives at some point but it's ultimately a dead end with no realistic prospect to grow in capacity at will or even to remain maintanable beyond the life time of the current drives. Hopefully, the drives will hold long enough that there will be a fair supply of refurbished >10 TB U.2 drives.

 

[Fastline]

9300-16i is two LSI3008 controllers (9300-8i) and a switch; the whole lot might draw a little over what the PCIe slot can deliver.
9305-16i is a single LSI3216 controller and feeds off the PCIe slot.

Thank you for clearing the confusion i had. This is what i was going to ask next. xD

I hope it is safe, but "practical" is a lost cause. SATA SSDs are a dying breed while you can buy 30 TB NVMe drives (if you have deep pockets…).
I have a SSD pool of second-hand 3.84 TB enterprise drives. It's working to satisfaction—the first point being that it's totally silent—but it has little future. I might, or not, find an opportunity to upgrade to 7.68 TB drives at some point but it's ultimately a dead end with no realistic prospect to grow in capacity at will or even to remain maintanable beyond the life time of the current drives. Hopefully, the drives will hold long enough that there will be a fair supply of refurbished >10 TB U.2 drives.

I totally get your point and yeah, what you said does makes sense.

However, for SATA SSDs connected to HBAs like 9305-24i and/or 9400-16i, would all the SATA SSDs connected to these HBA will perform at its max speed and will not bottleneck? Disregarding the other factors such as CPU and Network for the time being.

Also, I'm quite sure that you're already aware of this fact, but would like to bring your attention to the fact that the newer cards like 9400-16i has two firmware available which are "Mixed Profile" which is for SAS/SATA (HDD/SSD) + NVMe (U.2) and the "SAS_SATA Profile" which is for SAS/SATA (HDD/SSD) device only. Is this implementation/workaround still fine with SAS/SATA HDD? Or still one needs to use the LSI 9300 HBAs to have the validation thingy you mentioned above?

Would like to get your opinion on this. Your two cents will help me out here.

Also, i was looking at 9305-16i and i found that it says MegaRAID. Why is that so? Is this HBA no good?

I would like to repeat it again that i still have the option to return the card. What do you suggest at this point? Return my 9400-16i and get 9305-16i? (No 9300-16i as it heats up a lot and requires extra power). The Card is as it was delivered. I have just checked the package contents as I'm still collecting the parts and researching on things.

 

[nabsltd]

Also, on a side note, i would like to ask whether it is safe and practical to use SATA SSDs with the 9300/9305 and 9400 HBAs.

I use a mirrored pair of SSDs connected to a 9300 as my boot drive, but all other SSDs I use are NVMe (not connected via any LSI hardware), for all the reasons that @Etorix gave.

 

[Fastline]

I use a mirrored pair of SSDs connected to a 9300 as my boot drive, but all other SSDs I use are NVMe (not connected via any LSI hardware), for all the reasons that @Etorix gave.

Of course, of course. Like i have always said, i prefer the PLX Cards to connect NVMe drives over these fancy Tri-mode Adapters which adds an extra latency as compared to the PLX ones. That's one of the factor i know and hence i am avoiding it. What I'm confused is if i have got the wrong HBA i.e. 9400-16i ad whether should i use 9305 for all the reasons @Etorix mentioned.

 

[Etorix]

You did not get the cheapest HBA, that's for sure. As to how suitable or desirable the 9400 is, in default TriMode or with the SAS/SATA-only firmware, I leave it for others to comment. (Maybe @Ericloewe can come back to this?)

 

[Ericloewe]

I don't expect the 9400 or 9500 cards, in SATA/SAS mode, to be any worse than the 9300s. Possibly a different set of firmware issues, due to the different versions involved and the craziness that goes on in the driver.
I can see the NVMe support mode complicating things a bit, but I would not go out of my way to avoid it, in the absence of solid data. It's not like a few extra lines of code make a difference when you're waiting a millisecond or ten for the disk to seek anyway.

And SSDs... NVMe all the way, without the tri-mode nonsense, so that leaves only spinning rust attached via SAS. Maybe tape, if you're one of those people who think that 200 IOPS is two or four orders of magnitude higher than anyone really needs for cold storage.

 

[Fastline]

I don't expect the 9400 or 9500 cards, in SATA/SAS mode, to be any worse than the 9300s. Possibly a different set of firmware issues, due to the different versions involved and the craziness that goes on in the driver.

I'm so much in relief now. I was so worried whether if i had got the wrong set of cards.

I can see the NVMe support mode complicating things a bit, but I would not go out of my way to avoid it, in the absence of solid data. It's not like a few extra lines of code make a difference when you're waiting a millisecond or ten for the disk to seek anyway.

Yes, implementing the TriMode thingy (NVMe) has indeed complicated things a bit.

And SSDs... NVMe all the way, without the tri-mode nonsense, so that leaves only spinning rust attached via SAS.

Yes, for the NVMes, its suggested to use the PCIe switch based Cards right for better performance and low latency right?

Maybe tape, if you're one of those people who think that 200 IOPS is two or four orders of magnitude higher than anyone really needs for cold storage.

Oh, well, i saw the datasheet and i saw tape disks.

Also, i had a question that Broadcom is a manufacturer and are they not aware that the TriMode Adapters will add latency when utilizing the U.2 NVMe drives. So, why they even did that if it does not work well? They could have continued to manufacture the PCIe Switch based cards for the NVMe. To give comfort, there is a downside.

In addition, i would like to know how much difference is between the IOPS/Speed/Throughput and latency for the same SSD, same hardware but with one key difference which is one hand it uses the TriMode Adapter and the other hand it uses the PCIe Switch based Card.

Now that i have got this card and as you have clarified the fact that it should be okay with the SAS/SATA device, i would like to know that in the future if i have to build another NAS, which HBA Card do you guys suggest? 9305 or 9400?

@Ericloewe Thanks for your input!

 

[Ericloewe]

Also, i had a question that Broadcom is a manufacturer and are they not aware that the TriMode Adapters will add latency when utilizing the U.2 NVMe drives. So, why they even did that if it does not work well?

I fully believe that it is to protect the lucrative SAS business at all costs. PCIe switches have long been widespread, just not high-performance ones for many lanes. A lot of manufacturers could cash in on that and eat into Broadcom's margins.
SAS meanwhile, has two players: Broadcom and Microchip. The latter probably a step behind in terms of SATA/SAS gear, with LSI/Avago/Broadcom sweeping up two of the big three OEMs (HPE uses both LSI and what is now Microchip, Dell and Lenovo used LSI pretty much exclusively until recently) and much of the whitebox market. That's a lot of servers.

Let's examine a typical server you may have bought circa 2015, a Dell R630 10-bay. That thing needs one HBA (customized for Dell) and one expander (integrated with the backplane). Retail pricing for a typical RAID controller plus the expander might have been somewhere in the 500-600 buck range, say a third of that is what Broadcom charged Dell for the parts. That's a pretty juicy piece of the action, and there was little that could be done to get away (maybe use fewer disks, and SATA only, but neither of those options were great for a variety of reasons). This R630 had four U.2 tri-mode bays, which was cutting edge at the time and not something you would use willy-nilly.

Fast forward a few years, and in early 2020 you could buy a Dell R6515, same 10-bay form factor. It's still sold with an HBA for SAS, but here's the kicker: Only 8 SATA/SAS disks are supported. They cut the expander. Two bays only do NVMe. Instant savings for everyone and instant sad face for Broadcom execs. And you can even buy it without SAS at all, which is even worse. This is possible because U.2 takes the "we'll tack it onto the side" approach that SAS took with SATA - it's entirely backward-compatible. Just like you could hook up a second SAS port, you can now hook up PCIe and it's independent of the SAS/SATA pins.

Clearly, this scenario was bad for Broadcom and Microchip, so they came up with U.3. Ostensibly, it allows for cheaper backplane PCBs by virtue of needing fewer high-speed differential pairs (by re-using the four existing ones that support the two SAS lanes), more or less halving the count of differential pairs per bay (the clock situation may vary a bit, so the exact count depends on the product). The catch is that the only way it can work is with an expensive tri-mode expander - you could throw away three of four ports per SATA/SAS device by connecting a controller directly to a bay, but then Broadcom can sell you a monster -32i card just so you can have 8 tri-mode bays. It also means server vendors can't later decide to cut out SAS and still offer SATA from the PCH or SoC, like they can with U.2 - they're stuck with SAS if they want to keep tri-mode U.3 bays.

tl;dr:

1. Tri-mode expanders are more expensive than simple PCIe switches;
2. U.3, tri-mode HBAs and tri-mode expanders reinforce each other, they make little sense without the other two
3. For Broadcom, SAS is a "money printer go BRRRRR" product line, with likely high margins and widespread adoption