HDD's connected to HBA don't show up

[NugentS]

I use RAID cards with ESXi. Actually on the ESXi server itself as emergency storage to swap VM's to if I need to take down the main storage. Whilst I also have secondary storage its very slow (runs like treacle) as its pure HDD in a RAID 6 (non ZFS) configuration. The H710 in the ESXi boxes runs about 2TB useable of SSD

 

[ClassicGOD]

but people must be desperate to sell these if they're willing to lobotomize a relatively decent hardware RAID controller and make it into a mere HBA.

No one is lobotomizing anything. Those cards are usually just re-branded LSI cards and LSI usually offers it's cards in 2 flavors - RAID and HBA. Those are identical hardware, the same controller etc - the only difference is firmware. To the point that if you download firmware package for card like LSI 9210-8i it contains both IR (raid) and IT (hba) firmware. It's just an option provided by the manufacturer.

The entire cross-flashing procedure is only required because re-branded cards often come with different hardware IDs and not the best firmware support.

 

[jgreco]

No one is lobotomizing anything. Those cards are usually just re-branded LSI cards and LSI usually offers it's cards in 2 flavors - RAID and HBA. Those are identical hardware, the same controller etc - the only difference is firmware. To the point that if you download firmware package for card like LSI 9210-8i it contains both IR (raid) and IT (hba) firmware. It's just an option provided by the manufacturer.

This is incorrect and is a complete misunderstanding of the LSI ecosystem.

LSI has three fundamental product lines: HBA, low end RAID, and high end RAID. For the remainder of this conversation, we'll focus on the 6Gbps product line. The 12Gbps product line details are slightly different, but differentiated in a similar manner.

The HBA's are only ever HBA's.

The low end 2008 based RAID cards, such as the 9240-8i, can be optionally crossflashed to HBA mode. This is supported by LSI, and is the normal "retail" option for people wanting an LSI 9211-8i style HBA, which are generally not sold in the retail channel.

The LSI HBA cards, including the low end 2008 RAID cards when crossflashed, run the LSI IR or IT firmware, both of which are operative under the FreeBSD mps driver, which is the thing we want to see for FreeNAS. Vendor provided cards may run customized variations of the firmware that is not a good match for FreeNAS. Dell does this, for example.

The LSI low end 2008 RAID cards with their default MFI firmware run under the MFI driver. This is highly undesirable for FreeNAS and is known to cause problems. The LSI low end 2308 RAID cards can be run as MFI or MRSAS, depending on fiddly details that aren't really relevant here.

The LSI high end RAID cards also typically run under MFI, with newer ones being able to run under MRSAS as well. LSI has never supported crossflashing of these to HBA mode. As I previously mentioned,

The RAID cards and the HBA's share a common compute platform

they do share a common compute platform, but the high end RAID cards have support for cache, battery or supercap, and cachevault modules, none of which are present on the lower end cards. They also have better CPU than the low end RAID cards, as outlined in their specifications. None of these cards are shipped with IR or IT firmware, a fact which seems to have escaped you. They all come with the MFI/MRSAS firmware, with no downgrade option.

Using one of these high-end RAID cards as a mere HBA is basically lobotomizing a decent high end RAID controller, leaving the cache, battery, and cachevault hardware unused.

If you really think that the only difference between all these cards is firmware, I'll be happy to send you a 9211-8i, and I will pay you the full retail price for a 9271CV-8i when you return it to me after you have flashed it with "the firmware" that transforms it into a functional 9271CV-8i. That's a good deal, which, of course, you can't do, because unlike what you claim, it isn't just different firmware.

 

[ClassicGOD]

I freely admit my lack of knowledge about LSI higher end offering. I also admit that I may misunderstand LSI offering as it's acient history for some and I'm willing to learn. I've seen LSI 9211-8i offered as both HBA and RAID card. And the official firmware package on Broadcom website has both IR and IT firmwares. I assumed that it was offered in both flavors. Similarly with many other models.

I never said you can crossflash a card to a totally different model - so your insinuation on flashing low end card to high end one seems a bit hostile. Also is not using every hardware feature and disabling not used ones always a bad thing?

 

[jgreco]

And the official firmware package on Broadcom website has both IR and IT firmwares.

According to LSI,

https://docs.broadcom.com/doc/12352167

"Integrated RAID (IR) is a processor-based RAID solution designed for system environments requiring redundancy and high availability where a full-featured RAID implementation is not desired" and specifically does not include RAID5, but, rather, just RAID1 and striping variants. They further note "LSI can offer the Integrated RAID solution at a lower cost than a full-featured RAID implementation."

So, while it does have both IR and IT firmware, it isn't really what anyone today would consider a RAID card, it can only do mirroring and striping.

I freely admit my lack of knowledge about LSI higher end offering.

Then, perhaps, don't correct somebody who has written extensively about the topic?

And the official firmware package on Broadcom website has both IR and IT firmwares. I assumed that it was offered in both flavors. Similarly with many other models.

Actually, really not. Basically just the 9240 and 9340.

I never said you can crossflash a card to a totally different model

Okay, so you did not say

LSI usually offers it's cards in 2 flavors - RAID and HBA. Those are identical hardware, the same controller etc - the only difference is firmware.

Which is categorically false, yet you chose to try to correct me when I said that this was lobotomizing a high end RAID controller.

And you also said

The entire cross-flashing procedure is only required because re-branded cards often come with different hardware IDs and not the best firmware support.

Which is also misleading at best, because the actual issue is that the device driver in TrueNAS has to match the firmware on the board, and the only supported firmware is LSI 20.00.07.00. It's not a function of "not the best firmware support", because some vendors, including Supermicro, actually do provide it.

But the real issue here is

Also is not using every hardware feature and disabling not used ones always a bad thing?

this; high end controllers are generally not lobotomized into HBA's. Historically, it has not been done, because it is potentially dangerous, especially if there are batteries involved. Lobotomizing the code that monitors the state of the battery could result in severe consequences (Patrick Kennedy's exploding RAID batteries), and the high end controllers are often more heavily customized to work with features in a given vendor's chassis. People didn't used to do it because there was no point; a 9271CV-8i, for example, used to cost half a dozen times what a 9211-8i type card cost. But if there are cheap 2208 based boards on the market, and they don't have anything hazardous like batteries, there's nothing technically except some firmware hackery to prevent them from being lobotomized into 9211-8i's. Someone in the last few years seems to have worked out the details of actually doing this, so, yay, great.

The problem for us here on the forum is that it makes it MUCH harder to figure out when someone comes to the forum with a card that wouldn't normally be running IT firmware but only officially supports MFI/MRSAS. Then, we have someone like the OP in this thread, who said he had bought a 9211-8i, turns out to have bought an IBM OEM 9265-8i, which is a high end RAID controller with no official path to IT mode. For a decade, we've had people showing up with LSI RAID controllers, and we've had to warn them off.

So when I say "fine ok if it's really been flashed to IT, but I don't know why someone would lobotomize a nice RAID card" and then you decide to take exception to that, well, here we are.

seems a bit hostile

I think it's a bit hostile of you to correct me with incorrectness. But that's just me. It's a detail oriented business.

 

[ClassicGOD]

Then, perhaps, don't correct somebody who has written extensively about the topic?
[...]
I think it's a bit hostile of you to correct me with incorrectness. But that's just me. It's a detail oriented business.

I misread your post as someone that is confused about the entire flashing process. I had no way of knowing about your prior experience and I was in no way hostile - just providing information. Maybe not completely correct information but not attacking anyone, just trying to help.

 

[gyorfitam]

The problem for us here on the forum is that it makes it MUCH harder to figure out when someone comes to the forum with a card that wouldn't normally be running IT firmware but only officially supports MFI/MRSAS. Then, we have someone like the OP in this thread, who said he had bought a 9211-8i, turns out to have bought an IBM OEM 9265-8i, which is a high end RAID controller with no official path to IT mode. For a decade, we've had people showing up with LSI RAID controllers, and we've had to warn them off.

I was buying this with a fair amount of confidence since the seller was specifically mentioning unRAID, FreeNAS, ZFS and Host Bus Adapter in the eBay listing, seems to be specialising in NAS hardware (literally calling himself "The Wizard of NAS") and only has positive reviews (more than 300). When I was looking for an HBA, I saw his stuff coming up first, seemed reasonably priced as well (£65 for an 8-drive unit) and it's actually coming from the UK and not from China (I prefer to buy from UK sellers). I believe when I bought it, he had the IBM, a Fujitsu and maybe a third one available and basically I just chose randomly because they were seemingly the same to me for the exact same price - I don't have actual experience with server-grade stuff (or anything remotely similar) so I am still puzzling the basic pieces together.
Regardless... so far the NAS works great, I had to replace one drive though but today I finally got the new one as well so... I'm happy now :). I'm only waiting for some fans as this "Raid card castrated to HBA" is surprisingly hot. And later on 4 more drives but I don't want to talk about it, it's like a never ending story :D.

 

[jgreco]

Ah yeah, the higher clock speed on the RAID cards is an issue too.

 

[Constantin]

Funky stuff can also happen on plain HBAs. See my thread on my pool going kaploink! after upgrading to 12U7. All three (and only!) my sVDEV SSDs were no longer recognized post-upgrade, which then led the system to conclude that the pool was permanently hosed, should be destroyed, and rebuilt from backups.

Thankfully, I ignored that helpful advice, hot-swapped my sVDEV SATA SSDs around the case and all was well. But the real question is what would cause a bog-standard LSI HBA or SM motherboard to stop recognizing those SSDs were attached (and only the the three SSDs associated with the sVDEV) All other SSDs (L2ARC, SLOG, 2 SATADOMs) were fine.

 

[jgreco]

I ignored your PM on that in favor of responding here. I have occasionally seen a need for I/O systems to be power cycled. I think it is just the nature of the beast, that when you have multiple devices working in concert, sometimes things just "get out of sync" somehow. That is, after all, one of the big reasons we're so adamant about running very particular firmware versions for the HBA's. There's not really any guarantee that there aren't still edge case bugs where the HBA CPU is "left hanging" when you suddenly switch out the host driver unannounced. We just know that day-to-day operations are pretty rock solid.

I know this isn't a comfortable answer. Given enough time and incentive, it would probably be a debuggable problem, but the old IT Crowd jewel applies even to servers.

 

[Constantin]

I guess power cycling should then be added to the standard advice for any sVDEV-using pool having disconnected. Note: restarting was not enough, the drives had to have their power fully interrupted.

I agree it’s not a comfortable answer as this sort of thing shouldn’t happen without the system throwing an error when three sVDEV drives go offline simultaneously. But of course the system cannot throw an error if the issue arises as it is being reset as opposed to when it’s operating normally.

For all we know, it’s some sort of BIOS issue. But the drives were not there anymore as far as the UI or CLI were concerned until I power cycled each one of them by pulling their sleds out of the backplanes and reinserting them.

I’d appreciate better help from the UI though when it comes to troubleshooting a disconnected pool. Presumably the pool import dataset has a list of drives it expects to import… once a import fails, listing which drives are missing by SN, type, and capacity would go a long way towards speeding recovery efforts.

 

[jgreco]

I’d appreciate better help from the UI though when it comes to troubleshooting a disconnected pool.

That sounds like an ideal suggestion to launch at iX in Jira, IMHO.

 

[Constantin]

That sounds like an ideal suggestion to launch at iX in Jira, IMHO.

Your wish is my command

 

[Ericloewe]

Interesting to see the 2208s being crossflashed. I'd seen that for the Dell Gen 12 proprietary cards (Gen 13 introduced HBAs as an option and there's extra feedstock for straightforward crossflashes in the form of H330s), but it was finicky and prone to bricking cards if not done carefully.

 

[jgreco]

Interesting to see the 2208s being crossflashed.

It's probably not really that shocking.

Years ago, it was the glut of M1015's, H200's, H310's, etc., at lease-end that drove some enterprising hackers with more time and skill than money to figure out what was going on in those cards and create the proper sequence of steps to lobotomize them into 9211's. They ended up with cheap HBA's.

Those have been somewhat more scarce in recent years. It's long been known that the fancier cards were just feature-plus versions of the basic design. For a long time, it seemed like there wasn't a lot of pressure to go to 12G SAS or beyond, since HDD storage was acceptable on 6G, and SSD is basically headed to NVMe. So we never really saw a heavy push towards upgrading pricey high end 6Gbps RAID controllers.

However, with the Linux driver deprecation in ESXi 7.0, and the retirement of 6.5 and 6.7 in less than a year, I suspect there's a lot of pressure to deprecate old ESXi gear, and that may be where some of this is coming from. No one really wants these controllers anymore for their original purpose, but since they are the LSI platform, they can be bricked down to an HBA if you have the right tools.

The part about that which I find concerning is basically the battery or supercap tied to many of these cards, along with some experience along the way with "special" versions of these cards which may not really be compatible with generic firmware.

 

[Ericloewe]

I have to make a mental note of these M5110s, they're selling for HBA prices. I have a pair of Dell Nehalem-era rackmount workstations that I'm still on the fence about upgrading, and part of that would be an upgrade from SAS1 to SAS2.
Though realistically I'll just end up migrating them to VMs and donating the hardware to projects more strapped for hardware or maybe a project I'd like to have in my debt...