Resource icon

What's all the noise about HBA's, and why can't I use a RAID controller?

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are.

1) An HBA is a Host Bus Adapter.

This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort, and even in "JBOD" or "HBA mode" generally hide the physical device. If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA. If you cannot get the output of "smartctl" for a device, you DO NOT HAVE A TRUE HBA. A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.

2) FreeBSD has incredibly robust support for the LSI HBA's.

FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The FreeNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.

3) You must crossflash to IT/IR firmware

If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT-20.00.07.00 HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this is probably no longer relevant.

The LSI 9240 (etc) default MFI firmware is apparently being sold on eBay as "IR" by clueless sellers. The MFI firmware is unsuitable for FreeNAS and may cause your pool to get eaten.

The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware 20.00.07.00.

The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware ... 13? I don't have one of these online right now so my appreciation to whoever follows up with the current correct answer.

4) FreeBSD may or may not have good support for other HBA's/RAID controllers.

FreeBSD does support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.

It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.

If you have come to FreeNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.

5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.

In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.

6) A RAID controller with write cache is particularly bad.

A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.

For more information on related topics, please see:

Confused about that LSI card? Join the crowd...

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA
 
Last edited:

Ctiger256

Dabbler
Joined
Nov 18, 2016
Messages
40
On the firmware question, here's what I have on my 3008
Code:
sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  16.00.01.00    0e.01.30.28    08.37.00.00     00:01:00:00

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.


(Shameless plug that it would be good if we could further troubleshoot whether there is a regression with this driver in 11.3)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours.
Also because LSI cares. That's more than you can say for much of the storage controller market, which I guess is part of why it's down to LSI - although I hear Microchip is investing in storage controllers now?

The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware ... 13? I don't have one of these online right now so my appreciation to whoever follows up with the current correct answer.
The official party line is that we should use "the latest firmware". I guess it's better than the old "driver version minus one, but then careful with the firmware rev, but the driver gets updated without bumping the version number to denote compatibility with firmware n by being at version n+1" nonsense.

A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.
There is nothing quite as irritating as opening an SSH session and being told by smartctl "Sorry, I have no idea what kind of disk this is".
 

Tigersharke

BOfH in User's clothing
Administrator
Moderator
Joined
May 18, 2016
Messages
890
I do not question whether one raid card is better than another, or that the strong recommendation to use LSI 9240-8i, for example, is valid. Let me also preface this with the admission that I am not running FreeNAS but that I would like to have the best benefit of ZFS as possible in a multi-HDD pool. Right now I am not using best practices as described, though I am not (specifically or intentionally) using any raid capability of my "sata expansion" card. Supposing that I were to buy an LSI 9240-8i, card, how would I attach the SATA drives to it? I see various adapters and cable configurations on Amazon but that is definitely not the go-to site for technical questions. My motherboard has only 8 SATA connectors, though two are 'AS Media' controlled, and I have nine HDD, two SSD cache drives plus a DVD drive for a total of 12 SATA devices. Certainly the dvd drive need not be placed anywhere special, it might be on the 'AS Media' sata port.

What is the best practice recommendation for LSI 9240-8i connections to SATA devices?

How many drives can one LSI 9240-8i support, is it 8 or does the SAS spec permit more?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I do not question whether one raid card is better than another, or that the strong recommendation to use LSI 9240-8i, for example, is valid. Let me also preface this with the admission that I am not running FreeNAS but that I would like to have the best benefit of ZFS as possible in a multi-HDD pool. Right now I am not using best practices as described, though I am not (specifically or intentionally) using any raid capability of my "sata expansion" card. Supposing that I were to buy an LSI 9240-8i, card, how would I attach the SATA drives to it? I see various adapters and cable configurations on Amazon but that is definitely not the go-to site for technical questions. My motherboard has only 8 SATA connectors, though two are 'AS Media' controlled, and I have nine HDD, two SSD cache drives plus a DVD drive for a total of 12 SATA devices. Certainly the dvd drive need not be placed anywhere special, it might be on the 'AS Media' sata port.

What is the best practice recommendation for LSI 9240-8i connections to SATA devices?

How many drives can one LSI 9240-8i support, is it 8 or does the SAS spec permit more?

Seriously, look around the forums ;-)

https://www.ixsystems.com/community...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

An "8i" card indicates that it has 8 internal lanes. This can directly handle 8 drives (with the correct cabling) or potentially more than a hundred drives (with an SAS expander). You would need a SATA breakout cable to hook up your drives if you went the direct route.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
As an aside that is very irrelevant to any realistic scenario... Do LSI HBAs play nice with ATAPI devices?
As a tangent to my tangent (which I guess is the same tangent, by definition of tangent), are there non-optical ATAPI devices? Is there some obscure piece of hardware out there that uses ATAPI and isn't an optical disk drive?
 

guemi

Dabbler
Joined
Apr 16, 2020
Messages
48
Worth adding, because I've seen a lot of people regarding the MegaRAID 9xxx series.

Much like other people, we unfortunately bought out raidcard before deciding on FreeNAS and checking HW requirements, and a 1200 USD card isn't fun to throw out so we wanted to try as much as we could before giving up and:


We had similar issues getting MegaRAID 9460 to work (Lenovo 930-24i to be exact) despite putting it in "JBOD MODE".

What we had to do was not only put the raidcard in JBOD, but also go on every individual drive and configure it from "disk" to "JBOD" - otherwise it will still go through the raidcard server.


I hope it helps someone. :)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Worth adding, because I've seen a lot of people regarding the MegaRAID 9xxx series.

Much like other people, we unfortunately bought out raidcard before deciding on FreeNAS and checking HW requirements, and a 1200 USD card isn't fun to throw out so we wanted to try as much as we could before giving up and:

We had similar issues getting MegaRAID 9460 to work (Lenovo 930-24i to be exact) despite putting it in "JBOD MODE".

What we had to do was not only put the raidcard in JBOD, but also go on every individual drive and configure it from "disk" to "JBOD" - otherwise it will still go through the raidcard server.

I hope it helps someone. :)

But I suspect you're still using the MFI driver, which will eventually find a way to bite you, I think. It's really only MPR/MPS based cards that are known to be good.
 

guemi

Dabbler
Joined
Apr 16, 2020
Messages
48
But I suspect you're still using the MFI driver, which will eventually find a way to bite you, I think. It's really only MPR/MPS based cards that are known to be good.

Most likely, it found the raidcard out-of-box (11.3-u2) and I'm aware it might not be the most secure solution, but I went with a secure zfs-level and will be taking backups to our other NAS more often. It's better than Windows Server and better than scrapping the raid card and buying a new one, for us.

I'll leave decisions up to others, but at the very least if they're in the same boat as me with an already bought card, they should at least be told how to get it working in my humble opinion.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'll leave decisions up to others, but at the very least if they're in the same boat as me with an already bought card, they should at least be told how to get it working in my humble opinion.

We generally do not do that here, usually with big red flashy warning signs, because it can lead to data corruption, data loss, or even pool loss.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
But I suspect you're still using the MFI driver, which will eventually find a way to bite you, I think. It's really only MPR/MPS based cards that are known to be good.
I think mrsas is the default for devices that support it, which is a step or five up from mfi. Still less than ideal, but at least it's not the same sort of trainwreck.
 
Joined
Dec 29, 2014
Messages
1,135
I think mrsas is the default for devices that support it, which is a step or five up from mfi. Still less than ideal, but at least it's not the same sort of trainwreck.
mrsas is definitely what is selected for a MegaRAID 9271. I discovered that while trying to do a proof of concept and pry ~=$100 for a 9207-8i out of a customer. :smile:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeah, sorry, guys, I meant MRSAS, things just stick in my head. Used to be MFI.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,599
As an aside that is very irrelevant to any realistic scenario... Do LSI HBAs play nice with ATAPI devices?
As a tangent to my tangent (which I guess is the same tangent, by definition of tangent), are there non-optical ATAPI devices? Is there some obscure piece of hardware out there that uses ATAPI and isn't an optical disk drive?
When I started transcoding my DVD collection, I threw together more DVD drives. My desktop had 1 internal DVD drive and I added 1 USB DVD drive.

But, to get more discs going, I added my Sun x2200 M2 server into the mix. It had;
  • LSI 4i4e SAS
  • LSI 2Gbps FC
So I installed my eSATA DVD drive on a SAS port, (with the proper cable, it turned into an eSATA port).
Then put together my IDE DVD drive, with a IDE to SCSI adapter. Which then went to a SCSI to FC bridge. That bridge was connected to my Brocade FC SAN switch. And the server's LSI FC card connected to Brocade FC switch.

Believe it or not, both DVD drives in that Frankenstien setup ran at their top speeds. Had me running around changing discs for days.

In the distant past I do belive there were non-optical ATAPI devices. I want to say they were tape or perhaps those removable media drives, (Zip drives?).
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
So I installed my eSATA DVD drive on a SAS port, (with the proper cable, it turned into an eSATA port).
Cool, the weird ATAPI edge case works. Wait, why does the paragraph go on...
Then put together my IDE DVD drive, with a IDE to SCSI adapter. Which then went to a SCSI to FC bridge. That bridge was connected to my Brocade FC SAN switch. And the server's LSI FC card connected to Brocade FC switch.
Holy crap.
 

boggit

Cadet
Joined
May 9, 2020
Messages
4
Can an absolute newbie ask a simple question, please?
I've tried to understand this thread - would a Broadcom 8 Port 6Gbps SAS 9207-8i SGL PCI-E Host Bus Adaptor flashed to IT/IR firmware be a sensible purchase for ZFS?
TIA - Martin
 
Joined
Dec 29, 2014
Messages
1,135
would a Broadcom 8 Port 6Gbps SAS 9207-8i SGL PCI-E Host Bus Adaptor flashed to IT/IR firmware be a sensible purchase for ZFS?
Yes. I have them in several FreeNAS systems and they work perfectly.
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
I just recently crossflashed an IBM M1015 to LSI9211-8i IT mode .. took a little Google-Fu to find the right files that finally worked, but working great now. :cool:
 

Miroman

Cadet
Joined
Oct 28, 2020
Messages
9
I want to install Freenas on this server HP ML 110 G7. I want to buy 2 WD Red 2 TB. What RAID controller to buy?
 
Top