Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

What's all the noise about HBAs, and why can't I use a RAID controller?

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
1) An HBA is a Host Bus Adapter.

This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort, and even in "JBOD" or "HBA mode" generally hide the physical device. If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA. If you cannot get the output of "smartctl" for a device, you DO NOT HAVE A TRUE HBA. A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.

Note that having device names in "camcontrol devlist" and getting "smartctl" results is not any sort of proof that you do actually have an HBA instead of a RAID card. It's just an easy test that weeds out a wide range of RAID cards.

2) FreeBSD has incredibly robust support for the LSI HBA's.

FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The FreeNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.

3) You must crossflash to IT/IR firmware

If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT- HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this may no longer relevant.

The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware

The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware or the special available via iXsystems.

4) FreeBSD may or may not have good support for other HBA's/RAID controllers.

FreeBSD does support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.

It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.

If you have come to FreeNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.

5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.

In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.

6) A RAID controller with write cache is particularly bad.

A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.

7) Do not fear the HBA.

For many SOHO, power users, home users, hobbyists, etc., the HBA may be the first exposure to unfamiliar server hardware and different kinds of cabling. It is simply a server-grade version of a SATA add-on card. The cabling is designed for high density inside servers, but you can buy break-out cables to transform SFF8087 to SATA connectors. It can support SAS and SAS expanders. The forums have a nice resource to bring you up to speed on SAS, and it's really quite nice to work with.

8) Fear the HBA.

There's one caveat. Keep your HBA cool. It is an embedded computer and throws off about 10 watts. Failure to have airflow directed over your HBA can cause overheat, and in extreme cases LSI HBA's have been found to vomit random bits all over, which isn't good for ZFS.

9) The only exception.

Because LSI uses a similar MIPS CPU and software design on all its cards, people long ago learned to crossflash LSI 9240 low end RAID cards and OEM low end RAID cards into SAS2008 or SAS2308 IT-mode HBA cards. It is the software running on the card and the driver in use in FreeNAS that matters here.

It is theoretically possible to turn the LSI high end RAID cards into IT-mode HBA's as well, but this is veering off into "that might not be a good idea"-land, because the high end RAID cards have additional components such as cache, flash, and battery circuits that the IT firmware doesn't expect to be there. If you can figure out the black magic to crossflash one of these to IT-mode and it actually works (be sure to test extensively!), then from the perspective of this article, congratulations, you have an HBA, and it's actually kinda likely to work correctly with FreeNAS. However, there might have been software components in the RAID firmware that controlled things like battery charging, so I'd suggest removing batteries or supercaps to reduce the chance of some unexpected issue that might result in a battery fire. Changing the card out for one that's known to work correctly is still a better option.

2021/12/29: Update for LSI high end RAID cards:
@bartuc dug up this link for the ServeRAID M5110/H1110 cards. I have not tried it. If it works for you, be sure to go on over to this thread and say thanks.
First release
Last update
5.00 star(s) 1 ratings

More resources from jgreco

Latest reviews

Very descriptive and helpful.