SAS controller compatibility

[LP1]

Hi all,

I am new to TrueNAS and wanted to check if my hardware would be compatible to run it, and if not - what would be things which I must change before it works. Specifically, I want to change as little as possible, so it should be something that absolutely won't work before changing it. To be fair I am kind of getting annoyed with how specific the controller has to be for it to work...

I am sure this question has been asked many times, so sorry if it sounds repetitive, but I have seen a lot of "recommended hardware" threads that are from 2012 or 2016 etc. and I am sure that by now that advice may be outdated. Besides I am not looking to get the best possible hardware in each slot, merely the hardware which works at the lowest possible price.

So, to start with - I already have the following and I ideally I don't want to change any key components:

System 1#
MB Asus KGPE-D16
CPU 2x6386 SE
RAM 128GB ECC REG (8GBx16)
PSU 1000W DELL H1000E-00

System 2#
MB Asus Maximus III Extreme
CPU Xeon X3480
RAM 32GB (8GBx4)
PSU 860W Corsair AX860i

I have purchased 10x4TB SEAGATE SAS drives (ST4000NM0023) - my end goal here is a relatively fast array of 8x4TB with 2 cold spares, or maybe I get another 6 drives and make it 16x4TB.
As well I have a bunch of SSDs for SLOG and OS, or to aid configuration where SSD may be useful - old 60GB/120GB SSDs of various brands, 250/500GB Samsung 840s, 860s Pro/EVOs, some M2 SSDs.

Initially, I wanted to build on System 1# using ASUS PIKE, but then I realised that I would need either SATA to SAS cable (expensive), or SATA drives (even more expensive) and in the end ASUS PIKE may conflict with TrueNAS and it is hard to flash into IT mode. At this point, I was not yet decided if I want to go for RAID, unRAID or TrueNAS, so just grabbed myself the cheapest SAS adapter which could connect 8 drives and in hindsight probably made not the best choice (PERC H700) and for time being just use System 2# to play around with it until I find the configuration which I am happy with. Like that is not enough I picked an integrated version over an adapter version of H700 - it could be flashed as LSI 9260 8i, or H700 Adapter, but it seems that would be waste of time anyway as TrueNAS won't work on it without questionable workarounds. And on top of that, I can't even find where to get the right firmware anymore.

So far my assumption is that as far as TrueNAS is concerned the main issue I have is SAS controller? I assume any SAS drives should be fine, whenever to use SDD for SLOG I can decide later, TrueNAS itself I can put on some older SSD, or in the case of System 1# MB even has a dedicated USB slot for OS. So as far as drives goes I should be ok?

What would be the best and the cheapest SAS adapter I could get, or could I still do something with ASUS PIKE or PERC H700?

At this point in time I don't have NICs, as I am not decided on 10GbE switch or whether it is going to be SFP or RJ45 connections, so will use integrated 1GbE for time being and will worry about 10GbE LAN later.

Anything else I may be missing?

 

[jgreco]

I am kind of getting annoyed with how specific the controller has to be for it to work...

That's nice, but it is what it is because of the way it all is.

Bearing in mind that you don't actually have a RAID controller, this article below still talks about "the way it is", and you will hopefully walk away with a better understanding:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Despite the thing you said in the quote above, there is absolutely ZERO controller specificity. You can use SATA, you can use SAS, some people use FC. What *is* important is that it has to be 100% solid, and that's the thing that bites you in the arse. You can get Intel AHCI SATA, or Intel PCH SCU, or LSI HBA, or certain Marvell or Asmedia chipsets, and these work 100%. Most other things have ... failings ... of one sort or another. This isn't some conspiracy where iX developers are invested in LSI stock and want you to buy brand new LSI HBA's. If you want to go out and fab your own silicon and write your own driver, and it works 100%, I will heartily endorse it and encourage people to use it. But if it's 99% ... I won't. And you don't want to use anything that's not 100% -- this is how people become sad when using ZFS.

"recommended hardware" threads that are from 2012 or 2016 etc. and I am sure that by now that advice may be outdated.

It's not; they're relevant and useful. My thread from 2013 is as relevant as ever, but it's sort of generalized. You do not need the newest blazing fast hardware for a NAS. Sandy Bridge systems are still very serviceable. Gear older than that is not recommended, mostly because it runs really hot and burns lots of watts, and at this point they're old enough you have to start expecting parts to fail soonish. You can spend a LOT of money for the latest mainboards and high end CPU's, but the fact of the matter is that you can pick up 2014's hottest CPU, the E5-2643v2, for $45 used on eBay, and it is going to be competitive with a much pricier modern equivalent.

The PERC H700 is not recommended as it is a full high end LSI RAID controller. It *may* be possible to flash IT firmware onto this.

You can use an H200 or H310, each available for about $20-$30 on the used market, crossflashed to IT mode, which is by far the cheapest and most reliable way to attach your disks. The H310 is somewhat faster but also runs a bit warmer.

 

[LP1]

Thanks,

The reason why I would like to use H700 over H200/H310... is very simple - cost. I already have H700 so it is basically free (it was only £13 anyway). whereas counter-intuitively H200 and H310 now cost closer to $80 (at least for me in UK) and most of them are already in IT mode at which point they are $100. I assume that is because people know H200/H310 could be used on TrueNAS whereas H700 mostly can't.

And this is why I said older guides are out of date - not because they are wrong in any way particular way, but because the market has moved a lot since then. I often see this exact statement "buy H200 and then sell H700 for 5 times the price" - it was probably true in 2012, but today it is exactly the opposite. Not that $100 is the end of the world, but I rather not spend it if I can.

...and I did read the link you provided (actually I have seen it before). Which in summary reads - buy H200/H310 - not H700.

Any advice on flashing IT firmware on H700? Most threads I see say it can't be flashed to IT mode, but even those who say they could are very old. As far as I can see any links in all the advice on how to do it are long dead, so unless somebody has stored it in some private place and can share I am struggling to find any firmware or tools. Or is the process the same as for H200/H310, just nobody explains it for H700, because years ago it would have been wasteful and nobody expected anyone to turn "high-end" raid card into HBA?

 

[jgreco]

The LSI controllers use a common architecture, based around the MIPS architecture, generally embedded on a RAID-on-Chip ("ROC"). Because it is much better for driver and firmware development to use a common starting point, most of their cards seem to be designed similarly, and it was discovered early on that the low-end cards were nearly identical hardware.

As it happens, the high end cards are also similar, except that they have cache RAM, battery backup or CVM, etc., tacked on, usually with more CPU power.

This does not mean that it will always be able to crossflash as an end user. For example, the Dell PERC H200/H310 (or maybe just one of them?) can only be taken over to IT mode by winding it back to a very early firmware version and then "taking it across". Later firmware and toolset versions prohibit the conversion, based on (IIRC) data in the serial boot ROM, which identifies the card.

Someone well-versed in the arcanities of the MIPS CPU and toolset could probably figure out the viable options. There are a number of people who have been doing this, and some of them have made various tools such as

GitHub - marcan/lsirec: LSI SAS2008/SAS2108 low-level recovery tool for Linux

LSI SAS2008/SAS2108 low-level recovery tool for Linux - marcan/lsirec

github.com

The problem is that you may not actually have the files, or tweaked files, to make the conversion happen, and it may require some significant skill to dig into this far enough to have success, even if you have tools available. If you do not love doing this kind of thing just for the sake of doing it, it's probably not realistic to try to figure this out on your own. If someone has a prebuilt recipe for how to do it, by all means, go ahead...

But, I suspect, at the end of the day, it isn't really worth the trouble. The prices on these cards have generally fallen from the $80-$150 that they were a decade ago on the used market, to $30-$40 today. I'm sorry if that's not true in the UK, you can feel free to thank Louis DeJoy's screwing up of the USPS. I do understand that the H700 is basically worthless today, but I can't really do anything about the economics of all this.

 

[LP1]

The problem is that you may not actually have the files, or tweaked files, to make the conversion happen, and it may require some significant skill to dig into this far enough to have success, even if you have tools available. If you do not love doing this kind of thing just for the sake of doing it, it's probably not realistic to try to figure this out on your own. If someone has a prebuilt recipe for how to do it, by all means, go ahead...

That is exactly my point... I had glance at google before getting into TrueNAS and generally I have seen suggestions of getting HBA or using RAID card flashed into IT mode. Now when I started looking how to turn my particular RAID card into HBA I realised there are no tweaked files, nor tools available. Reading some older threads it seems that these may have existed at some point, but by now all links are dead.

On one hand I don't mind doing "this kind of thing" for the sake of it, but I honestly struggling to piece together what needs to be done. Take for example tool you tagged... ok so it will try to write on any SAS2xx8 chip, which in theory what I need for H700 (SAS2108), but even then assuming that I could figure out how to use it in Linux (which I generally don't use), I still need new firmware which I should flash instead of whatever Dell put on H700. Is it just a matter of getting latest firmware from Broadcom for LSI 9260-8i (some suggests 9211) and hope for the best?

And what about ASUS PIKE - if I read correctly 2008 should be easy enough to crossflash as LSI 9220-8i.... but again just at glance. Would it not be the same problem, that when it comes to it won't find the correct firmware to use?

It seems like the only way is to get H200/310/PIKE 2008 with IT mode or do crossflash myself, I am just kind of annoyed that I have to get technically worse hardware for more money... are there any other HBA options? I am kind of surprised considering popularity of TrueNAS that there is no dedicated HBA which just works without any hacks? Please let me know if there is....

 

[jgreco]

Well of course there is. The LSI 9211-8i is the LSI branded HBA you're seeking.

Unfortunately, this is not a retail part. LSI does sell the LSI 9240-8i as a retail part, which is effectively the same card, but with the low end RAID feature set firmware. This is crossflash-able to IT mode without too much drama. However, being a retail part, it has always sold for a lot of money. So the people who can't find a real 9211-8i often get the 9240-8i instead.

Some enterprising cheapskates noticed that LSI sold 9240-8i based designs to OEM's such as IBM and Dell, who sold these as the IBM ServeRAID M1015 and Dell PERC H200, who produced them inexpensively and sold them integrated into servers with firmware that proclaimed them to be IBM or Dell cards. When these servers came to the end of lease, they were often parts'ed up by recyclers and the SAS cards sold cheaply on eBay.

But there's an underlying assumption in the industry that no one wants HBA's. Big storage has always been focused on RAID, whether low end or high end. RAID controllers work poorly with ZFS for all the reasons I already linked to. So ZFS is in this weird situation where the server vendors don't really put out a card that's compatible with ZFS, and, worse, they're fine with that because they'd rather sell you one of their own storage appliances, or get you to build a Windows server for serving your files. They don't want to make low-margin cards that might only sell for a hundred bucks new. They want to sell the high margin thousand dollar RAID controller with CacheVault and all the other value-add features. The popularity of TrueNAS with hobbyists is not of concern to the vendors, because they are focused on selling Windows Servers to corporations and enterprise deployments.

 

[LP1]

I guess you are right - apart of TrueNAS there are no corporate use... so nobody cares. But you know how in china they make gaming motherboards for recycled Xeon processors and you can buy whole brand new MB for like £30 ($40), so one would thing chinese would realise the market for HBAs and make inexpensive controllers... if MB can be made for $40, then HBA would be $10-15 max.

i.e. - https://www.alibaba.com/product-detail/Big-sell-X79-Motherboard-lga2011-E5_1600084259136.html

In either case, I have checked and for me there isn't much difference in cost whenever to buy LSI 9211-8i or flashed H310... the only thing I still confused about is that you say LSI 9211-8i is non retail... so what are these then?
https://www.amazon.co.uk/SAS9211-8I-8PORT-Int-Sata-Pcie/dp/B002RL8I7M (£78.82/$105.77)
https://www.ebay.co.uk/p/19017381395 (£64.80/$86.96)

 

[jgreco]

It isn't clear that LSI would want to sell their silicon cheaply to some Chinese outfit in order to have mass produced "cheap" HBA's. They would probably be afraid that the Chinese outfit would then be able to easily compete with the low-end RAID cards that LSI sells, simply by putting different firmware on there. LSI would probably prefer that people continue to buy the 9240-8i and downgrade it.

The hardware market is generally dominated by volume. Because any given part tends to be expensive to design, if you want to design an HBA and it costs you a hundred thousand dollars to pay some engineers to do all the design and manufacturing setup work involved, and you only sell ten thousand of them, that's not going to end up profitable. However, if you have them design an Intel clone ethernet card, and it costs a hundred thousand, and you sell a hundred thousand or a million of them, the volume there is where profit lies. The engineering cost is amortized across a much larger number of items.

This, along with some logistics, is why there is a thriving market in passable Intel knockoff ethernet cards, but no Chelsio fakes (that I know of). There's no market for "new" fake Chelsio cards. The companies that want Chelsio are willing to pay for them and buy them from Chelsio.

The market for new knockoff HBA's is probably substantially undercut by the glut of used HBA's on the recycled market. But there are definitely knockoff LSI's from China. I have a few, somewhere around here, sent by a forum member or two. They didn't work right, which was a problem.

The 9211-8i has historically not actually been available in the retail channel. Just like Intel has a lot of CPU parts that are targeted towards OEM's. One could spitball that this is because they wanted to encourage sales of the 9240-8i instead. However, LSI got sold to Avago, which then got sold to Broadcom, and none of these cards are high volume cards. It's possible that Broadcom decided to make these available in the channel again. Or it is entirely possible that the seller has acquired knockoff LSI's from China. These things can come boxed in convincing-looking packaging. I have some nice fake Intel ethernets in packaging virtually identical to legit Intel, except that the label affixed to the box says "DAUL" instead of "DUAL"...

Anyways, as I said above, it is what it is because of the way it all is. If I had MY way, I would really like to make specialized NAS units optimized for FreeNAS/TrueNAS, sort of like a QNAP or Synology, small footprint, high efficiency, lots of connectivity, lots of RAM, decent fast CPU. Doesn't really exist, alas, and if it did, it'd be hella-expensive.

 

[LP1]

Ok, so the products in the links above are potentially "fake", non-licensed, refurbished or re-flashed OEM? Meaning it is probably safer to get H200/310 and crossflash - at least it is guaranteed average quality HP server part, rather than knock-off which may or may not work?

 

[jgreco]

Ok, so the products in the links above are potentially "fake", non-licensed, refurbished or re-flashed OEM? Meaning it is probably safer to get H200/310 and crossflash - at least it is guaranteed average quality HP server part, rather than knock-off which may or may not work?

It's hard to guarantee anything. I've written a guide on how to shop carefully on eBay for parts, which basically just comes down to looking for parts being cycled out of data centers by reputable recyclers.

I believe that LSI considered their PCIe 2 6Gbps lineup (9211, 9240, other 2008-based) obsolete at least half a decade ago, and the PCIe 3 6Gbps stuff probably not much after that, since SAS 12Gbps was coming into play at that time. This suggests that those SAS 6Gbps retail parts should have become harder to come by. OEM cards may have been sold (or continue to sell!) for somewhat longer. For example, Supermicro manufactures their own AOC-S2308L-L8i, and WiredZone is a drop-ship reseller, so if I buy one of those from WZ, I expect I would get a brand new and fully legit LSI 2308 card. Only the chipset comes from LSI. It is entirely possible that there are companies that bought a truckful of LSI 6Gbps retail cards in 2014 and are selling legitimate new boxed retail cards, stranger things have happened, but it also seems quite likely to me that "new" cards today are probably some sort of fraud, unless you buy them from a trusted source.

SATA is a dead end, because SATA 6Gbps is fast enough for HDD, and even a hypothetical 12Gbps SATA is too slow for SSD, so we may continue to see availability of things like the OEM Supermicro 6Gbps LSI based cards for awhile in order to populate and repurpose the massive numbers of 6Gbps SAS chassis that already exist, but it is good to be a bit paranoid about what's available and why. This means that Supermicro's low-end RAID card is probably the closest thing to the "Chinese" offering you're looking for, and it's legit, but it's also still selling at (around?) MSRP from a decade ago, so only companies who require "new" gear are going to be buying it.

Sorry, I know I'm being a bummer.