Detailed newcomers' guide to crossflashing LSI 9211/9300/9305/9311/9400/94xx HBA and variants

[Stilez]

This tutorial is so great! Thanks for the write up!

This may not be relevant on a FreeNAS forum, but this is the only site I could find close to what I wanted to do. I purchased a LSI 9300-8i (default IT mode) for a windows machine and would like to flash to IR mode. The firmware file on the broadcom site has 9300-8i IT and 9311-8i IR firmwares (this is the same for both cards). They are both SAS3008 chips. Does that mean I can crossflash my card with the 9311 IR firmware and use hardware RAID instead of windows dynamic disk raid? I tried it before reading this with SAS3Flash and it didn't allow because the NVROM did not match 9300 vs 9311. I assume that's why the NVROM must be erased before crossflash. Also no mention in the tutorial of the 2 pin jumper, is that required on any cards (read this on another tutorial)?

Any info is greatly appreciated, TIA

Quick sketchy answers as best I know, as I'm out being social...

Thanks for the compliment! Every time.someone says it helped, I get a bit more of a smile.

The process is agnostic as to IT/IR so its as valid for cross flashing to IR as IT. Should be absolutely no difference. Slight risk because you're adding not removing software for a function, so when you flash to IT you absolutely know the hardwares there for it... if you crossflash IT to IR though is stuff like cache or whatever all there? Hard to be so sure. But gut feel says the card will work. Either the firmware will detect its resources, or the reference design has all it needs. Its the same chip so I would expect its fine. Try it and let us know?

2 pin - never needed it. But check on the manufacturer card technical PDF/user guide. They will say for sure what those pins are, for your card.

 

[curvingedge]

Quick sketchy answers as best I know, as I'm out being social...

Thanks for the compliment! Every time.someone says it helped, I get a bit more of a smile.

The process is agnostic as to IT/IR so its as valid for cross flashing to IR as IT. Should be absolutely no difference. Slight risk because you're adding not removing software for a function, so when you flash to IT you absolutely know the hardwares there for it... if you crossflash IT to IR though is stuff like cache or whatever all there? Hard to be so sure. But gut feel says the card will work. Either the firmware will detect its resources, or the reference design has all it needs. Its the same chip so I would expect its fine. Try it and let us know?

2 pin - never needed it. But check on the manufacturer card technical PDF/user guide. They with say for sure what those pins are, for your card.

I will try it out and post the results.

 

[datashadow]

First of all thank you for an awesome and thorough resource!

One addendum to the "Bricking Risk" section that is worth mentioning...

The Dell PERC (PowerEdge RAID Controller) MINI MONOLITHIC (AKA Mini Mono) cards can, and will, be "bricked" if you blindly follow many of the cross flashing guides. Electronically these cards are the same as their "standard" form factor brethren, but physically they have a proprietary form factor. The models are "H310" (LSI SAS2008, no cache), "H710" (LSI SAS2208 512 MB cache) and "H710p" (LSI SAS2208 1 GB cache). There are multiple Dell part numbers for some, if not all, of these cards. The proprietary Mini Monolithic socket shows up in the 12th generation Dell hosts such as the R620 and the R720. For their 11th Generation servers (e.g. R710) Dell also made "H200" and "H310" cards in a standard PCiE form factor (but without a rear bracket) for use in special PCiE slots that are dedicated to storage controllers.

The "bricking" danger with the Mini Mono cards comes from the fact that they have a specific Subsystem ID and Product ID values recorded in the card's SBR (Serial Boot ROM). If the Dell server doesn't see the correct Subsystem ID and Product ID in the Mini Mono's SBR then it halts the boot process. Many flashing guides recommend erasing the SBR (e.g. in your Crossflashing Details section). Since these cards are a proprietary form factor you can't just put them in "another slot" and re-flash the SBR. You end up with a chicken-and-egg problem that you need to reboot the server to update (reflash) the SBR, but you can't boot the server because the SBR is blank.

My first exposure to the solution to cross flashing a Mini Mono card came from a You Tube poster and eBay vendor that goes by the moniker "Art of Server". I don't know if he was the first to figure out the SBR issue, and how to get around it while cross flashing, or if he was just one of the first to publicize the technique. There are now (May 2020) multiple resources available (AKA Google is your friend). Disclaimer: I am not associated with "Art of Server", but I appreciate the knowledge and humor shared in his videos.

Take care, and stay well!

David

Here are some resources:

H310/H710/H710P Mini & Full Size IT Crossflashing

Introduction - Fohdeesha Docs

fohdeesha.com

How to flash the Dell PERC H310 (and keep it in the dedicated storage slot)

Wayback Machine

Dell PERC H310 Mini Mono flashing to IT Mode

Dell PERC H310 Mini Mono flashing to IT Mode

Up until now it wasn't possible to use the Dell Perc H310 Mini Mono cards with FreeNAS because it just seemed not to be possible to flash them to IT mode or if you did that, your Dell server wouldn't boot up because it didn't found the proper Mini Mono card. Now someone seems to have found a...

www.ixsystems.com

Dell H310 mini with LSI IT mode firmware

Dell H310 mini with LSI IT mode firmware

In this video, I'll show you my Dell H310 mini monolithic module in my R720XD that is running LSI IT mode firmware. This was previously thought to be impossi...

youtu.be

H710 mini with IT mode firmware

H710 mini with IT mode firmware

In this video, I'll show you my Dell H710 mini monolithic module in my R720XD that is running LSI SAS2308 IT mode firmware. The H710 is a true hardware RAID ...

youtu.be

Will I be able to recover a bricked H310 mini?

Will I be able to recover a bricked H310 mini?

In this video, I bring you along my little adventure into recovering a bricked H310 mini provided by one of my customers.ch341eeprom program: https://github....

youtu.be

 

[eborduas]

Great write-up!

>2. Copy from the zip file, all the files under "USB" to the root of your USB stick. The download link is at the end of the page.

The latest ZIP file on Google Drive does not contain a USB folder.

 

[Stilez]

The latest ZIP file on Google Drive does not contain a USB folder.

The clue is in the zip file name........

The files you're looking at, *are* the content....... :)

 

[eborduas]

Thanks for the reply.

That's what I did initially, copy all the files to the USB stick, but that borked the Rufus FreeDOS config with an error message saying it could not find COMMAND.COM. I recreate the Rufus FreeDOS boot and only copied the _SAS2FLASH directory and was able to update the driver.

 

[zbyte]

This is obviously dedicated to the process of updating good HBA's, if the following post is OT please feel free to delete or move to the correct thread.

Recently building a TrueNAS server went ahead and purchased off Ebay an Avago LSI 9400-8i from China (see where this is going) my only purpose with this comment is just to add to the already existing awareness.

Seller has impeccable feedback 99.5% out of 33000, the card looks new and amazing. I even compared the card with valid images found online not from Ebay and it looks the same.

The card has a serial number which was sent to Broadcom and they confirmed is still under warranty.

From Broadcom:

"The S/N shows in warranty. Please note that Broadcom holds the final word on an RMA approval which includes checking the S/N. This should clear based on what I see.
I suggest you connect to our Portal and proceed with creating an RMA.
Best Regards,
<Name omitted>
Warranty Record ID: <ommited>
Serial Number: <ommited>
Customer Serial Number: <ommited>
Warranty Start Date: 01/26/2018
Warranty End Date: 01/26/2021
Strike Count: 0
Feed ID: <ommited>
Part ID: <ommited>
Part Number: 03-50008-16
Part Description: 9400-8i, NA NA M 00 F L"

Obviously even the serial numbers they are replicating are valid, or maybe the card is actually authentic but doesn't seems to be.

During post errors are reported by the card. Then when TrueNAS loads starts displaying errors in a loop and it doesn't finish booting.

Then all the sudden after 10 min or so magically the motherboard starts showing no errors at POST and TrueNAS recognizes the card and loads a pre existing pool.

When running S.M.A.R.T. short test though with drives that previously showed no errors now it spits read failures

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short captive Completed: read failure 90% 38313 2611945788
# 2 Short offline Completed: read failure 90% 38313 2611945790
# 3 Short offline Completed: read failure 90% 38291 2611949008
# 4 Short offline Completed without error 00% 8860 -

With another drive that had no errors now the HBA resets and never finishes the test

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short captive Interrupted (host reset) 80% 38336 -
# 2 Short captive Interrupted (host reset) 80% 38336 -
# 3 Short offline Completed without error 00% 38336 -
# 4 Short offline Completed without error 00% 38313 -
# 5 Conveyance offline Completed without error 00% 38313 -
# 6 Short offline Completed without error 00% 38312 -

This is the HBA error

mpr0: Controller reported scsi ioc terminated tgt 0 SMID 1490 loginfo 31130000
mpr0: Controller reported scsi ioc terminated tgt 0 SMID 1452 loginfo 31130000
mpr0: Finished abort recovery for target 0
mpr0: Unfreezing devq for target ID 0
(da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0a c0 3b 58 00 00 08 00
(da0:mpr0:0:0:0): CAM status: CCB request completed with an error
(da0:mpr0:0:0:0): Retrying command, 3 more tries remain
(da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0a c0 3b 58 00 00 08 00
(da0:mpr0:0:0:0): CAM status: SCSI Status Error
(da0:mpr0:0:0:0): SCSI status: Check Condition
(da0:mpr0:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da0:mpr0:0:0:0): Retrying command (per sense data)
pid 1807 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)

I would have assume the adapter was good and authentic, even with the failures I thought that maybe the Avago 9400-8i wasn't entirely supported by TrueNAS but the motherboard failing initially to detect the adapter without errors properly was a red flag.

Hope this helps.

 

[Ericloewe]

the motherboard failing initially to detect the adapter without errors

have you updated the firmware?

I ask because the 9400s were somewhat flaky at launch, so it's possible it may still be a firmware/driver issue.

 

[zbyte]

have you updated the firmware?

I ask because the 9400s were somewhat flaky at launch, so it's possible it may still be a firmware/driver issue.

Thanks @Ericloewe Not yet, downloaded 17.0 from the Broadcom site but I would have to plan the update for next week. I will give it a try, what puzzles me is the board reported errors through the POST and did not finished booting with TrueNAS or even Windows, then after some time it did not displayed more errors and booted properly, still reporting resets within TrueNAS
I will report back next week,

 

[averyfreeman]

Is there any resource someone knows about like this for Adaptec/Microsemi cards?

I have an HP branded SmartHBA 2100-4i4e which looks like a super cool card, but HP only released firmware up to 1.98 in 2018, while Microsemi just dropped 3.21 a few months ago.

The only difference between the two cards as far as I can tell is the HP cards won't update using Microsemi firmware - the firmware versions even track the release numbers.

I recently crossflashed an HP 9214-4e to P20 LSI 9214-4i4e using megarec + sas2flash in freedos. It seemed like the secret was wiping the old firmware first using megarec, then the card was ripe for whatever firmware was thrown at it.

If I could only find some way to do that with Adaptec software - I'm thinking afu.exe (?)

There's also a jumper for "hda mode" which is a recovery mode. However, when I shorted this jumper none of the software I tried could even find a controller, so it wasn't very useful with normal flashing utils.

Anyone know of a util to "force flash" a card based on pci address, etc.? I could really use a guide like this for Adaptec cards...

 

[Ericloewe]

The adaptecs are known to not work well at all with FreeBSD, so there's not much interest in them here.

 

[averyfreeman]

The adaptecs are known to not work well at all with FreeBSD, so there's not much interest in them here.

Yeah, I get it - Adaptecs are usually RAID cards, but this is an HBA. Oh well, just thought I'd ask...

 

[paradoxiom]

I flashed my card to IT mode back in like 2016, I'm not sure if I'm on the latest version or not but would like to make sure I have the latest version - I downloaded the pack from gdrive, I'm assuming I can skip most of the steps since I am not cross-flashing just updating the IT firmware? (or overwriting whatever the existing is). Where should I begin? Thank you.

 

[Alecmascot]

I flashed my card to IT mode back in like 2016, I'm not sure if I'm on the latest version or not but would like to make sure I have the latest version - I downloaded the pack from gdrive, I'm assuming I can skip most of the steps since I am not cross-flashing just updating the IT firmware? (or overwriting whatever the existing is). Where should I begin? Thank you.

For many of the common HBAs in that guide, there are no updates since 2016.
Try Broadcom to check

Support Documents and Downloads

Search technical documentation and downloads including firmware and drivers.

www.broadcom.com

 

[Agnemon]

Dell R710

My understanding is that, with Dell R710's and possibly all of Dells Gen 10 servers, if you wish to use an H200 card in the integrated slot then you need to flash the card with Dell's firmware (H200i variant) which can be got from Dell support (search for H200). Using any other firmware will mean that you have to use one of the other PCI slots. This may or may not be an issue depending on what additional cards you wish to install.

This may also be the reason that you cannot flash the card in a Dell server. The system sees the integrated slot ID and sulks.

Fantastic (and detailed) guide.

 

[zamana]

Hello everybody!

I decided to flash the IT Mode into my LSI 3008, but I confess that I'm having some difficulties in order to find the "binaries" that I need to download for my motherboard.

I've found this thread, but it's closed, so I can't interact there.

I have this motherboard, and I found this page at Supermicro site. But I can't understand where do I need to go to get the files for my case.

By the way, here is the information at IPMI interface:

Firmware Revision :​	01.39​
Firmware Build Time :​	07/18/2017​
BIOS Version:	2.0c
BIOS Build Time:	10/12/2017
Redfish Version :	1.0.1
CPLD Version:	02.b1.01

Does anybody can help me?

Thanks in advance.
Best regards.

 

[Redcoat]

Does anybody can help me?

Start by going here: https://www.truenas.com/community/t...mware-to-use-for-truenas12.97090/#post-670361 for info on a special firmware version for TrueNAS

You can find the full "standard" p16 package as a .zip on this page: https://www.broadcom.com/site-search?page=2&per_page=10&q=9300-8i

The SM website only had a dead link to the firmware for your board's HBA anyway.

I see that you may need a BIOS update also - one of the listed fixes "a few versions ago" was to "10. Fixed inability to load Broadcom SAS3008 configuration utility".

 

[zamana]

Hi!

Thanks for all the references. They are very helpful.

Two specific questions:

1) At the SM site there is a "bundle" with BIOS and BMC

May I use the "IPMI->Maintenance->Firmware Update" to update the whole thing (BIOS and BMC)?


2) SAS3008

From one of the links above, I ran these 2 commands:

Code:

root@freenas[~]# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:02:00:00
        SAS Address                    : 5003048-0-2447-bb00
        NVDATA Version (Default)       : 0a.00.21.00
        NVDATA Version (Persistent)    : 0a.00.21.00
        Firmware Product ID            : 0x2721 (IR)
        Firmware Version               : 10.00.00.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : LSI3008-IR
        BIOS Version                   : 08.25.00.00
        UEFI BSD Version               : 12.00.00.00
        FCODE Version                  : N/A
        Board Name                     : LSI3008-IR
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Code:

root@freenas[~]# sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  10.00.00.00    0a.00.21.00    08.25.00.00     00:02:00:00

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Now, how do I know which one of these needs to be flashed?

Thank you very much!

 

[Redcoat]

At the SM site there is a "bundle" with BIOS and BMC

May I use the "IPMI->Maintenance->Firmware Update" to update the whole thing (BIOS and BMC)?

Doesn't look like it. I downloaded both and the content is competely different - one is for the BMC, the other for the BIOS. I'm certain you need both. I think the header including Bundle is a stock header that's been carelessly applied!

Now, how do I know which one of these needs to be flashed?

It's 9300-8i - so you can follow the details at Example output when flashing a 9300-8i:

 

[zamana]

Doesn't look like it. I downloaded both and the content is competely different - one is for the BMC, the other for the BIOS. I'm certain you need both. I think the header including Bundle is a stock header that's been carelessly applied!


It's 9300-8i - so you can follow the details at Example output when flashing a 9300-8i:

8i -> "Eight Internal"

Got it!


Thank you!