SOLVED LSI SAS2008 HBA (aka 9211-8i ?) Q: Firmware Upgrade? to stop BIOS detection + fix hotswap not working.

[genBTC]

Main Question:
-----------------------
Running Freenas 11.2-U8 on a Dell Poweredge T310 on an aftermarket OEM LSI HBA card already in IT mode, that I thought was an LSI 9211-8i, but it turns out its actually an OEM IBM 9200 / SAS2008.
I honestly have no idea what LSI card is equal to what, I am guessing that LSI 9211 = IBM 9200 = SAS2008 chip. Please correct me if I'm wrong.
Currently the whole system is working, but it only detects drives during BIOS, and drives passed through to FreeNas works, except HotPlug/Hotswap is not working...
The dmesg output and mpsutil output shows that it detected *something* new when hot-plugged into New Slots, but FreeNas is not getting updated to attach them.
I also know from the past, that hotswap in the Same Slots also does not register enough to work right.
I need hotplug/hotswap to work. Hopefully upgrading firmware will solve it. Any other idea? I don't want to fix whats not broken, if I can just run a command to detect them.

Diagnosis:
----------------
mpsutil also shows its only on firmware 10.
I know firmware 20 is out, and I now wish to upgrade = hopefully then it will detect hotswap. (unless i'm unaware of something)
mpsutil also shows BIOS version 7.19 and UEFI version 7.18 - these also seem outdated. I found a 7.39 BIOS and a 7.27 UEFI.
sas2flash bundled with FreeNas also seems to be on an older version, 16.0 (not 20.0)
I am using BIOS boot in an EFI system (a personal workaround from the past), but the new non-grub FreeNas bootloader seems to have resolved my old issue, and I think now is a good time to switch.
I heard I can skip BIOS drive detection entirely and just rely on Freenas, and I'm fine with that, but I am confused on the method of removal. How do I do it ?
I also don't enjoy trawling through thousands of forum posts and I had to open about 50 tabs, just to advance this far. Maybe this will help someone later.
Visiting LSI.com takes you to Broadcom.com and I would prefer to get all files directly from them, but their website is a pain to navigate with the number of clones that exist.

Starting here:
https://www.broadcom.com/support/do...=SAS+9211-8i+Host+Bus+Adapter&pa=&po=&dk=&pl=

Firmware Section:
Installer_P20_for_MSDOS_and_Windows / Windows Installer
https://docs.broadcom.com/docs/12350530
https://docs.broadcom.com/docs-and-...ckage_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip

Code:

Firmware                    :  \firmware\HBA_9211_8i_IT\2118it.bin          Version no: 20.00.07.00     Release date: 11-FEB-16
BIOS                        :  \sasbios_rel\mptsas2.rom                     Version no:  7.39.02.00     Release date: 03-AUG-15
Installer for DOS (SAS2FLSH):  \sas2flash_dos_rel\sas2flsh.exe              Version no: 20.00.00.00     Release date: 18-SEP-14

(no idea why the dates are off, probably a documentation bug, most everything in this .zip file is dated March 2016, except the BIOS date is right)

Firmware Section:
Installer_P20_for_UEFI / UEFI Installer
https://docs.broadcom.com/docs/12350820
https://docs.broadcom.com/docs-and-...es/sas_sata_6g_p20/Installer_P20_for_UEFI.zip

Code:

Installer for EFI (SAS2FLASH)    :  \sas2flash_efi_ebc_rel\sas2flash.efi    Version:20.00.00.00     Release Date:18-SEP-14

Firmware Section:
Installer_P20_for_FreeBSD / FreeBSD Installer
https://docs.broadcom.com/docs/12350815
https://docs.broadcom.com/docs-and-...sas_sata_6g_p20/Installer_P20_for_FreeBSD.zip

Code:

Installer for FreeBSD (SAS2FLASH):  \sas2flash_freebsd_amd64_rel\sas2flash  Version:20.00.00.00     Release Date:18-SEP-14

BIOS section: (actually a UEFI rom)
UEFI_BSD_P20 / UEFI BSD
https://docs.broadcom.com/docs/12348628
https://docs.broadcom.com/docs/UEFI_BSD_P20.zip
LSI Corporation SAS2 MPT UEFI BSD HII Driver Release: \x64sas2.rom (for Flashing on X64 platforms) Version: 7.27.01.01
Supported Controllers: SAS 2008 (also: SAS2004, SAS2108, SAS2116, SAS2208, SAS2308)
Also includes a Signed SecureBoot version.

This seems to be all thats relevant.

Commands (taken from another thread):
-----------------
sas2flsh -o -f 2118it.bin -b mptsas2.rom sas2flsh -o -f 2118it.bin (if OptionROM is not needed)

Conclusion
----------------
So I think I have all the files for the firmware upgrade, and an idea of the firmware flash command / process,
but I'm not sure what to do about the BIOS removal or Bios flash / UEFI flash additional process.
The UEFI document says it needs the latest firmware and latest NVdata, and to unload the driver.
I also am not sure if I can even flash from Freenas, because a pool is running on 2 old drives on it, and I dont know how to unload a pool, or unload the driver.
So rebooting seems to be my only option.
Boot from a new USB stick into DOS? Then flash? What about the OptionROMs? Can I remove it? How?
Am I missing anything?
Thanks.

Diagnostic Commands used:
------------------------------------------
dmesg mpsutil sas2flash sysctl camcontrol

Full Diagnostic Data:
-------------------------------
# Something _was_ detected. Just not as a /dev/da* device

Code:

[EOFL@freenas ~]$ dmesg | grep mps0
mps0: <Avago Technologies (LSI) SAS2008> port 0xfc00-0xfcff mem 0xdf1bc000-0xdf1bffff,0xdf1c0000-0xdf1fffff irq 16 at device 0.0 on pci2
mps0: Firmware: 10.00.08.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
mps0: SAS Address for SATA device = 382a272f6e953836
mps0: SAS Address from SATA device = 3b2a27328d845551
mps0: SAS Address from SATA device = 382a272f6e953836
da1 at mps0 bus 0 scbus0 target 1 lun 0
da0 at mps0 bus 0 scbus0 target 0 lun 0
<--- starting here, new drives were hotplug inserted into new slots -- >
mps0: SAS Address for SATA device = 4f666550e095d970
mps0: SAS Address from SATA device = 4f666550e095d970
mps0: SAS Address for SATA device = 4f666550e0b5d394
mps0: SAS Address from SATA device = 4f666550e0b5d394

# da1 and da2 are working on this HBA already, through BIOS detection, (da2 is my flash drive)

Code:

[EOFL@freenas ~]$ ls /dev/da*
crw-r-----  1 root  operator  0x6f May  7 17:19 /dev/da0
crw-r-----  1 root  operator  0x78 May  7 17:19 /dev/da0p1
crw-r-----  1 root  operator  0x79 May  7 17:19 /dev/da1
crw-r-----  1 root  operator  0x82 May  7 17:19 /dev/da1p1
crw-r-----  1 root  operator  0x7a May  7 17:19 /dev/da2
crw-r-----  1 root  operator  0x83 May  7 17:19 /dev/da2p1
crw-r-----  1 root  operator  0x84 May  7 17:19 /dev/da2p2

da2 at umass-sim0 bus 0 scbus10 target 0 lun 0 da2: <SanDisk Ultra 1.00> Removable Direct Access SPC-4 SCSI device

#The first two, Handle 0009 and 000a, are what was hotplugged, and are missing a B____T whatever that is #

Code:

[EOFL@freenas ~]$ sudo mpsutil show all
Adapter:
mps0 Adapter:
       Board Name: IBM 6Gb Perf HBA
   Board Assembly: H3-25113-03A
        Chip Name: LSISAS2008
    Chip Revision: ALL:
    BIOS Revision: 7.19.00.00
Firmware Revision: 10.00.08.00
  Integrated RAID: no

PhyNum  CtlrHandle  DevHandle  Disabled  Speed   Min    Max    Device
0                              N                 1.5    6.0    SAS Initiator
1                              N                 1.5    6.0    SAS Initiator
2       0001        0009       N         6.0     1.5    6.0    SAS Initiator
3       0002        000a       N         6.0     1.5    6.0    SAS Initiator
4                              N                 1.5    6.0    SAS Initiator
5                              N                 1.5    6.0    SAS Initiator
6       0003        000b       N         6.0     1.5    6.0    SAS Initiator
7       0004        000c       N         6.0     1.5    6.0    SAS Initiator

Devices:
B____T    SAS Address      Handle  Parent    Device        Speed Enc  Slot  Wdt
          4433221102000000 0009    0001      SATA Target   6.0   0001 01    1
          4433221103000000 000a    0002      SATA Target   6.0   0001 00    1
00   01   4433221106000000 000b    0003      SATA Target   6.0   0001 01    1
00   00   4433221107000000 000c    0004      SATA Target   6.0   0001 00    1

Enclosures:
Slots      Logical ID     SEPHandle  EncHandle    Type
  08    500605b00544bd30               0001     Direct Attached SGPIO

Expanders:
NumPhys   SAS Address     DevHandle   Parent  EncHandle  SAS Level

#Shows the version numbers for everything.

Code:

[EOFL@freenas ~]$ sudo sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2)

    Controller Number              : 0
    Controller                     : SAS2008(B2)
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500605b-0-0544-bd30
    NVDATA Version (Default)       : 0a.02.00.20
    NVDATA Version (Persistent)    : 0a.02.00.20
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 10.00.08.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : Undefined
    BIOS Version                   : 07.19.00.00
    UEFI BSD Version               : 07.18.04.01
    FCODE Version                  : N/A
    Board Name                     : IBM 6Gb Perf HBA
    Board Assembly                 : H3-25113-03A
    Board Tracer Number            : SP23001554

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

#Sysctl output for good measure

Code:

[EOFL@freenas ~]$ sudo sysctl -a | grep mps
device    mps
dev.mps.0.use_phy_num: 1
dev.mps.0.encl_table_dump:
dev.mps.0.mapping_table_dump:
dev.mps.0.spinup_wait_time: 3
dev.mps.0.chain_alloc_fail: 0
dev.mps.0.enable_ssu: 1
dev.mps.0.max_io_pages: -1
dev.mps.0.max_chains: 4096
dev.mps.0.chain_free_lowwater: 4093
dev.mps.0.chain_free: 4096
dev.mps.0.io_cmds_highwater: 22
dev.mps.0.io_cmds_active: 0
dev.mps.0.driver_version: 21.02.00.00-fbsd
dev.mps.0.firmware_version: 10.00.08.00
dev.mps.0.disable_msi: 0
dev.mps.0.disable_msix: 0
dev.mps.0.debug_level: 3
dev.mps.0.%parent: pci2
dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0072 subvendor=0x1014 subdevice=0x03cb class=0x010700
dev.mps.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
dev.mps.0.%driver: mps
dev.mps.0.%desc: Avago Technologies (LSI) SAS2008
dev.mps.%parent:

#Tried to rescan the SCSI bus with camcontrol to no avail.

Code:

[EOFL@freenas ~]$ sudo camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
Re-scan of bus 3 was successful
Re-scan of bus 4 was successful
Re-scan of bus 5 was successful
Re-scan of bus 6 was successful
Re-scan of bus 7 was successful
Re-scan of bus 8 was successful
Re-scan of bus 9 was successful
Re-scan of bus 10 was successful

Code:

[EOFL@freenas ~]$ sudo camcontrol devlist
<ATA ST4000DM000-1F21 CC54>        at scbus0 target 0 lun 0 (pass0,da0)
<ATA ST4000DM000-1F21 CC54>        at scbus0 target 1 lun 0 (pass1,da1)
<OCZ-REVODRIVE X2 1.20>            at scbus1 target 0 lun 0 (ada0,pass2)
<OCZ-REVODRIVE X2 1.20>            at scbus2 target 0 lun 0 (ada1,pass3)
<ST2000DM006-2DM164 CC26>          at scbus5 target 0 lun 0 (ada2,pass4)
<Hitachi HUA722020ALA330 JKAOA3MA>  at scbus5 target 1 lun 0 (ada3,pass5)
<ST2000DM001-1CH164 CC29>          at scbus6 target 0 lun 0 (ada4,pass6)
<ST4000DM000-1F2168 CC52>          at scbus6 target 1 lun 0 (ada5,pass7)
<SanDisk Ultra 1.00>               at scbus10 target 0 lun 0 (pass8,da2)

#da0 and da1 are on the HBA
#ada0/1 are two SATA connected SSDs in 1 PCI-e slot (old OCZ tech, don't ask)
#ada2,3,4,5 are existing HDDs on internal SATA ports.
#da2 is my boot flash drive

 

[Yorick]

All answered here: https://www.ixsystems.com/community...-lsi-9211-9300-9305-9311-hba-and-variants.54/

Not only is that resource excellent and thorough, it also has all the files you need and explains their use.

 

[genBTC]

Thank y

All answered here: https://www.ixsystems.com/community...-lsi-9211-9300-9305-9311-hba-and-variants.54/
Not only is that resource excellent and thorough, it also has all the files you need and explains their use.

Wow, thats a very comprehensive thread, I will be reading for hours. Thank you.
I actually was wrong about what model it was, so I probably would have messed it up. Seems like its an IBM M1015 already crossflashed...
The ebay listing I bought it from says this:
"Genuine LSI 6Gbps SAS HBA LSI 9200-8i = (9211-8I) IT Mode ZFS FreeNAS unRAID US"
"DELL H200 H310 M1015 9220-8I 9201-8I IT Mode"
So now I think it was the IBM M1015 variant, cross-flashed to 9211-8l IT already.
At first glance:
I missed the steps of: "Wiping the card" using MegaRec or "sas2flash -e 6 (or 7)". But I'm not sure if I have to do that, since I'm already on IT mode. He says maybe I don't.
Also missed the SAS address needing to be added back in, with "sas2flash -c CARDID -o -sasadd HEXSTRING".
And I think if I flash the new firmware over top, and just leave off the -b BIOS or UEFI part, it will remove the BIOS, and I can skip BIOS detection.
Can you confirm any of these offhand ?

 

[genBTC]

1. I finally got it working ! The thread was very helpful but my Dell PowerEdge T310 system had issues. I had to stray off the beaten path to get it to work. He made me confident that I could not brick it.
2. EFI mode still did not work at all, thats my server's fault though, it just hung at "Initializing the UEFI". So I reverted back before I started.
3. BIOS mode with DOS works.
4. He said to use MSDOS instead of FreeDOS and to use Rufus to make it, but all the versions of Rufus I tried only have a FreeDos option, and the old "make bootable dos disk" seems to be gone, a thing of the past. Probably not kosher to distribute those M$ files.
5. Megarec.exe would execute, but all attempts at accessing the card hung, no matter the command, so I was unable to -readsbr, or -cleanflash
6. sas2flsh.exe worked wonderfully, especially P20 DOS version from the authentic Broadcom links I found above. I did not need P5 or EFI.
7. I determined sas2flash P20 has 1 additional important sounding option, than P16 (as was in Freenas): "-sbr" however I did not even need to use it. It seemed to detect that it needed to run this option itself.
   -sbr: If mismatch found between Current and NVDATA MFG Page 2,
   then update Persistent MFG Page 2, which will also update SBR.
8. Instead of Megarec.exe, i just ran the sas2flsh.exe -c 0 -o -e 7 command to erase it first, and hoped for the best.
9. I was then able to flash P20 2118it.bin firmware with the P20 sas2flsh.exe installer in Freedos, even with a "MfgPage2 Mismatch Detected" it corrected itself automatically. sas2flsh.exe -c 0 -o -f 2118it.bin -l flashlog.txt
10. There were only 3 commands I actualy needed. The firmware erased fine, and updated fine to version 20.00.07.00, and now shows up as an "LSI SAS9211-8i" unlike whatever ambiguous "IBM 6Gb Perf HBA" that it said before.
11. Both the BIOS and the EFI OptionROMs were fully cleared out with just those 2 commands.
12. I then re-added the old SAS ID: sas2flsh.exe -c 0 -o -sasadd 500605b00544bd30
13. Booting went super fast now that detection is skipped! However there was an error message in the early stages of the Freenas BSD bootloader, saying that it couldn't detect one of my ZFS pools (the one that was half on/half off this HBA) - because the BIOS had not loaded the Option ROM nor had Freenas loaded the mps0 driver in at that point. I don't think this matters for my use case, but I'm curious if you can force the mps driver to load in /boot/loader.conf and then still gptZFSboot off the pools on the HBA in that way.
14. HOT PLUG and HOT SWAP WORKS NOW! Everything was successful, and it only took me 12 hours... :)

Thanks for the help finding me the link. I did not stumble across it, and it would have probably taken twice as long without this thread.

 

[genBTC]

Only one download needed:

https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_6g_p20/9211-8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip

Rufus was used to create FreeDos Bootable USB.
Entire process log:
---------------------------
C:\SAS2FLSH.EXE -c 0 -o -e 7 (forgot to capture the erase command output)

Code:

C:\SAS2FLSH.EXE -c 0 -o -f 2118it.bin -l flashlog.txt
    Adapter Selected is a LSI SAS: SAS2008(??)   

    Executing Operation: Flash Firmware Image

        Firmware Image has a Valid Checksum. 

        Firmware Image compatible with Controller. 

        Valid NVDATA Image found. 

        Valid Initialization Image verified.
        Valid BootLoader Image verified.

        Chip is in RESET state. Attempting Host Boot...
        Firmware Host Boot Successful !
        MfgPage2 Mismatch Detected.
        Writing Current MfgPage2 Settings to NVRam
        Updated MfgPage2 !
        Resetting Adapter...
        Adapter Reset Completed.
        Chip is in RESET state. Performing Host Boot...
        Firmware Host Boot Successful !

        Beginning Firmware Download...
        Firmware Download Successful.

        Resetting Adapter...
        Adapter Successfully Reset.

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

C:\SAS2FLSH.EXE -c 0 -o -sasadd 500605b00544bd30 -l sasadd.txt 
    Adapter Selected is a LSI SAS: SAS2008(B2)   

    Executing Operation: Program SAS Address

    SAS Address Successfully Programmed!

    Resetting Adapter...
    Adapter Successfully Reset

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

C:\SAS2FLSH.EXE -c 0 -list -l newcard.txt
    Adapter Selected is a LSI SAS: SAS2008(B2)   

    Controller Number              : 0
    Controller                     : SAS2008(B2)   
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500605b-0-0544-bd30
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : N/A
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

# Hot plug working:
mps0: mpssas_prepare_remove: Sending reset for target ID 2 da2 at mps0 bus 0 scbus0 target 2 lun 0 da2: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K4TDZ5R0 detached (da2:mps0:0:2:0): Periph destroyed mps0: Unfreezing devq for target ID 2 mps0: SAS Address for SATA device = 4f666550e095d970 mps0: SAS Address from SATA device = 4f666550e095d970 da2 at mps0 bus 0 scbus0 target 2 lun 0 da2: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device da2: Serial Number WD-WCC7K4TDZ5R0 da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 3815447MB (7814037168 512 byte sectors) da2: quirks=0x8<4K>

 

[Yorick]

I’m glad this worked for you!

IBM 6Gb Perf HBA

That’s a cousin to the M1015. I have the same card, also came with P10 IT Mode, just that I flashed it directly to P20 without erasing it first. Works for my hardware, it’s a board that boots into EFI without CSM.