SAS3 firmware to driver mismatch after update to FreeNAS-9.10.2-U3

[Kevin Horton]

Yesterday I updated from FreeNAS-9.10.2-U2 to FreeNAS-9.10.2-U3.

After rebooting, I got a warning in the GUI:

Firmware version 12 does not match driver version 14 for /dev/mpr0. Please flash controller to P14 IT firmware.

I'm very surprised to see this driver mismatch, as none of the publicly listed changes in FreeNAS-9.10.2-U3 mention any driver updates, and this warning was not there before the update.

I'm running a Supermicro X10SRH-cF, which has an on board Avago 3008 SAS3 controller.

sas3flash says:

Code:

# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 10.00.00.01 (2015.06.18)
Copyright 2008-2015 Avago Technologies. All rights reserved.

	Adapter Selected is a Avago SAS: SAS3008(C0)

	Controller Number			  : 0
	Controller					 : SAS3008(C0)
	PCI Address					: 00:01:00:00
	SAS Address					: 5003048-0-1cb3-6501
	NVDATA Version (Default)	   : 0b.02.30.26
	NVDATA Version (Persistent)	: 0b.02.30.26
	Firmware Product ID			: 0x2221 (IT)
	Firmware Version			   : 12.00.02.00
	NVDATA Vendor				  : LSI
	NVDATA Product ID			  : LSI3008-IT
	BIOS Version				   : 08.29.01.00
	UEFI BSD Version			   : 14.00.00.00
	FCODE Version				  : N/A
	Board Name					 : LSI3008-IR
	Board Assembly				 : N/A
	Board Tracer Number			: N/A

	Finished Processing Commands Successfully.
	Exiting SAS3Flash.

Looking on the Supermicro FTP site, I haven't found anything newer than firmware version 13. I know that there was a bug with SAS2, where there was a mismatch between the needed firmware and driver versions. Is it the same with SAS3?

If I need a newer version, where can I find it? I tried the Broadcom site, but I'm getting nowhere figuring out how to navigate it.

 

[Ericloewe]

I know that there was a bug with SAS2, where there was a mismatch between the needed firmware and driver versions. Is it the same with SAS3?

It's not a bug, it's a "feature" (thanks Broadcom /s).

For the P14 driver, you'll need P13 firmware.

 

[Dave Cheng]

Can try this https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#downloads


Sent from my SM-N920W8 using Tapatalk

 

[Kevin Horton]

OK. I'll update the firmware. I've never done that on in-service FreeNAS box before. My previous firmware updates were on new boxes, before setting up any storage.

Questions:

1. Can I use the sas3flash utility in FreeNAS to do the update? I see it's help screen says it can do firmware updates. I would copy the firmware to /root/, which is on the SSD that contains the OS. I hope to avoid the ugly fight to create a usable USB stick with the firmware on it, as it has always been a struggle in the past to get it to work.
2. I assume I should detach all volumes first, being sure to not select the option to destroy them. Is this sufficient precaution, or should I also disconnect all drives too? I'm leaning towards also disconnecting the drives. I do have an up to date backup via nightly replication to another server, but I hope to avoid the pain to use it.
3. Anything else I should be aware of here?

 

[Ericloewe]

Can I use the sas3flash utility in FreeNAS to do the update?

You can, but I wouldn't. I like to be sure the disks aren't being touched during the update.

I hope to avoid the ugly fight to create a usable USB stick with the firmware on it, as it has always been a struggle in the past to get it to work.

You're using Supermicro, there's really nothing difficult to it. Just grab the UEFI version of sas3flash, sas3flash.efi, copy it and the firmware files to a FAT32 USB drive, on reboot choose to load the UEFI shell and that's it.

I assume I should detach all volumes first, being sure to not select the option to destroy them. Is this sufficient precaution, or should I also disconnect all drives too? I'm leaning towards also disconnecting the drives. I do have an up to date backup via nightly replication to another server, but I hope to avoid the pain to use it.

See above, my method is simpler and less dangerous.

 

[TXAG26]

Not to be a stick in the mud, but why does this keep happening with new updates??? I think it's a pretty big deal to sneak in an update that changes the LSI driver version, thus requiring a firmware flash as well, without saying a peep in the update pop-up, the change logs, etc. I'm amazed this is still an ongoing problem.

Also, I can't seem to find the correct P14 firmware on Broadcom's site.
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#downloads

The old Supermicro flash routine batch file for P12 actually references three separate ROMs: 3008IT13.rom (1,037 KB), mptsas3.rom (208 KB), and mpt3x64.rom (223 KB).
It runs through all 3 when sas3flash.efi is executed. Yet, I can't find anything P14 related that is in the 1,000+ KB ballpark. I'm not sure what the difference is between the 3 ROMs, but the largest one looks important...

It's crickets RE: P14 on Supermicro's ftp site as well.
ftp://ftp.supermicro.com/driver/SAS/LSI/3008/Firmware/

 

[FadeToBlack]

I updated earlier today and got the firmware warning as well so I decided to update it. I have a 9305-16i and downloaded https://docs.broadcom.com/docs/9305_16i_Package_P14_IT_FW_BIOS_for_MSDOS_Windows.zip
I used the UEFI shell update tool and updating went without errors, both firmware and bios.

After that my FreeNAS box stopped booting and hung on the ZFS imports.

Now I'm trying to figure out how I can get it to boot. Any ideas? (I'm kinda new to this)

 

[mav@]

I am not sure this panic is related to LSI firmware update. It looks like pool corruption, that could be there for some time, but got detected now just because of reboot.

 

[FadeToBlack]

It rebooted just fine before the firmware update, so I'm guessing that was the issue. I know about 1 disk from a 3 disk raidz that had smart errors (1 bad sector), but that never stopped me from booting.
Is there any way to get passed this?

 

[mav@]

ZFS has no mechanisms to easily recover from such corruptions, since it is based on assumption that checksums should make that impossible. But there are still non-ECC RAM and other ways in this world how it may happen. Supposed way of recovery in such situation is using a backup. If there is no backup, pool may possibly still be imported in recovery read-only mode, that does not need access to the corrupted space map, to offload the data.

 

[Ericloewe]

Also, I can't seem to find the correct P14 firmware on Broadcom's site.

That's because it doesn't exist. You need P13, as I said above.

Not to be a stick in the mud, but why does this keep happening with new updates

I agree, it's really irritating.

 

[mav@]

To slightly relieve the irritation, there was no any change between those driver versions, it is just a version bump to match new firmware versions. So there is absolutely no problem to continue running previous firmware here if it works fine.

 

[Ericloewe]

To slightly relieve the irritation, there was no any change between those driver versions, it is just a version bump to match new firmware versions. So there is absolutely no problem to continue running previous firmware here if it works fine.

Ok, that's good to know.

Now, it would be nice if the people at LSI/Broadcom/Avago did it one more time to get the dammed firmware and driver versions in sync.

 

[mav@]

Now, it would be nice if the people at LSI/Broadcom/Avago did it one more time to get the dammed firmware and driver versions in sync.

They are not going to do that. In opposite, they are going to release new HBA models with different firmware starting numeration from 1, but using the same v15 mpr(4) driver.

 

[Ericloewe]

In opposite, they are going to release new HBA models with different firmware starting numeration from 1, but using the same v15 mpr(4) driver.

Someone at Broadcom needs to get fired over that.

SAS2 version numbering was great. New card? No problem, the firmware version is still P16 and it matches the P16 driver. Then they had to mess it all up by making a P21 mps driver.

 

[Kevin Horton]

To slightly relieve the irritation, there was no any change between those driver versions, it is just a version bump to match new firmware versions. So there is absolutely no problem to continue running previous firmware here if it works fine.

Please confirm which versions you are referring to. If the driver in FreeBSD is at version 14, are you saying that either firmware P13 or P14 are acceptable?

The reason I ask is that I just finished updating to P13 in the firmware, and I've still got the warning in the GUI. I'm trying to decide how high a priority to put on updating to P14.

 

[mav@]

I am talking about "13.01.00.00-fbsd" and "15.01.00.00-fbsd" version of the driver. They are practically the same, aside of few new constants defined in the new one.

 

[Kevin Horton]

I found the P14 firmware in a download for Windows.

The firmware file is named SAS9300_8i_IT.bin

Also download sas3flash.efi found here

Put both files on a FAT32 format USB stick. Insert the USB stick into the server. Reboot, go into BIOS, and change the Boot to UEFI. In the UEFI shell, change to the USB stick:

map -b # shows available drives.
fs0: # mount drive. or fs1:, depending on which slot the stick is in
ls # confirm you see the expected files
sas3flash.efi -fwall SAS9300_8i_IT.bin # checks the firmware is valid, uploads it to the SAS controller, checks the checksum of the uploaded firmware, and switches to the new firmware if the checksum is valid.

Now I've got rid of that pesky firmware alert.

Code:

# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 10.00.00.01 (2015.06.18)
Copyright 2008-2015 Avago Technologies. All rights reserved.

	Adapter Selected is a Avago SAS: SAS3008(C0)

	Controller Number			  : 0
	Controller					 : SAS3008(C0)
	PCI Address					: 00:01:00:00
	SAS Address					: 5003048-0-1cb3-6501
	NVDATA Version (Default)	   : 0e.00.00.07
	NVDATA Version (Persistent)	: 0e.00.00.07
	Firmware Product ID			: 0x2221 (IT)
	Firmware Version			   : 14.00.00.00
	NVDATA Vendor				  : LSI
	NVDATA Product ID			  : SAS9300-8i
	BIOS Version				   : 08.31.00.00
	UEFI BSD Version			   : 15.00.02.00
	FCODE Version				  : N/A
	Board Name					 : SAS9300-8i
	Board Assembly				 : N/A
	Board Tracer Number			: N/A

	Finished Processing Commands Successfully.
	Exiting SAS3Flash.

 

[heidi]

Why did you flash to P14?
P13 (FW13.00) for driver mpr0 14.

Just as a good practice, use -cpci, especially if you have more than one controller.
#sas3flash -o -f SAS9300_8i_IT.bin -b mptsas3.rom -cpci 00:01:00:00

P13 (FW 13.00) was delivered with BIOS version 08.31
P14 (FW 14.00) was delivered with BIOS version 08.33

If there was an alert with P13 on -U3, maybe another bug?
I'm still on -U2 with P12 (FW12) without any warning - on a downgraded 3008 with 12.00/08.31.

 

[asmodeus]

I had the same warning after upgrade during boot and switched back to the previous version for now. It's quite disconcerting that changes suggesting a firmware upgrade were introduced with an update release. If there were no actual changes in the driver, what's the significance of the user facing warning in the gui? Will I be fine with the P12 version, which was fine before the upgrade?

If not, what's the recommended procedure to upgrade sas controller firmware for an (encryped) zfs pool? Do I need to do more than shutdown, disconnect hd cables, upgrade, connect cables?

Edit: typo