SOLVED WARNING: ... /dev/mps0

[Norleif]

"WARNING: Firmware version 15 does not match driver version 16 for /dev/mps0"

Searching for FreeBSD and mps pointed me to the MPS(4) driver for LSI SAS, which is reasonable as I have a reflashed M1015 in there.
Just to make sure, I flashed it again with the P20 IT firmware from LSI and now the warning changed to this:

"WARNING: Firmware version 20 does not match driver version 16 for /dev/mps0"

I couldn't immediately find P16 firmware on the LSI website plus I think it's a bit silly to flash back to an earlier version just to get a green status light.

 

[DrKK]

This is definitely a thing. Both the firmwares must be precisely the same version. Well known.

 

[cyberjock]

^^^ That.

It's a very common problem. Yes, the expectation is that you *will* keep the versions matching. We've had people lose their pools permanently because of the mismatch (which is what made us add the warning).

Yes, you should (and need) to reflash your card to P16. It's on the LSI website if you check the historical/archived downloads.

 

[jgreco]

This is definitely a thing. Both the firmwares must be precisely the same version. Well known.

More precisely, not "must," but "maintaining a mapping of firmware-to-driver revisions that work correctly is a nightmare, so instead we do this simpler, safer thing."

There are definitely nonidentical pairings that are just fine, but do you really want to risk it? LSI tests and debugs its current driver against the current firmware, so that's what you should expect to work.

 

[Norleif]

Seems to be the case...
It threw a few SCSI CRC errors during boot, and broke entirely when I used IOmeter to stress an iSCSI target over 10GbE.
The console appeared to roll the credits from every Hollywood movie ever made before the machine rebooted by itself.

Flashed P16 just now but I fear I might have done permanent damage to the pool.
Apart from the benchmarking files, it was just replicated data from another FreeNAS (9.2.1.8) so there is no data loss.
I'm not cool enough to put important data on experimental releases.

 

[Apollo]

Just experimented with upgrading from 9.3 to 10 throught the GUI, just for the fun of it, and without any volume attached.
Along the way of the reboot, I think, I came across that warning, just because I had the display on at the time.
If automatic upgrade is possible between different releases, will there be any fail-safe to prevent driver and firmware mismatch on LSI based system?

 

[Ericloewe]

Just experimented with upgrading from 9.3 to 10 throught the GUI, just for the fun of it, and without any volume attached.
Along the way of the reboot, I think, I came across that warning, just because I had the display on at the time.
If automatic upgrade is possible between different releases, will there be any fail-safe to prevent driver and firmware mismatch on LSI based system?

What failsafes? You're expected to update the firmware as needed.

 

[Mlovelace]

Just looking at the "readme" included in the P16 firmware package makes me think FreeNAS should move off this particular driver/firmware version. LSI recommends moving to P17 in the documentation "Due to the possibility of this issue being experienced in the field, LSI recommends customers with the aforementioned configuration update their IR firmware at the earliest opportunity to Phase 17".


EDIT: I'll have to call LSI about this as it does say "IR firmware" not just "Phase 16" I am curious if the IT Phase 16 has the same warning/issue.

Code:

 LSI Host Bus Adapter(HBA) - LSI SAS 9211_8i

Package Contents-

Readme first note : README_9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows.txt

Installer(SAS2FLSH) : \sas2flash_dos_rel\sas2flsh Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_x86_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_x64_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_ia64_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13

Reference Guide : SAS2Flash_ReferenceGuide.pdf Version no: 2.1 Release date: JUNE-11

Firmware : \firmware\HBA_9211_8i_IT\2118it.bin Version no: 16.00.00.00 Release date: 27-FEB-13
Firmware : \firmware\HBA_9211_8i_IR\2118ir.bin Version no: 16.00.00.00 Release date: 27-FEB-13

BIOS : \sasbios_rel\mptsas2.rom Version no: 7.31.00.00 Release date: 28-FEB-13
Readme for BIOS : \sasbios_rel\mptbios.txt Version no: NA Release date: NA

----------------------------------------------------------------------------------------------------------------------------

An issue has been identified with the IR firwmare v16.00.00.00


Integrated RAID (IR) configuration can show an HDD as offline under UEFI during OS installation, causing the installation to fail

Required Configuration:

Phase 16 IR Firmware

Volume over 2TB under any OS but only with UEFI


Due to the possibility of this issue being experienced in the field, LSI recommends customers with the aforementioned configuration update their IR firmware at the earliest opportunity to Phase 17


More details are outlined in the release notes.

 

[Ericloewe]

Just looking at the "readme" included in the P16 firmware package makes me think FreeNAS should move off this particular driver/firmware version. LSI recommends moving to P17 in the documentation "Due to the possibility of this issue being experienced in the field, LSI recommends customers with the aforementioned configuration update their IR firmware at the earliest opportunity to Phase 17".


EDIT: I'll have to call LSI about this as it does say "IR firmware" not just "Phase 16" I am curious if the IT Phase 16 has the same warning/issue.

Code:

 LSI Host Bus Adapter(HBA) - LSI SAS 9211_8i

Package Contents-

Readme first note : README_9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows.txt

Installer(SAS2FLSH) : \sas2flash_dos_rel\sas2flsh Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_x86_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_x64_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13
Installer(SAS2FLASH) : \sas2flash_win_ia64_rel\sas2flash Version no: 16.00.00.00 Release date: 01-MAR-13

Reference Guide : SAS2Flash_ReferenceGuide.pdf Version no: 2.1 Release date: JUNE-11

Firmware : \firmware\HBA_9211_8i_IT\2118it.bin Version no: 16.00.00.00 Release date: 27-FEB-13
Firmware : \firmware\HBA_9211_8i_IR\2118ir.bin Version no: 16.00.00.00 Release date: 27-FEB-13

BIOS : \sasbios_rel\mptsas2.rom Version no: 7.31.00.00 Release date: 28-FEB-13
Readme for BIOS : \sasbios_rel\mptbios.txt Version no: NA Release date: NA

----------------------------------------------------------------------------------------------------------------------------

An issue has been identified with the IR firwmare v16.00.00.00


Integrated RAID (IR) configuration can show an HDD as offline under UEFI during OS installation, causing the installation to fail

Required Configuration:

Phase 16 IR Firmware

Volume over 2TB under any OS but only with UEFI


Due to the possibility of this issue being experienced in the field, LSI recommends customers with the aforementioned configuration update their IR firmware at the earliest opportunity to Phase 17


More details are outlined in the release notes.

It's only a problem using the boot ROM, by the sound of the warning. Not an issue with FreeNAS.

 

[Mlovelace]

It's only a problem using the boot ROM, by the sound of the warning. Not an issue with FreeNAS.

It's a problem with how the controller presents disks in UEFI. I spoke with LSI tech support and this is a known issue with the Phase 16 firmware and they "recommend moving to the current Phase 20 firmware/driver combination for the best drive support and optimization, however Phase 17 should be adopted at a minimum".

There's a bug report with people on P16 having disks drop: https://bugs.freenas.org/issues/6193#change-28498

 

[Ericloewe]

It's a problem with how the controller presents disks in UEFI. I spoke with LSI tech support and this is a known issue with the Phase 16 firmware and they "recommend moving to the current Phase 20 firmware/driver combination for the best drive support and optimization, however Phase 17 should be adopted at a minimum".

There's a bug report with people on P16 having disks drop: https://bugs.freenas.org/issues/6193#change-28498

Which is only relevant on boot (and explains why some many people don't see their drives when booting... One mystery solved.). FreeNAS interfaces with the controller directly using the driver, not through the BIOS, so UEFI is irrelevant at that point.

Kinda like Windows 7 would easily handle data drives that were GPT formatted, but was a pain to install to a GPT drive.

 

[Mlovelace]

Which is only relevant on boot (and explains why some many people don't see their drives when booting... One mystery solved.). FreeNAS interfaces with the controller directly using the driver, not through the BIOS, so UEFI is irrelevant at that point.

Kinda like Windows 7 would easily handle data drives that were GPT formatted, but was a pain to install to a GPT drive.

You're absolutely right, the problem isn't with freeNAS, its with the Phase 16 LSI firmware, but you are suppose to use that firmware version because of the Phase 16 MPS driver version included in the freeNAS kernel. FreeNAS itself isn't the problem at all.

 

[jkh]

To also be clear, even if we wanted to go to a different firmware version, someone at LSI would need to work with FreeBSD to update the *driver* to match. Until that happens, the point is entirely moot. Updating the firmware is only half of the battle, and we didn't write the LSI driver so it's not in FreeNAS's power to change this unilaterally.

 

[Black Ninja]

To also be clear, even if we wanted to go to a different firmware version, someone at LSI would need to work with FreeBSD to update the *driver* to match. Until that happens, the point is entirely moot. Updating the firmware is only half of the battle, and we didn't write the LSI driver so it's not in FreeNAS's power to change this unilaterally.

I have version 19 firmware I didn't even know there is a 20 as of now, but there is a driver "Free_BSD_Driver_P20" http://www.lsi.com/products/host-bus-adapters/pages/lsi-sas-9211-8i.aspx#tab/tab4 is that mean that freenas can implement the new driver and if we update to the latest version 20 of the firmware and have matching firmware with the driver ?

If not possible what is the better option then stay with latest firmware 20 or re-flash the card with much older firmware 16 ?!

 

[Brosif_My_Nif]

Same error message here. Back to P16 it is!

Good to know. I was running P17 with my 9201-16i on 9.2.x and had no idea this was an issue or that it could potentially cause problems.

 

[Borja Marcos]

Avoid firmware version 20. At least two people (me included) have experienced problems with it. Actually I upgraded a card just to cross-check some problems reported on FreeBSD and, indeed, it dropped disks like crazy when connected to a backplane with SATA disks.

Previous versions haven't been problematic.

http://lists.freebsd.org/pipermail/freebsd-scsi/2014-October/006505.html

Other than that, I've been trying firmware versions <=19 with FreeBSD 10 and I haven't seen any problems. But version 20 is problematic. Moreover, going back to 19 or an older version requires a bit of voodoo because some NVRAM data is converted to a new format that is not backwards-compatible.

 

[Black Ninja]

Avoid firmware version 20. At least two people (me included) have experienced problems with it. Actually I upgraded a card just to cross-check some problems reported on FreeBSD and, indeed, it dropped disks like crazy when connected to a backplane with SATA disks.

Previous versions haven't been problematic.

http://lists.freebsd.org/pipermail/freebsd-scsi/2014-October/006505.html

Other than that, I've been trying firmware versions <=19 with FreeBSD 10 and I haven't seen any problems. But version 20 is problematic. Moreover, going back to 19 or an older version requires a bit of voodoo because some NVRAM data is converted to a new format that is not backwards-compatible.

This is good to know. I checked change-log from p19 to p16 to is if anything terrible was fixed , but couldn't find anything significant so re-flash back to p16 , now no alerts are gone, hope that page is closed now and will be no more complications on the controllers side.

 

[Ericloewe]

The only bug we've seen that affects P16 is irrelevant: During POST, the boot ROM doesn't show attached disks. Since you're not booting from it, the boot ROM is useless anyway.

 

[laacid]

Hi, I just want to make sure I'm clear on how this issue of the firmware/driver issue was resolved. If you have an LSI card that gives this warning you should flash back to P16? I have the LSI sas9211-8i (p20) and received this error when I install the stable release of 9.3 yesterday- "WARNING: Firmware version 20 does not match driver version 16 for /dev/mps0"

The other reason I ask is because on the FreeNAS BUG REPORT site this post says that things should work with a mismatch- https://bugs.freenas.org/issues/6678

I thought the bug post was a bit contrary to the posts here and wanted to make sure it was best practices to flash to p16 to have a stable NAS on an LSI that has this issue.

 

[Black Ninja]

Well I went back to P16 on my cards , no more warning in FreeNAS.