Upgraded to 9.3 and getting WARNING: Firmware version 19 does not match driver version 16 for /dev

[mjws00]

Great. What was the fix?

 

[Ray Milyard]

Great. What was the fix?

Had to download and copy DOS4GW.exe to the USB boot disc.

 

[cyberjock]

Yeah, that's because you used FreeDOS. Unless I'm mistaken LSI doesn't recommend FreeDOS for flashing the controller.

 

[mjws00]

There's lots of guides using FreeDOS. In an effort to simplify I just didn't throw DOS4GW.EXE in the zip. I'm sure it was in the package I used, or I burned the USB a different way, used UBCD or something. So many ways to skin the cat.

Feel free to mention the tool you like better, @cyberjock. I really have no preference. Just wanted simple.

The file is in the zip for the next guy.

 

[cyberjock]

I use MS DOS from Windows 98 Second Edition. Mine is a bit custom because I make a USB stick, and with a command on my desktop I covert it to an ISO. I do this because ISOs work fine via IPMI, but I can also grab the USB stick and go.

 

[Ericloewe]

DOS4GW... now there's a name I hadn't heard in a long time. You'd think they could manage to write a flasher that fit in 640KB of RAM.

 

[Scharbag]

So much good info out there. With all the help and info on this forum, I managed to get both of my 9211-8i controllers (1 on board, 1 a PCI-E) flashed from V13IR and V12IR to V16IT. Little nervous about it but it all worked out!!

I used Rufus to make a FreeDOS bootable USB stick and placed the V16IT 2118it.bin file in the root along with the SAS2FLSH.exe file. Downloaded all files from LSI (have to do a custom search and look in the archives).

I booted the USB stick and ran the following (edited to remove mptsas2.rom as it is not needed for FreeNAS):

Code:

sas2flsh -listall
this returned 0 and 1 for the controller numbers of my 2 cards
sas2flsh -c 0 -o -e 6 (DO NOT REBOOT AFTER THIS STEP UNTIL YOU DO THE NEXT STEP!!)
sas2flsh -c 0 -f 2118it.bin
(reported success)
sas2flsh -c 1 -o -e 6 (DO NOT REBOOT AFTER THIS STEP UNTIL YOU DO THE NEXT STEP!!)
sas2flsh -c 1 -f 2118it.bin
(reported success)

Rebooted the system and all works very well.

Cheers,

 

[Ericloewe]

The mptsas2.rom part is redundant. Don't flash the ROM in the future, as it's more trouble than it's worth with FreeNAS. No point in going back and redoing the whole thing if it's working, though.

 

[Scharbag]

The mptsas2.rom part is redundant. Don't flash the ROM in the future, as it's more trouble than it's worth with FreeNAS. No point in going back and redoing the whole thing if it's working, though.

Ah Ha... So that is what is meant by not having the ROM? Makes sense. Thank you for the information!!

When FreeNAS 10 comes out (sounds like the driver might change to a new version) I will revisit this.

Cheers,

 

[Reyssor]

Hello,

Not a FreeNAS user (yet), but I feel like sharing my personal experience with an IBM M1015.
I am quite new at using ZFS, but I feel that it might save some time to other people facing the same problems in the future.

So, context: recently ordered an IBM M1015 from Ebay (2/3 weeks ago), received it quickly, no problem (the label on it says FRU: 46C8933, unlike the usual 46M0861 and 46M0831 that I read about on the web. Not sure what it means. Newer version?).

I don't have a dedicated server for it (yet), so I just put it in my desktop computer for now: the processor is an Intel i7 3770, 16GB of DDR3 RAM (so, no ECC) and the motherboard is a Gigabyte GA-Z77X-UD3H (the M1015 is in a physical PCIe 16x port, actually connected with 4x lanes). The motherboard didn't see the M1015 until I updated the BIOS (of the motherboard, not the M1015) with the latest available version.

Good, now the M1015 is recognized. Gonna flash it to IT mode. Did it with firmware P20 and the ROM BIOS. Got a few issues on the way (yay UEFI...), but I have finally been able to do it.

Connected 6 disks WD Red 6TB to it. Booted my system (Linux Mint 17 using the ZFS-on-Linux package from the PPA) and created the pool (RAIDZ2). Played with it for a week, and while I didn't lost any file, I got a bunch of weird things:

• The pool sometimes failed to mount automatically at startup.
• Writing seemed good (about 600MB/s), but read seemed slow, and scrubing was painfully slow (about 5MB/s).
• Scrubing reported a lot of checksum errors and "repaired data" (+ some read and write errors, sometimes resulting in a resilvering), even if I scrubed just before.
• zpool status sometimes reported permament errors on some files, but those errors disappeared after a partial scrub and a reboot, and the concerned files (some JPG images) looked visually OK.

There was clearly a problem somewhere. Memtest looked Ok, but when I fired smartctl on the 6 disks, I saw that they all had their UDMA CRC error count around 80'000 (after only one week. The disks were brand new). And that number kept increasing when using the disks.

Disconnected 3 of the disks from the M1015 and connected them directly on the motherboard. Launched a scrub again and kept an eye on those CRC errors. The disks connected on the M1015 detected more of them, the disks connected on the motherboard did not.

• Switched the M1015 to another PCIe port: no change
• Changed the SFF-8087 cables (maybe 1m was too long, I changed them for 50cm): no change

I searched on the web and ended up on this topic, where 9C1 Newbee recommended P16 firmware. At this point, it was worth a try. So I reflashed the M1015 with P16 (and without the ROM BIOS this time). And yeah, you guessed it, it solved my problem: no more CRC error and my scrubing is now at 400+MB/s.

So while I can't say for sure what was the source of the problem (driver incompatibility with P20? P20 not compatible with IBM M1015 46C8933? Screwed up the flash procedure the first time? ROM BIOS present? Don't know), flashing to P16 without the ROM BIOS solved my situation.

Just for those who would be interested, I used this procedure to flash my card (reminder: I'm using Linux Mint 17):

1. Sandisk Cruzer Extreme (32GB) flash drive
2. Used GParted to create a single FAT32 partition on the flash drive (ms-dos table partition, of course)
3. Used UNetbootin to install FreeDOS 1.0 on the flash drive
4. Copied the following files from LSI at the root of the FAT32 partition: 2118it.bin, 2118ir.bin, mptsas2.rom and sas2flsh.exe (the one in the sas2flash_dos_rel folder)
5. Copied the following file from LSI at the root of the FAT32 partition: sas2flash.efi
6. Copied the following files from some tutorial at the root of the FAT32 partition: sbrempty.bin, MegaRec.exe and dos4gw.exe (I guess they are the exact same files that you can find in the other flashing tutorials for the M1015, but I have not found the officiel source for them)
7. Retrieved the UEFI Shell from here and put that file on the FAT32 partition with the path EFI/boot/bootx64.efi (apparently the path the UEFI uses seems to depend quite heavily on your motherboard, that one works for my Gigabyte GA-Z77X-UD3H)

Shutdown the computer insert the flash drive in an USB 2.0 port (for some reason, it seems the BIOS can't use USB 3.0 ports) and boot from the flash drive (normal boot mode). Start FreeDOS in live mode and I guess you know the commands, but just in case here is mine (the "C: path may change depending on your configuration I guess):

Code:

C:
megarec.exe -writesbr 0 sbrempty.bin
megarec.exe -cleanflash 0

Reboot (Ctrl + Alt + Delete) and boot again from the flash drive, but in UEFI mode this time (I get the famous "PAL" error with this motherboard with a normal boot and the EXE binary). Again the "fs0" path and the "sasadd" parameter may change depending on your system:

Code:

fs0:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605bxxxxxxxxx

Reboot on your normal system. Enjoy.

PS: Not a native English speaker, feel free to point the eventual mistakes. I will gladly correct them.

TL;DR
P20 firmware with BIOS caused problems on my system, P16 firmware without BIOS is fine.

 

[Ericloewe]

Your English is fine, don't worry about it.

Last I checked, and from what I recall, the Linux LSI SAS2 driver was at P19. P20 is said to have introduced major changes, so I imagine even P19 drivers would have trouble interfacing with P20 firmware.

The official downloads are a bit hidden on LSI's website, but they can be found in the downloads section, navigating to the product you want (LSI SAS 9211, in this case) and the going to the "archive" section of the page.

 

[mib]

Fwiw, I've had P20 firmware running in a Supermicro AOC-USAS2-L8e with the P16 driver from 9.3 and have not seen any overt problems other than the yellow warning in the GUI, but I've also only been doing the lightest of burn-in, so this is not a clean bill of health. I have a bias (borne of experience) toward keeping firmware versions up to date whenever possible; bitrot and support-rot are often worse than staying up to date.

Two questions:

1. Are there specific reasons why the FreeNAS devs aren't keeping the built-in driver version in lockstep with the official releases from LSI? P20 was released in September of 2014, and at this point should've had enough burn in to be considered Not An Intrinsically Bad Idea. I can't even find when P16 was released on the LSI site since it's scrolled off the bottom and all they're showing in the web interface are P20 and P19, ie current and current-1.
2. Since there is so much lore here about never mixing and matching drivers, has anyone installed the P20 FreeBSD driver from the LSI site into FreeNAS?

Thanks.

 

[mjws00]

The FreeNAS devs don't write the drivers. They use the FreeBSD drivers. They way they are set up they can cherry pick the best pieces and most stable driver versions from FreeBSD. There is a balance to be maintained between jumping on the "new thing." and maintaining stability within the system. So in this case we know p16 is stable and fast, we are using IT drivers so we don't need a bunch of RAID improvements, or bug fixes. So the choice to stay with "known good" is pretty reasonable.

We also know mixing and matching versions causes problems. Often we see timeout issues in the camcontrol system. But things can also manifest in random performance issues.

To get newer drivers into the system would require a custom build. Knock yourself out, there is really no significant upside or it would already be done. Feel free to document your problems running p16 linked to p20, it might take 6 months and a heavy load... but you'll see what the fuss is about eventually. Took 4 months before I saw timeouts on my box first hand.

Don't catch me wrong. I HATE and DESPISE updating firmware. But I will do it all day every day in this scenario. I was even stubborn enough to wait for issues before I bought into the theory.

 

[Ericloewe]

Fwiw, I've had P20 firmware running in a Supermicro AOC-USAS2-L8e with the P16 driver from 9.3 and have not seen any overt problems other than the yellow warning in the GUI, but I've also only been doing the lightest of burn-in, so this is not a clean bill of health. I have a bias (borne of experience) toward keeping firmware versions up to date whenever possible; bitrot and support-rot are often worse than staying up to date.

Two questions:

1. Are there specific reasons why the FreeNAS devs aren't keeping the built-in driver version in lockstep with the official releases from LSI? P20 was released in September of 2014, and at this point should've had enough burn in to be considered Not An Intrinsically Bad Idea. I can't even find when P16 was released on the LSI site since it's scrolled off the bottom and all they're showing in the web interface are P20 and P19, ie current and current-1.
2. Since there is so much lore here about never mixing and matching drivers, has anyone installed the P20 FreeBSD driver from the LSI site into FreeNAS?

Thanks.

There's plenty of experience to the contrary. Several people had errors stop when they switched to P16. Some people lost pools by using mismatched drivers/firmware.

1. The FreeBSD version wasn't at P20 last I checked, it was P19 or so. What's included in FreeBSD is P16 with cherry-picked fixes and fixes that have not been integrated by LSI, AFAIK.
2. You can't do that without recompiling FreeNAS.

 

[mib]

Thanks for the responses. I'd assumed that the driver was a blob; guess it isn't. The reasoning behind using P16 firmware also seems sound.

My next question is whether anyone has downgraded from >P16 IT to P16 IT? I just tried going from P20 => P16 and I get an error from sas2flash (both the one that comes with P16 and the one that comes with P20) about invalid NVDATA and the inability to downgrade.

Suggestions?

Thanks.

 

[Ericloewe]

Oh yeah. P20 changes things around. You'll have to clear the ROM as per the crossflashing instructions. You'll then be able to write P16 to it.

 

[mib]

The crossflashing instructions upthread seem to assume a M1015 and a (DOS?) binary called megarec, neither of which I have.

I'm currently booted from an Ubuntu LiveCD image, so all I have is the LSI sas2flash Linux executable.

`sas2flash -o -e 6` fails, as does a more mundane `sas2flash -o -e 1`.

All I get is "ERROR: Flash Operation Failed", so I don't know why it's failing.

Am I missing something painfully obvious?

Thanks.

 

[demon]

I'm currently booted from an Ubuntu LiveCD image, so all I have is the LSI sas2flash Linux executable.

`sas2flash -o -e 6` fails, as does a more mundane `sas2flash -o -e 1`.

All I get is "ERROR: Flash Operation Failed", so I don't know why it's failing.

Am I missing something painfully obvious?

Pretty sure you can't clear the ROMs from anything but either the DOS or EFI versions of sas2flash. At least, when I tried sas2flash on FreeNAS, that's what it told me with my LSI 9211-8i. It worked when I did it from the UEFI shell though.

 

[mib]

Update.

TL;DR Reyssor is a genius.

Details:

I built a FreeDOS drive using UNetbootin, and at first only used megarec to wipe the card's flash. I then rebooted into Ubuntu and tried using sas2flash to put P16 onto the card.

This turns out to not have been such a productive idea.

The megarec wipe appears to have also nuked the PCI characteristics that Linux uses a boot time to detect devices. After the megarec wipe, there's no there there, so subsequent invocations of sas2flash will return an error claiming no card is present, and therefore nothing can be done. Yipee.

I have an Intel S1200KP mobo, which also exhibits the unhelpful EFI PAL error well-described elsewhere; I've learnt not to try anything EFI-related on this machine. Being otherwise stuck, I grabbed the shell and the sas2flash.efi tools described in Reyssor's post, booted in UEFI mode, and to my pleasant surprise was able both to flash P16 (sans BIOS) onto the card and reset the SAS address.

Protips:

1. Whatever in Linux that causes it not to see a wiped card does not apply to UEFI; people with cards believed to be bricked really should try the UEFI tools. They just might bring the card back from the dead.
2. Query the card's SAS address and write it down before performing the megarec wipe in order to avoid having to open the server and read the address written in teeny tiny characters on the label on the card.

Thanks to all for the help, and for taking the time to document your efforts. This saved me a tremendous amount of trouble and effort.

FreeNAS really has an impressive community.

 

[mib]

demon, based on what I discovered, you're almost certainly right.

In hindsight, now that I know the UEFI tools work on this motherboard after all, all of this should have been possible to do purely via UEFI, no megarec or FreeDOS needed.