X10SL7-F and LSI 9211-8i card and disk detection problems.

[fukie]

Hi Guys,
I have been using consumer parts and have recently purchased SM X10SL7-F and accompanying Crucial ECC RAM to move my FreeNAS build to server grade parts.
I also have an existing LSI 9211-8i card that I had from my previous build which I'm moving over to the new build.
Build wise, everything seems good, RAM checks out after running 20 passes with no errors. However, with the LSI 9211-8i plugged in, the SAS MPT boot ROM does not detect it (whether in the x16 or x8 slot). So I thought it could be a version issue. The 9211-8i was originally on v16 IIRC and the onboard SAS 2308 was on v15. At this point, the SAS 2008 still could detect any HDDs plugged into it. And the motherboard either detects either the 2308 or 9211 but not both. So I try to upgrade the SAS 2308 to v16 with no issues.
After the FW upgrade, still no change. So I decided to fiddle more around with the BIOS. Not much change, but there was a glimmer of hope when I disabled the "On Board SAS Option ROM". The MPT boot ROM. I noticed this option only works when the 9211 card is in the x8 slot. So it seems with this option disabled. The 9211 ROM is loaded which detects both 9211 and 2308
I plugged in all my HDDs and give it a whirl. But to my dismay, I noticed the SAS 2308 only detected one HDD. I swapped cables and HDD and the same thing happens. It only detects on HDD in the first port. Removing the 9211 and loading optimised defaults in the BIOS
I checked for latest FW for both chipsets, the 9211 had a v19 but I could only find v16 for the SAS 2308. Updated the 9211 to v19 and 2308 again to v16 but no change. The 2308 still could detect one HDD. I have also enabled "On Board SAS Option ROM" which didn't change anything.

1. General hardware information (CPU, RAM, Motherboard model, etc.).
   1. Supermicro X10SL7-F
   2. Intel G3430
   3. Crucial 16GB Kit (8GB x2) CT2KIT102472BD​160B
   4. LSI 9211-8i
   5. Nothing plugged in except for a USB drive for FW upgrade and a KB.
   6. LAN cables plugged in on both Intel ports and the IPMI port.
   7. Norco 4224 case
   8. Seasonic Platinum 1000W (I know its overkill and I plan to downgrade it)
   9. Reverse breakout SFF8087 - SATA cables (Onboard SAS2308 to Norco backplane) and SFF8087 - SFF8087 cables (9211-8i to Norco backplane)
   10. Mixture of WD Green, WD Red and a Hitachi drive.

One thing I also noted was that sometimes when the 9211-8i was plugged into the x16 slot, it would get some PCH / PCI error (can't remember what it was but something to do with addressing) while loading the onboard 2308 ROM. I guess its either I did not seat the card correctly or the card was fighting for resources with other devices on the motherboard.
I guess the 9211-8i detection problem is secondary as I may have a work around by disabling "On Board SAS Option ROM" (I'm not sure what would the impact but please advise if I should keep it enabled and if enabled, how would I get the motherboard to detect the 9211 card?
Most importantly, can anyone advise how or have any solution to the onboard 2308 detecting only one HDD?
For your kind advice
fukie

 

[Ericloewe]

That's an unusual issue.

Why do you want the boot ROM? You're not booting off the controllers, are you? It's only useful for booting (in IT mode).

You'll want to match the firmware to FreeNAS's driver version, too (currently v16, I believe).

Have you tried removing the LSI 2008?

Is it always the same drive that is recognized?

What does FreeNAS say regarding the drives? It kinda sounds like you're just messing around with the controllers' boot ROMs and system BIOS, and those are mostly irrelevant for FreeNAS's use of disks.

We need a description that is as accurate as possible. Something like "Drives x, y, z are recognized on controller 1 with controller 2 removed. Adding controller 2 makes only x recognizable."

 

[fukie]

Yea its quite weird and I searched around and can't seem to find much people with similar issues.

Don't I need the boot ROM for the motherboard to identify and locate the drives so I can subsequently use it with FreeNAS?

Noted, so I have to downgrade the FW for my LSI 9211 to v16.

Yep, I tried and to no effect, still detects the first drive only on the onboard 2308.

I never tested swapping out the port 1 drive but I did swap the cables around between two backplanes and it detects on the SATA0 drive and not the remaining ones.

I did not boot up FreeNAS yet as I believe FreeNAS would not be able to see the drives if the boot ROMs can't load properly and detect the drives. Correct me if I'm wrong here.

There isn't much description, for the onboard 2308, it will show the normal boot ROM stuff, checking for connected devices etc... Then shows only the SATA0 drive details and off to the next screen.

With the LSI 9211-8i and onboard 2308, it will load show the boot ROM and do the same process for the LSI 9211, shows the 7 drives connected, then move on to the onboard 2308 which shows only the SATA0 drive.

 

[Ericloewe]

Don't I need the boot ROM for the motherboard to identify and locate the drives so I can subsequently use it with FreeNAS?

[...]

I did not boot up FreeNAS yet as I believe FreeNAS would not be able to see the drives if the boot ROMs can't load properly and detect the drives. Correct me if I'm wrong here.

As I said, the boot ROM is only used to boot from drives attached to the controllers, it's an extension of the system BIOS which can be executed.

Boot FreeNAS and you'll most likely see all drives functioning.

Next time you flash the controllers, don't include the boot ROM, it'll save you a few seconds at boot.

 

[fukie]

Ah okies.

So I don't need the boot ROM to detect the drives in FreeNAS? That's new!

So when I flash the controller using the efi tool, I use erase level 6? Then flash the FW only and leave the boot ROM out?

sas2flash.efi -o -e 6
sas2flash.efi -f 2308IT16.ROM

Thank you.

 

[indy]

The controller works fine without the boot-rom in IT-mode.
Dont know about the erase level though, I used 7 and reprogrammed the sas-address.

 

[fukie]

Hmm. I didn't remember seeing anyone mention about going boot ROM-less for FreeNAS.
I shall give it a whirl later tonight.

Noted, seems like the difference between erase level 6 and 7 is that 6 leaves the manufacturing area while 7 does a full wipe.

What about the SAS-address thing? Why the need to reprogramme it? Do I use my existing SAS address on both devices or make up a new one?
Is the command for the action you are referring to this -> sas2flash.efi -o -sasadd xxxxxxxx:xxxxxxxx?

 

[fukie]

I've just re-updated the FW for my 9211 and 2308 without boot ROM.
Ran a new version of FreeNAS, imported one pool and now doing a scrub.

These are my commands for anyone's reference:
I'm using the EFI shell on the mobo and v16 FW for both chips, -c is not needed if you are using only 1 controller.

I recommend doing one FW upgrade first, restart into FreeNAS, check whether the HDDs appear and if alright, repeat for the subsequent controller if any.

1. sas2flash.efi -o -listall (to see all my controllers and their ID number. Also take down the SAS address especially if you have more than 1 controller on board)
2. sas2flash.efi -o -c 0 -e 7 (-c to select controller using ID, starts from 0 which in my case is my 9211, -e is to erase and 7 is a full erase)
3. sas2flash.efi -o -c 0 -f 2118it.bin
4. sas2flash.efi -o -c 0 -list (to check firmware version is correct, at this point of time, you will notice the SAS address are 0s, so means the wipe is good)
5. sas2flash.efi -o -c 0 -sasadd 0123456789012345 (16 alphanumeric characters without colons)
6. sas2flash.efi -o -c 0 -list to check the SAS address is updated correctly.

For the onboard 2308, I used the v16 FW from SM website

1. sas2flash.efi -o -listall (to see all my controllers and their ID number. Also take down the SAS address especially if you have more than 1 controller on board)
2. sas2flash.efi -o -c 1 -e 7 (-c to select controller using ID, starts from 0 which in my case is my 9211, -e is to erase and 7 is a full erase)
3. sas2flash.efi -o -c 1 -f 2308IT16.ROM
4. sas2flash.efi -o -c 1 -list (to check firmware version is correct, at this point of time, you will notice the SAS address are 0s, so means the wipe is good)
5. sas2flash.efi -o -c 1 -sasadd 0123456789012345 (16 alphanumeric characters without colons)
6. sas2flash.efi -o -c 1 -list to check the SAS address is updated correctly.

Thanks all for helping me realise the boot ROM for the controllers is not necessary and FreeNAS can still run normally. Another 6 more hours before the scrub is finished but everything is looking good.

 

[tscholak]

Hi, I've been experiencing very similar difficulties with exactly the same controller configuration.
I wrote both supermicro and lsi, but they were unwilling to help
because the combination of a 9211 and the x10sl7 has apparently not been validated by their labs ...
Here's my email to lsi's tech support:

Dear Sir or Madam:

i am trying to use a 9211-8i on a supermicro X10SL7-F board with build-in 2308 chip. this works somewhat, but not reliably, bear with me while i explain the situation.

i flashed both controller bioses to an IT firmware.
that's 16.00.01.00 with bios 07.31.00.00 in case of the 9211-8i
and 16.00.03.00 with bios 07.23.01.00 in case of the onboard 2308.
(i also tried the most recent IT firmware 19.00.00.00 and bios 07.37.00.00 on the 9211-8i without improvement,
supermicro doesn't offer yet anything newer than 16.00.03.00 for the 2308.)

the 9211-8I is seated in CPU SLOT6 PCI-E 3.0 X8 (IN X 16) and has PCI address 00:01:00:00. the onboard controller has address 00:02:00:00.

both controllers work flawlessly on their own.
i tested and ran the 9211-8I in another machine (w/o onboard sas) without problems. the 2308 also works well in solo configuration from what i can tell.

in post, the bios of the 9211-8I initializes, i can enter it and change the configuration. i cannot access the bios of the 2308 at all.
sometimes, not for every boot, but with somewhat high probability,
I see the message "mpt bios fault 01h" twice during post,
once prior to the 9211-8I bios messages, and once after.
the 9211-8I bios does not list the 2308 in its configuration screen.
i tried disabling the 2308's option rom in the board's bios without improvement. this way i can only get rid of the "mpt bios fault 01h" messages.
my conclusion is that "mpt bios fault 01h" is issued by the 2308's bios
that for some reason fails to initialize...

i would call the functionality of both controllers in this configuration erratic. it alternates from operational to non-operational between reboots.
maybe ~1/2 of the time it does work, ~1/2 of the time it doesn't.

when it does work, the mpt2sas linux driver (version 16.100.00.00) initializes both controllers and all of their devices.

the times when it does not work are 100% correlated with
the occurrence of the "mpt bios fault 01h" messages.

if this is the case, mpt2sas cannot initialize the controllers
and the kernel tells me that mpt2sas is in "fault_state(0x2622)".

the erratic nature of the problem means i usually can get both controllers to work by resetting the machine. once they work, they remain functional
at least until the next reset.

i also tried reseating the 9211-9i into SLOT5 (PCI-E 2.0 X4 IN X 8), but then the 9211 is not operational and not detected during post. the situation is the same as if it weren't present at all.

i hope you can help me,

Sincerely Yours

Since I didn't get a helpful response,
I eventually had to disable the onboard controller by setting a jumper.

It surprises me that you were able to get both controllers to work by flashing only firmware and not bios.

Can you tell me whether you ever got any "mpt bios fault" messages?
Did you try to disable option roms for specific pci-e slots prior to deleting the bioses?
What's the bios version of your x10sl7?

Cheers,
t

 

[Ericloewe]

Just don't use the boot ROMs. They're useless if you're in IT mode and not booting from the LSI controllers. If you absolutely want to boot from an LSI controller (which you shouldn't with FreeNAS, as thoroughly documented), installing the boot ROM in only the controller that you'll be booting from should also avoid the problem.

 

[Z300M]

I have an M1015 that I had already flashed to IT mode (which I think makes it the equivalent of a 9211-8i) installed in a Supermicro X10SL7-F motherboard with its LSI controller flashed to IT mode. All the drives (8 connected to the M1015, the other two to on-board SAS ports) are detected by FreeNAS 9.2.1.3, but doing Ctrl-C during boot shows me only the drives connected to the M1015 -- no sign of the other drives, or even of the on-board controller, at all.

Is this expected behavior?

 

[tscholak]

Is this expected behavior?

i don't think so.
i might be wrong on that, but i expect that the controllers negotiate during post and only one bios (the newest one or the one for the controller with the lowest/highest pci id) initializes and then manages all controllers and devices. so, after pressing ctrl+c, you would see a list of all controllers and, after selecting one, the respective drives attached.

 

[Ericloewe]

I have an M1015 that I had already flashed to IT mode (which I think makes it the equivalent of a 9211-8i) installed in a Supermicro X10SL7-F motherboard with its LSI controller flashed to IT mode. All the drives (8 connected to the M1015, the other two to on-board SAS ports) are detected by FreeNAS 9.2.1.3, but doing Ctrl-C during boot shows me only the drives connected to the M1015 -- no sign of the other drives, or even of the on-board controller, at all.

Is this expected behavior?

Does the LSI 2308 have the boot ROM flashed?

Honestly, I don't know what the expected behavior in BIOS is. The simplest way to avoid problems is to have both controllers flashed to the same firmware version (if the boot ROM goes looking for other controllers, it might freak out if it sees a newer firmware version on the other controllers) and only flash the boot ROM in the controller from which the OS will be booted.

 

[fukie]

Hi tscholak,

I rmb seeing some message about mpt bios fault, I suspect it could be because of the seating of the LSI 9211 card on the PCIx16 slot. It disappeared after I moved it to the PCIx8 slot and did not come back after I put it back to the PCIx16 slot carefully. May I need a bit a of a nudge here and there maybe? I can't 100% attribute this to the seating because lack of sleep these few weeks has made my head fuzzy and having really bad short term memory some times.

But do try on the PCIx8 slot, I was sure the fault message disappeared after I moved it there.

I believed after indy's and ericloewe's replies, I think I reset the motherboard BIOS to mostly defaults, so I didn't disable any option ROMs.
As I found out that the two controllers didn't need their boot ROMs to function normally in IT mode. So right now, I don't have access to the interface via CTRL+ C.

Although I still wasn't certain it will work without boot ROMs, I did a FW flash without boot ROM on the 9211 first. And loaded FreeNAS up afterwards to find my HDDs connected to the 9211 appearing in FreeNAS (the HDDs on the 2308 weren't powered up at this point). So I repeated it for the onboard 2308 and viola. So yep, the whole boot ROM, CTRL+C isn't required. Its some what scary not knowing whether your HDDs are detected on boot but yea, SMART, temperatures works fine, my pool was imported without any hiccups, scrubs are good.

I didn't notice my BIOS version for the motherboard, but from IPMI View, there is a FW version at 1.4.2 and IPMI version at 2.0. SM website lists the latest at R2.0. So 1.4.2 could be my BIOS version.

Honestly, I don't know what the expected behavior in BIOS is. The simplest way to avoid problems is to have both controllers flashed to the same firmware version (if the boot ROM goes looking for other controllers, it might freak out if it sees a newer firmware version on the other controllers) and only flash the boot ROM in the controller from which the OS will be booted.​

​

​

I think Ericloewe is right here, and I got a feeling that both the 9211 and 2308 has some issues working together, so removing the boot ROM might be the best way to ensure the whole chain of devices have less interference as possible​

i might be wrong on that, but i expect that the controllers negotiate during post and only one bios (the newest one or the one for the controller with the lowest/highest pci id) initializes and then manages all controllers and devices. so, after pressing ctrl+c, you would see a list of all controllers and, after selecting one, the respective drives attached.

This is something I thought would do so too but like I mentioned in the previous paragraph, I think they have compatibility issues. I rmb reading on this forum that the X10s were not that compatible / stable compared to the X9s.​

Even with the v16 FW flashed on both controllers, I noticed the NVDATA between both controllers were different, 10.00.00.06 on the 9211 and 10.00.00.04 on the 2308. (Not sure whether its meant to be the same or different since they are different controllers). I think the v16 FW might have been different on the last 2 digits but I didn't take that down. Which was a little worrying but I just decided to assume they were the same =X​

 

[tscholak]

Hi fukie, thx for getting back to me.

But do try on the PCIx8 slot, I was sure the fault message disappeared after I moved it there.

I tried the other slot earlier (called SLOT5 in the manual; it's x8 mechanically and x4 electrically), but that made things worse. The controller didn't work at all in there and I switched back. While seated in slot 5, it had a higher pci id than the onboard 2308 controller.

Although I still wasn't certain it will work without boot ROMs, I did a FW flash without boot ROM on the 9211 first. And loaded FreeNAS up afterwards to find my HDDs connected to the 9211 appearing in FreeNAS (the HDDs on the 2308 weren't powered up at this point). So I repeated it for the onboard 2308 and viola. So yep, the whole boot ROM, CTRL+C isn't required. Its some what scary not knowing whether your HDDs are detected on boot but yea, SMART, temperatures works fine, my pool was imported without any hiccups, scrubs are good.

I reflashed both controllers now following the same procedure you outlined above. Now it looks like that:

I didn't notice my BIOS version for the motherboard, but from IPMI View, there is a FW version at 1.4.2 and IPMI version at 2.0. SM website lists the latest at R2.0. So 1.4.2 could be my BIOS version.

I decided not to buy a supermicro bios product key for the board and did the bios update by hand.
That was in a file called "x10sl74.424" dated April 24, 2014. No idea if it's the version 1.4.2.

My preliminary verdict is that without the lsi bioses, both controllers seem to work,
I know for sure though only after a couple of reboots... I come back when I know more.

So long,
Torsten

 

[cyberjock]

Looks like you flashed v19 firmware... FreeNAS doesn't use the v19 driver so you shouldn't be using the v19 firmware. ;)

 

[fukie]

I tried the other slot earlier (called SLOT5 in the manual; it's x8 mechanically and x4 electrically), but that made things worse. The controller didn't work at all in there and I switched back. While seated in slot 5, it had a higher pci id than the onboard 2308 controller.

That is true. I also noticed that. The 9211 controller won't work in the x8 slot because the 2308 controller is the primary one as it has a higher PCI ID. And I believe only one boot ROM will be used if two boot ROMs are detected.

Looks like you flashed v19 firmware... FreeNAS doesn't use the v19 driver so you shouldn't be using the v19 firmware. ;)

Yep, you should be using v16 for both controllers. The v16 FW for the 9211 can be found on the SM website. (Tip, you need to go to to the support then downloads rather than the product and downloads. From the support -> downloads page, you can find the older versions which are archived. The product -> downloads page only shows the latest version which is v19.

 

[tscholak]

Yep, you should be using v16 for both controllers. The v16 FW for the 9211 can be found on the SM website. (Tip, you need to go to to the support then downloads rather than the product and downloads. From the support -> downloads page, you can find the older versions which are archived. The product -> downloads page only shows the latest version which is v19.

Thx for the tip. I got some firmwares and I have now the following choices:

1. The original 9211-8I IT firmware version 16.00.00.00 from the lsi support website (2118it.bin dated Mar 13, 2013). The next newer version downloadable from lsi is already a P17.
2. Some 2008 IT firmware from the supermicro ftp server, version 16.00.01.00 (2008it.rom dated Apr 26, 2013). I already had it on the 9211-8I for a while. It works, but it isn't necessarily made for the card.

I also found a P16 IR firmware from Dell (?!) for the 2008 chip. The only interesting thing about it is that it's version 16.00.03.00 and thus the same version as i currently run the 2308 with. But it's IR and thus useless for my purposes. Ideally I would like to have v16.00.03.00 on the card. But, without knowing what the differences between these versions are, it's hard to know whether it would make any difference. The minor version change indicates bug or compatibility fixes.

 

[tscholak]

just for the record:
i have been using the original LSI IT firmware 16.00.00.00 on the 9211-8I for a while now without problems.
both controllers work fine. indeed, the trick was to not flash any controller bioses.

 

[jjstecchino]

Glad it is working for you guys. I also "upgraded" to a x10sl7 from an x8 mobo. I have a m1015 flashed to 9211-8I fw 16, no bios attached to an intel SAS expander on a separate enclosure. This was working perfectly on the old mobo. The x10 2308 is flashed to fw 16 no bios.
Everything is recognized ok but 50% of the time on reboot I get a mps0 (2003 controller) IOC Fault 0x4002622. and kdb crash.
When freenas completes boot everything works super. Very frustrating.
Care to share your x10 pci bios settings? maybe tinkering with the bios can improve things. can improve things.
By the way I went through all the same bios conflicts you guys noted but I figured out before reading this post that going w/o bios improved things.
I am not sure if my m1015 is bad or just there is some sw incompatibility issue with this mobo. I favor the latter as this controller worked fine on the old mobo. I am pulling my hair here, google did not help so far, any advice appreciated