troubles on esxi

[doraneko]

Hi guys,

After some sleepless nights I decided I need help :)

My config is as follows:

https://www.supermicro.com/en/products/motherboard/M11SDV-8CT-LN4F
64gb of ram
dell h200 flashed to it mode with these instructions https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/
4x wd red 2tb

I am trying to migrate from a gen8 microserver to this new box, but I need it virtualized so I can have some extra vm.

Esxi 6.7 manages to passthru the h200, at least it appears to be working.
However when the freenas vm boots there are no disks showing (I tested the controller on a windows machine and it's fine, I have also tested on this hw setup starting the freenas installer on bare metal and the disks show up).

I checked dmesg and there seems to be a problem. I have attached a print screen. It's the same in 11.2U7/11.3RC1.
I can't figure it out as there are not many references online on:

"mps0: IOC in unknow state 0xf0000000" or
"mps attach returned 22" or
"Device leaked MSI vectors"

Any help would be greatly appreciated

Thanks

 

[jgreco]

We advise against virtualizing FreeNAS for this sort of reason. Not every platform handles it properly, and there are dangers when you are layering one very complex bit of software on top of another very complex bit of software - everything has to work exactly right.

I do suggest you walk through every screen of your BIOS making sure that any virtualization features are fully enabled and take note of any setting that might seem to be related to your PCI card. You can try flipping these one at a time to see if there is any difference, and then Google them for details.

You can also try running ESXi 6.5 to see if it's an ESXi version thing. Not unheard-of.

If that fails, you can also try writing to Supermicro support to see if they are aware of any issues. This is often nonproductive, but it's the unexpected positive result you're fishing for.

One of the reasons we provide hardware suggestions is that some of us will already have done the heavy lifting of validating hardware compatibility. Often this validation includes virtualization under ESXi. It's always better to pick a platform that people have already found to be suitable.

If we can't fix you somehow, at least FreeNAS does support running VM's under bhyve, so you're not totally out of luck.

 

[doraneko]

Thank you for your answer. I think I narrowed it down to the motherboard. I tested the hba card on a different system with ESXI and everything works as expected -passthrough and freenas vm work fine.
I have also noticed that I can not enter HBA's bios while booting on the supermicro board, I get a strange message about a MPT BIOS Fault 01h....My only remaining option is to file a case with customer support as there are no updated mb bios files.

 

[Rand]

What are your settings for the slot (bios)? Uefi or legacy? Should be legacy I think...

 

[doraneko]

Tried both without success. I flashed both efi and normal bios on the board, separately and at the same time. No luck!
But at least the h200 card is working as expected on other systems so it means it must be the new mb (bios/firmware).

What is somehow strange is that while installing an operating system (tried ubuntu and freenas) the disks connected to hba become visible. Only ESXi has trouble (tried both 6.5u3 and 6.7).

 

[jgreco]

It's really not strange AT ALL. What's amazing is the number of boards that actually do work. There is a huge amount of complexity involved in getting the house of cards we call virtualization to work correctly, and years ago it was totally common for people to be coming into the forum with Nehalem or older systems and expecting virtualization to magically work on their board that was mostly designed before virtualization was a big thing. Even now, it's really a matter of test, test, test!

 

[Rand]

Alright so lets try to identify the actual issue...

• If you don't pass through the card to FreeNas but have it natively in ESXi - are the drives present? I.e. does the hba work fine under ESXi?

=> If so then there is no issue with the slot but with passing through.​

• Can you passthrough the HBA to a windows VM and it works?
• Does passthrough of other devices work if the HBA does not?

 

[doraneko]

Thank you for your answer,

The drives connected to the HBA are not present when esxi boots (apparently the driver is not loaded - I have noticed that during the boot up on another system with the same HBA something about mptsas shows up).
However the HBA appears in the pci list of devices and can be passed through. The same problem remains in the VM - disks don't show up.

Installing an operating system directly (no esxi) works - disks connected to the HBA show up during install (tried ubuntu and freenas).

The only difference I have noticed to running same hba on a different pc is that during boot I can not access HBA's bios "MPT BIOS Fault 01h" on my new setup.
I think the BIOS of this epyc 3201 mb has some sort of glitch (made apparent by the impossibility to enter hba bios), that only esxi is picking up.

I hope that supermicro guys will upgrade the bios soon.

 

[doraneko]

I guess I am not that lucky....My answer came back sooner that expected.
"The Broadcom (LSI) 2008 controller is very old and known to have some compatibility issues on various motherboards.
There is little we can do about this, Supermicro isn't quite willing to put much resources in trying to make an obsolete HBA compatible with current gen motherboards.
The best suggestion i can give is to update to a Broadcom 3008 based card."

Thanks everyone for taking their time, for the time being I will stay with the current microserver gen8 for freenas and use the new hardware exclusively for other vm running in esxi.
In the future when the microserver breaks down or freenas no longer works on old hardware I'll most likely take their advice.

 

[jgreco]

"The Broadcom (LSI) 2008 controller is very old and known to have some compatibility issues on various motherboards.
There is little we can do about this, Supermicro isn't quite willing to put much resources in trying to make an obsolete HBA compatible with current gen motherboards.
The best suggestion i can give is to update to a Broadcom 3008 based card."

That actually sounds like a blowoff to me. I'm pretty sure we've done a lot more support for the 2008 here in these forums than Supermicro ever has, and "known to have some compatibility issues on various motherboards" is not something that's familiar to me, the guy who wrote the LSI card sticky.. please tell me what compatibility on what boards, I've never heard of that. What *is* familiar is that this sounds like what a bunch of people with older boards had when trying to virtualize FreeNAS years ago... the mainboard BIOS support is off somehow and corrupting the MSI setup or something like that.

I'm no longer as young as I once was and I no longer enjoy debugging things like interrupt conflicts or other fundamentals -- in part my memory for arcane details isn't what it once was, haha. It's bad enough on a bare metal install, and trying to figure it out when you've got ESXi in the middle is no fun, because everything has its fingers in the pie. These days I usually leave it to the device driver authors to get it all worked out. There's some MSI configuration tricks you might try listed from years ago in the forum but I'm not finding it offhand.

Best guess here is that there's something a bit off with the mainboard BIOS. Whether or not substituting in another controller of a different type will work is questionable.

On the flip side, using the board for hosting VM's should be pretty amazing. And if Supermicro releases a BIOS in the future and suddenly passthru works, please feel free to come back and post an update in this thread. I really would like to know. Small NAS-capable boards are of keen interest.

 

[kdragon75]

I would remove the bios from the card unless you NEED to boot from attached disks. You just don't need it and having it loaded could be your issue.

 

[doraneko]

At first I tried without bios, once everything else failed I flashed bios (leagcy and/or efi).
There is a slight chance that the flashing of a dell h200 to lsi 9211 leaves some unwanted bits of data on the card. Maybe an original broadcom/avago 2008 would work, but at this point I feel like I had enough of it :)

 

[Rand]

Well given that 3008's can be had for 50 bucks every now and then I don't think it warrants spending any more time on that ;)