VT-D / PCI Passthrough with ESXi 5.5 and LSI 9240 in IT mode?

Status
Not open for further replies.

ser_rhaegar

Patron
Joined
Feb 2, 2014
Messages
358

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Gorgeous. Thank you very much for that research and link. I'm going to lock this thread now so that it remains visible as the (*cough*) "resolution" to this question.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
@Barry indicates that there's an update on this topic. Let's confine responses to useful stuff...
 

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25
@jgreco, thanks...

HP Document ID c03835455 has been updated last month; now it also says:
Starting with ESXi 5.5 Patch 4 (ESXi550-201501001), the VMDirectPath behavior has been modified to successfully enable device passthrough on the impacted HP servers. Refer to the VMware patch KB article for additional details on the release ( kb.vmware.com/kb/2099265 ). The patch can be applied to an ESXi host that has an existing HP Custom ESXi 5.5 image installed. The patch is based upon the ESXi 5.5 U2 release, so it is highly recommended to apply the patch to HP Custom Image based on ESXi 5.5 U2. Refer to http://vibsdepot.hp.com for additional details on the HP Custom ESXi images and the image contents. At the time of the publishing of this advisory, there is not an HP Custom ESXi image that integrates this patch.
http://h20565.www2.hp.com/hpsc/doc/...492883150&docId=emr_na-c03835455-1&docLocale=

I have not tried it as I'm currently running full virtual (because I couldn't get VT-D working before; I know it's bad) but I am considering trying it again.

Barry
 
Last edited:

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25
The good news is PCI passthrough (VMDirectPath) now works.

The bad news is I'm getting mfi0: Command Timeout errors, even though the controller is in IT mode. I'll find or open another thread about that.

Barry
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
The good news is PCI passthrough (VMDirectPath) now works.

The bad news is I'm getting mfi0: Command Timeout errors, even though the controller is in IT mode. I'll find or open another thread about that.

Barry

mfi0 means its a MegaRAID controller (uses the mfi driver). I could be off my rocker, but I'm not aware that there are ways to convert a MegaRAID controller to an LSI 9211 equivalent that uses the mps driver.

MegaRAID is a ball of trouble if you are actually trying to use that on FreeNAS. I've handled several calls where they used MegaRAID and lost their data as a result. They said that true JBOD wasn't support and so they did RAID0 of single disks (which isn't JBOD and was the chief reason why they had no clue things were bad until they lost their pool).
 

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
The 9240 is equivalent to the M1015, IIRC. It can be crossflashed in the same way (of this I'm sure since it's an SAs 2008).
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yes, in fact I have flashed it with a M1015 ROM... I posted more info at
https://forums.freenas.org/index.ph...card-join-the-crowd.11901/page-13#post-184021

I may have followed the wrong flashing instructions; will try again tonight.

Thanks,
Barry

Well, I'm still back to what I said earlier..

I've cross-flashed it to M1015 (or LSI9211) IT mode; The controller now says the firmware version is 2.130.394-2550.

That number is *nothing* like your standard LSI IT firmware that is used on the M1015. So I have no clue what you flashed it with (or think you flashed it with), but that firmware version doesn't exist in the IT land. AFAIK the oldest one is P7 (7.00.00.00).

So no clue what you are doing or what you think you are doing. My firmware version is 16.00.00.00, and that's not even the newest firmware out there!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well, I'm still back to what I said earlier..



That number is *nothing* like your standard LSI IT firmware that is used on the M1015. So I have no clue what you flashed it with (or think you flashed it with), but that firmware version doesn't exist in the IT land. AFAIK the oldest one is P7 (7.00.00.00).

So no clue what you are doing or what you think you are doing. My firmware version is 16.00.00.00, and that's not even the newest firmware out there!

That looks like a MegaRAID firmware (mfi) version number. It probably failed to flash or had some other misadventure that resulted in it not flashing correctly.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
And, by the way, good luck with the PCI passthru, and please, *test* *carefully* *and* *not* *quickly*.
 

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25

Attachments

  • sas2flsh-list.png
    sas2flsh-list.png
    289.9 KB · Views: 359

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25
And, by the way, good luck with the PCI passthru, and please, *test* *carefully* *and* *not* *quickly*.

Yeah, it's going to be used as a backup target and a media server, and I will be testing it thoroughly.

I'm hoping the performance is better with passthru than it was when I was using VMDKs... when I was doing large file copies to the NAS, ESXi seemed to be doing some really weird buffering and/or swapping even when there was plenty of RAM in both the host and the VM.
If that doesn't resolve, I'm going to make the HP Microserver a dedicated FreeNAS box (with the onboard SATA controllers or the LSI, undecided), and move all the VM stuff to a new box.

Barry
 

Barry

Dabbler
Joined
Dec 23, 2013
Messages
25
OK, the performance is MUCH better now, getting 99MB/s on large file transfers, with no more stalls / thrashing!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'm hoping the performance is better with passthru than it was when I was using VMDKs... when I was doing large file copies to the NAS, ESXi seemed to be doing some really weird buffering and/or swapping even when there was plenty of RAM in both the host and the VM.

ZFS tends to totally saturate systems with writes when doing large file copies, and while this is fine in the intended situation of an HBA with some disks attached, it can show up as strange performance problems in other I/O designs. It's one of the reasons you're not supposed to use a traditional RAID controller with ZFS, because the constant flushing of large amounts of data out to all drives totally blows out the write cache, in many cases actually slowing things down.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Please downgrade it to v16 of the firmware, to match the FreeNAS driver.

And feel free to rip out some clumps of hair in frustration while you're at it. Don'tchajusthateallthedetails.

"It's a detail oriented business." -me, many years ago, even truer today.
 
Status
Not open for further replies.
Top