SOLVED Help with SATA Controller Passthrough from ESXi 6.5 to FreeNAS

[djwarwick]

Hello all, I’m new here and I’ve run into a roadblock. I’m having an issue passing through my onboard SATA controlled from ESXi 6.5 to a FreeNAS VM and was hoping I might get a little help. Before I go any further I would like to provide a little background on what I have and have done up to this point. I bought an old SuperMicro 1U server to play with intending to setup a small home virtualization lab. I’m not an IT guy by trade but consider myself an enthusiast with a reasonable bit of experience. That being said this is my fist go at both vitalization and FreeNAS.


As I mentioned I have a SuperMicro X8DTU-F Server with the following hardware: (https://www.supermicro.com/manuals/motherboard/5500/MNL-1081.pdf)

2 x Xeon L5520 @ 2.27GHz

48 GB DDR3 ECC Ram

2 x 4TB Ironwolf HDD (Installed in Server Drive Bays attached to the onboard Intel ICH10 Controller)

120 GB Intel SSD Boot Drive (Installed on a separate PCI Card: www.amazon.com/dp/B01452SP1O/ )


I have installed ESXi 6.5 on the SSD and it boots and functions as it should. I have successfully deployed a VM’s with data stores located on the SSD along with one for FreeNAS. FreeNAS boots as it should running as a VM. I will also use a VM as an NVR with the intention of FreeNAS providing the storage for video recordings. In the end I would like to setup FreeNAS to serve media files and provide a backup storage location for photos and the like.

Before I got too far into the project I read the great post by @jgreco on “how not to lose your data” when running FreeNAS in a virtualized environment along with the complete .pdf guide for newbies on how to setup FreeNAS and not make stupid storage decisions. If I can get all this working as I expect I plan on adding two more 4 TB drives to the server and setting the 4 drives up in Z2 array. This is just a home setup, so nothing mission critical here, but I would like the emphasis to placed on reliability and data security.

After reading the good advice offered in the “how not to lose your data” post (and after screwing up my first ESXi deployment) I wiped my boot drive, and first installed FreeNAS on a bootable USB drive, loaded FreeNAS and created a mirrored volume on the two 4TB drives. That all worked like it should confirming the hardware is all compatible and it now provides me fail over to recoved my FreeNAS data should my virtualization environment go to crap. With that out of the way

My plan thus far has been to pass-through the entire onboard 4-port sata controller to FreeNas and then share that storage back to the other VM’s as needed. As part of the effort to reach that goal I added the PCI card to host the SSD boot drive so that it would be physically separated from the other drives. When logging into the ESXi host and managing the hardware there I now see 3 sata controllers:

1. Intel ICH10 4 port SATA IDE Controller (I assume to the be the controller for the 4 hotswap drive bays on the front of the server)

2. Intel ICH10 2 port SATA IDE Controller (I assume to be the controller for the onboard DVD drive and an auxiliary port?)

3. ASMedia Technology ASM1062 Serial ATA Controller (As this is nested under a PCI Express Root Port I assume to be the PCI controller for the SSD

All 3 of these devices show to be capable of passthrough and I am able to set them as such. I have enabled passthrough on the 4 port controller, gone into the FreeNAS VM and added the controller as a PCI device. After tweaking the RAM settings and setting the full amount of allotted RAM to be reserved for the VM I can get the FreeNAS VM to boot as it should. But the problem is it still does not see the 4TB drives installed in the machine. This is where I am stuck.

I understand that virtualized FreeNAS is not an overly popular topic and I am stretching my skill set trying to pull this off, but I feel like I’m pretty close and I’m hoping that someone might be able to chime in with an idea (or tell me where I’ve gone terribly wrong). Thank you in advance for reading my post and for any assistance you can offer.

-Dennis

 

[gpsguy]

Pick up a supported HBA and flash it to IT mode. The IBM M1015 is one of them, but there are PERC, LSI, etc. Do a forum search.

 

[garm]

Since you already created the pool when you where running bare metal FreeNAS you won’t necessarily see the drives in the way you expect. Have you tried importing the pool you created?

 

[djwarwick]

Thanks for the replies so far, I'll try and add a little more information.

@garm Unfortunately I have tried to import them, but the problem seems to be that FreeNAS doesn't physically see the disks to begin with. This was also the case when I previously installed ESXi first, then FreeNAS as a VM and tried to pass the controller through. Regardless of which comes first, when virtualized it just doesn't see anything attached to the controller when passed through; on bare metal it worked fine.

@gpsguy I appreciate the suggestion and have seen many references to that HBA card but my preference would be to make the hardware I already have work if possible. I read up on the topic a good bit before I got started and it has been mentioned in other posts that many onboard controllers passthrough just fine so that ultimately is what I'm trying to do. It seemed to me that because the CPUs are VT-d compatible and because ESXi allows the SATA controller to be set in passthrough mode and because FreeNAS will talk to the drives attached to controller when setup on bare metal that the it would still be able to do so when virtualized. Or was you intent for me to interpret your suggestion as statement that my hardware is simply not compatible? Given that my configuration meets (at least on the surface) the prerequisites for virtualization compatibility and that it works on bare metal I don't understand what I'm missing. If it can't work then so be it, but I sure would like to understand why.

Thanks again for the input so far.

 

[djwarwick]

The more I keep researching more it just seems like the answer keeps coming back to "if it should work but still doesn't it's probably your hardware" and try different specifically supported hardware. I still can't come up with a good reason why the onboard sata controller won't properly passthrough, but my ESXi-Fu is not strong so I've ordered an IBM1015 as suggested above. I've gone this far down the rabbit hole so what's another $50 bucks, right? Just to be clear, that card needs to be flashed with version 20 of the IT firmware as described on this site for FreeNAS 11, correct? I found an older post about some confusion regarding a driver version 21 and firmware version 20, but it's over a year old. Please correct me if I am wrong but the best I can tell version 20 is still the most current. I'll report back once the card comes in next week. Until then I'm still open to any other suggestions or trouble shooting steps anyone might have to offer.

 

[gpsguy]

Correct.

that card needs to be flashed with version 20 of the IT firmware as described on this site for freeNAS 11, correct

That's part of the reason for my response.

If you virtualize FreeNAS, you'll be in the minority among all the FreeNAS users on this forum. Most of the forum regulars will recommend a solution that we know will work, ie. a supported HBA. Could you get your SATA controller to work with virtualized FreeNAS? Maybe. In addition to searching the FreeNAS forums for an answer, you may need to look at the VMware one's too.

But, if a $50 card fixes the issue and is supported - *I* would go that direction.

I've gone this far down the rabbit hole so what's another $50 bucks, right?

 

[toadman]

Let's start with the basics. Before we get to the disks, are you sure the FreeNAS VM sees the controller you passed through? What is the output of pciconf -lv? You should see the SATA controller in that list.

 

[djwarwick]

@toadman Thanks for chiming in! It does appear that FreeNAS sees the controller, the last item returned after running pciconf -lv is this:

Code:

atapci1@pci0:3:0:0:	 class=0x01018f card=0x060015d9 chip=0x3a208086 rev=0x00 hdr=0x00						   vendor	 = 'Intel Corporation'																								
   device	 = '82801JI (ICH10 Family) 4 port SATA IDE Controller'																
   class	  = mass storage																									
   subclass   = ATA

Is this a good thing that FreeNAS is able see the controller or a bad thing because it can't see what is attached to it?

 

[toadman]

Yes, it's a good thing. It means the pass through worked. Make sure your controller is set to ACHI in your system's BIOS.

So now what do you get when you do the following commands? I would expect it to show your disks.

Code:

# camcontrol devlist
# geom disk list

 

[toadman]

Also, since you said this above:

1. Intel ICH10 4 port SATA IDE Controller (I assume to the be the controller for the 4 hotswap drive bays on the front of the server)

2. Intel ICH10 2 port SATA IDE Controller (I assume to be the controller for the onboard DVD drive and an auxiliary port?)

3. ASMedia Technology ASM1062 Serial ATA Controller (As this is nested under a PCI Express Root Port I assume to be the PCI controller for the SSD

I would say "don't assume." Make sure you know the drives in question are connected to the right ports on the motherboard for the 4 port controller you passed through. Verify the physical connections. The data sheet for the motherboard should tell you which 4 are for the ICH10 sata controller. You should see the cable go from there to wherever your drives are located.

When you said:

first installed FreeNAS on a bootable USB drive, loaded FreeNAS and created a mirrored volume on the two 4TB drives

Where are those drives plugged in? Can you verify they are in the ICH10 4 port controller? Or are they plugged into the 2 port controller, in which case they wouldn't be seen by the freenas VM since the VM can't see the 2 port controller.

EDIT: Or pass through both ICH10 controllers so all 6 motherboard SATA ports are available to the Freenas VM. (Unless you need 2 for a local ESXi datastore.)

 

[Chris Moore]

On some boards, the only controller that can be passed through is one in a slot. Usually, you can not split a controller and pass part to the VM with part for the hypervisor.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[joeschmuck]

On the X11SSM-F-O motherboard you cannot pass through a single SATA controller, it's all or nothing. So I tried to pass though all the ports and use an add-on card for my boot device, nope, the system would not boot from this add-on card connected SSD. It might now since the BIOS has been updated a few times since the initial release. So I was stuck with placing my hard drive storage on the add-on card and passing that through and all those motherboard connectors as being wasted, what a sad story.

The other option is RDM, it works and a few of us here run it, but it's not the promoted path. You can pass through individual hard drives and there is a few commands you need to use in order to achieve this. It adds complications into replacing a drive for failure, or in other words you really need to know what you are doing and pay attention. I would advise against using RDM but in the spirit of sharing knowledge I think you should be aware it does exist. I personally use RDM only on my backup FreeNAS box and for me it's more of a test that a permanent solution. If I loose data on this box, I can live with it because it's my backup to my main FreeNAS box. I would not use RDM on my main box but others have.

If you find out that you cannot pass though your motherboard controller then I completely agree with this comment below.

But, if a $50 card fixes the issue and is supported - *I* would go that direction.

Good Luck.

 

[Chris Moore]

Please correct me if I am wrong but the best I can tell version 20 is still the most current.

The IT mode firmware version number is actually longer that that: 20.00.07.00
I think they were paid by the number of zeroes... ;)

 

[djwarwick]

Thank so much for all the information guys, you've really helped make the new guy feel welcomed here! I apologize for the delayed response but I've been traveling with family and have been away from home and my VM host machine. While I will still be away for a couple more days I'm finally in a place where I can connect remotely to my server. So while I can't try out an HBA card just yet I can provide some feedback on your suggestions.

@toadman - I was able to confirm that my assumptions regarding port groups and their respective controllers were correct. The onboard SATA ports on my MB are split into two groups. Ports 0-3 are included in the 4-port controller, ports 4 & 5 are designated to the 2-port controller. I have the two 4TB drives connected to ports 0 & 1 and the factory installed DVD drive is connected to port 4.

Where are those drives plugged in? Can you verify they are in the ICH10 4 port controller?

The drives were plugged into the ports 0 & 1 (of the 4-port controller group) when I installed FreeNAS on bare metal onto a USB drive. Those drives have remained physically untouched in their same location and connected to the same ports.

EDIT: Or pass through both ICH10 controllers so all 6 motherboard SATA ports are available to the FreeNAS VM. (Unless you need 2 for a local ESXi datastore.)

Because of what @joeschmuck and @Chris Moore said about passing through all the ports or nothing and just to be on the safe side, I tried passing both controllers; the results in FreeNAS were unchanged. The only difference being the local DVD drive was no longer available to ESXi. The interesting thing to note from this was it seems it will allow for passing through individual controllers. Because if I set only the 4-port controller on passthrough it falls out of the view of ESXi (as expected) but I can still use the DVD drive connected to the second 2-port controller in ESXi.

When I run # camcontrol devlist I get this in return:

Code:

<VMware Virtual disk 2.0>	at scbus2 target 0 lun 0 (pass0,da0)

and # geom disk list returns:

Code:

Geom name: da0															
Providers:																
1. Name: da0															  
   Mediasize: 17179869184 (16G)											
   Sectorsize: 512														
   Mode: r1w1e2															
   descr: VMware Virtual disk											  
   ident: (null)														  
   rotationrate: 0														
   fwsectors: 63														  
   fwheads: 255

so it appears it still doesn't see the drives attached to the controller despite it seeing both controllers.

Code:

atapci1@pci0:3:0:0:	 class=0x01018f card=0x060015d9 chip=0x3a208086 rev=0x00 hdr=0x00									  
   vendor	 = 'Intel Corporation'																						  
   device	 = '82801JI (ICH10 Family) 4 port SATA IDE Controller'														  
   class	  = mass storage																								  
   subclass   = ATA																										  
atapci2@pci0:11:0:0:	class=0x010185 card=0x060015d9 chip=0x3a268086 rev=0x00 hdr=0x00									  
   vendor	 = 'Intel Corporation'																						  
   device	 = '82801JI (ICH10 Family) 2 port SATA IDE Controller'														  
   class	  = mass storage																								  
   subclass   = ATA

I appreciate @joeschmuck mentioning RDM and my fear is the additional difficulties it introduces in the event of a drive failure. I read somewhere that someone said that if you need to ask for help with RDM you probably shouldn't be using it. As I clearly don't know much about it that is a bit of advice I plan on following.

Thank you to @Chris Moore for the IT firmware version number clarification.

So at this point, it seems that I can successfully pass through one or both of the cards so that they are visible to FreeNAS but something is preventing the VM from seeing what is attached to them. I don't believe its a driver issue in FreeNAS as it all worked as it should on bare metal, so something continues to get lost in virtualization translation. Any other ideas save for the HBA card?

 

[Chris Moore]

As I mentioned I have a SuperMicro X8DTU-F Server with the following hardware: (https://www.supermicro.com/manuals/motherboard/5500/MNL-1081.pdf)
2 x Xeon L5520 @ 2.27GHz
48 GB DDR3 ECC Ram
2 x 4TB Ironwolf HDD (Installed in Server Drive Bays attached to the onboard Intel ICH10 Controller)
120 GB Intel SSD Boot Drive (Installed on a separate PCI Card: www.amazon.com/dp/B01452SP1O/ )

Are you still using the hardware you listed in the original post?
I am not a pro with ESXi, there are many others on the forum like @Stux that are much more versed, especially when it comes to passing hardware into the VM. The thing that occurs to me is that some of the older system boards and CPUs did not properly support PCIe pass through. The firmware of the system board needs to be updated to the latest version, but it might even require a newer model CPU (for that board) if it is possible to do this thing on a board that old.

 

[djwarwick]

Yes, the hardware remains the same. But that is an interesting comment regarding the lack of support for PCIe pass through on older hardware. I'll have to read up on this and see if I can find any more information on the subject. I'll also confirm that I am currently running the most current BIOS available on my MB, but I feel like I checked that before I got started. As far as the processor goes, the MB manual specifically states compatibility with 5500 and 5600 series Xeons, I'm not sure about compatibility beyond that, but I'll do some research. I'd doubt that going to a 5600 series CPU from the 5520's I'm currently running would make any difference as they are from the same era, but if someone knows otherwise please let me know.

 

[Stux]

Also, being able to pass-through chip-set sata controllers is supposedly hit & miss. FWIW, I was able to pass-through the SATA controller built-in to the Xeon D, but I had to modify some files in ESXi, and the XeonD is a broadwell generation (X10) chip... so much newer than X8.

Here's my build/esxi setup thread:
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[djwarwick]

@Stux, Thanks for your input and the link to your build thread. There is a lot of good information in there that I look forward to digging through. Do you happen to recall what ESXi files you had to modify or in more general terms why the modification was required to make pass through work? I'll have the HBA card to fall back on, but I really don't want this onboard controller to beat me!

 

[gpsguy]

Here's a message from @Stux -> https://forums.freenas.org/index.ph...he-x10sdv-tln4f-o-bad-idea.46915/#post-395008

Do you happen to recall what ESXi files you had to modify

 

[djwarwick]

I did all the reading I could on ESXi compatibility with my hardware and whatever information I could find on device passthrough. There wasn't much I could find specifically on my hardware but there was a lot of general information out there. The biggest problem seems to be devices not showing up as passthrough capable and edits that need to be made to set them as such; much like the issue @Stux had which @gpsguy linked to above. The most frustrating part was that was never my problem. My onboard SATA controllers showed as passthrough capable from the start and acted like they were playing nicely... save for not actually showing the drives attached!

After I beat my head against the wall for over a week, it seems that gpsguy's initial suggestion was correct:

Pick up a supported HBA and flash it to IT mode. The IBM M1015 is one of them, but there are PERC, LSI, etc. Do a forum search.

I got a hold of an IBM M1051 and cross flashed it following these directions: https://forums.freenas.org/index.ph...-no-lsi-sas-adapters-found.27445/#post-221154 I pulled the SATA cables running to the MB and replaced them with a SAS to 4-Port SATA cable plugged into the new HBA card. I went back into ESXi, configured the card for passthough, edited the VM settings and added the newly pass through'd card as a PCI card to the VM. And Acrabadabra... It just works! The drives show up in FreeNAS, my previously created volume imported without a problem and everything just works.

There was clearly a not-overly-obvious hardware incompatibility that presented as if it would work but simply refused to do so. Thank you so very much for everyone that chimed in, it was certainly a learning process.

Final question, now that it all everything seems to be on the up and up, is anything else I need to be aware of regarding running drives off the HBA in FreeNAS. I mean are there any settings I need to tweak or any pitfalls I need to be aware of?

Thanks again for everyone's help.