ESXi FreeNAS with Passthrough controller GEOM confusion

[grahambunce]

Hi,

I'm trying to follow best practice on running FreeNAS under ESXi, after my previous N54L server died, I've reinstalled FreeNas on an ESXi VM (version 8) with a dedicated Highpoint Rocket 640L 4 port SATA controller. This is running a Marvell 88E9230 6 Gb/s controller. I was a little wary of the Marvell, but I couldn't afford an LSI/IBM and others appeared to be running with the 88E9230 chipsets ok.

When I plug my old FreeNAS install into the new server (i.e. bare-metal) everything works ok. The 4 disks on the 640L are recognised as before, as ada0, ada1, ada2 and ada3. When I switch the server over to run under ESXi and FreeNAS in the VM the array is corrupt as 2 disks are missing.

"View Disks" shows that ada1 and ada2 are seen, but ada0 and ada3 are not.

Therefore it appears that under VMWare, even when using a Passthrough controller (that should then hand over control to the VM), GEOM names are being lost.

gpart status shows the following:

Code:

Name, Status, Components
ada1p1	  OK  ada1
ada1p2	  OK  ada1
ada2p1	  OK  ada2
ada2p2	  OK  ada2

Therefore it looks like the controller Ports are somehow being mapped to the same GEOM name, which is why it can only see 1 of the disks in that "port-pair".

gpart show only shows the 2 disks from ada1 and ada2

Code:

		  34  3907029101  ada1  GPT  (1.8T)									
		  34		  94		- free -  (47K)								
		 128	 4194304	 1  freebsd-swap  (2.0G)							
	 4194432  3902834696	 2  freebsd-zfs  (1.8T)							
  3907029128		   7		- free -  (3.5K)								
																				
		  34  3907029101  ada2  GPT  (1.8T)									
		  34		  94		- free -  (47K)								
		 128	 4194304	 1  freebsd-swap  (2.0G)							
	 4194432  3902834696	 2  freebsd-zfs  (1.8T)							
  3907029128		   7		- free -  (3.5K)

glabel status shows:

Code:

gptid/854cbbfe-542e-11e6-af00-9cb65402132b	 N/A  ada1p1					
gptid/855fc91b-542e-11e6-af00-9cb65402132b	 N/A  ada1p2					
gptid/a55f1f6a-a147-11e6-a968-9cb65402132b	 N/A  ada2p1					
gptid/a57a3352-a147-11e6-a968-9cb65402132b	 N/A  ada2p2

i.e. the 4 disks are seen via their gptids (I think, the numbers look suspiciously close)

"Volume Manager", to create a new Raid from my backup, only sees 2 disks (ada1 and ada2)

I assume that VMWare is messing up the GEOM names as these are based on the controller ports... but is there any way to fix this? The array is actually fine, but the GEOM names can't be mapped correctly so it appears to have failed. From what I can tell, I can't re-build an array on GPTID names, I need correct GEOM names.... but I don't know how these get created.

I'm still new to FreeNAS/Linux so am a bit unsure whether I can use gpart create to do this manually, and it would then be usable in the FreeNAS gui...

I'd like to keep the existing array but if necessary I can re-create and restore from a backup.

Is there a recommended way forward on this?

Thanks

 

[dlavigne]

Did you find a solution for this?

 

[m0nkey_]

glabel status shows:

Code:

gptid/854cbbfe-542e-11e6-af00-9cb65402132b	 N/A  ada1p1					
gptid/855fc91b-542e-11e6-af00-9cb65402132b	 N/A  ada1p2					
gptid/a55f1f6a-a147-11e6-a968-9cb65402132b	 N/A  ada2p1					
gptid/a57a3352-a147-11e6-a968-9cb65402132b	 N/A  ada2p2

i.e. the 4 disks are seen via their gptids (I think, the numbers look suspiciously close)

"Volume Manager", to create a new Raid from my backup, only sees 2 disks (ada1 and ada2)

You're not seeing 4 disks, you are only seeing two, ada1 and ada2, which is why your pool is unavailable. The p1 or p2 indicate the partition number, in this case you are seeing partitions 1 and 2 on ada1 and ada2.

What do you see when you run camcontrol devlilst?

Edit: Can you also confirm your ESXI machine support VT-d and that the HBA is indeed passed though to the VM?

 

[grahambunce]

No, I'm afraid not yet. It's quite confusing really.

 

[grahambunce]

You're not seeing 4 disks, you are only seeing two, ada1 and ada2, which is why your pool is unavailable. The p1 or p2 indicate the partition number, in this case you are seeing partitions 1 and 2 on ada1 and ada2.

What do you see when you run camcontrol devlilst?

Edit: Can you also confirm your ESXI machine support VT-d and that the HBA is indeed passed though to the VM?

Ah ok. I did wonder why the gptids were slightly different for a couple of the disks.

Yes, my ESXi supports VT-d, and it's turned on. It's an HP ML110 G6 running the Xeon processor. It looks like the HBA is passed through... being a bit stupid, how do I check this? In ESXi I've set the controller as a passthrough, assigned it to the VM as a PCI card... so I think I've done everything right...

 

[Spearfoot]

Ah ok. I did wonder why the gptids were slightly different for a couple of the disks.

Yes, my ESXi supports VT-d, and it's turned on. It's an HP ML110 G6 running the Xeon processor. It looks like the HBA is passed through... being a bit stupid, how do I check this? In ESXi I've set the controller as a passthrough, assigned it to the VM as a PCI card... so I think I've done everything right...

Of course you'd check the VT-d settings in the BIOS.

You can check the passthrough using VMware vSphere. Navigate to Configuration, then select Advanced Settings. You should see your HBA listed as a "device available for direct access by the virtual machines on this host." Here is a screenshot from one of my systems:


If you click on Edit..., you'll see all of the devices on your system which are available for passthrough, like this:


You might double-check this to see if your HBA is actually two devices...

Also, it's a good idea to tweak your FreeNAS VM's CPU and Memory settings under the Resources tab. I set the CPU with 'High' Shares and a 500MHz reservation; I reserve/lock all of the VM's memory. Screenshots below:


Hope this helps...

 

[grahambunce]

Thanks for these (and the VM advice). The passthrough looks ok:



Camcontrol shows the following;



I think this is saying it can see the 4 ports on the Marvell, but can't detect disks on 2 of those ports?

What's odd is that the 640L has 2 banks of 2 ports. I've worked out FreeNas detects 1 disk in a port in each bank; so if I swap disks around appropriately i can get my array back on line, but degraded as 2 disks are missing. Obviously, I'm still stuck as I can't get FreeNas to detect those disks as they are on the "missing" ports.

 

[grahambunce]

Missing image... the VM looks to be setup correctly too (I think.)



I see you're running VM Version 11, and I'm still on Version 8 (I use the free vSphere client and haven't tried to hack it to update it.... i read somewhere you can do this via a config setting).... maybe this is something to do with it?

 

[Spearfoot]

Missing image... the VM looks to be setup correctly too (I think.)

View attachment 15294

I see you're running VM Version 11, and I'm still on Version 8 (I use the free vSphere client and haven't tried to hack it to update it.... i read somewhere you can do this via a config setting).... maybe this is something to do with it?

I really don't believe the VM version matters.

I'm stumped. At this point I'd throw in the towel and buy an LSI 9211-8i / IBM M1015 / Dell H200, which are known to work well in this application.

 

[grahambunce]

Thanks for the help anyway. I think I have the problem, based on this https://www.v-front.de/2013/11/how-...howComment=1389913922093#c5396818548275219283.

I vaguely recall seeing this post, but got to the bit of "If you're using VT-D" then ESXi doesn't get involved.

This additional post I have now found suggests that older versions of the Marvell controllers get screwed up when using VT-D due to a bug in their firmware. I was hoping it would be fixed in the version I got, but apparently not.

It's odd, as the Marvell works fine with VT-D and bare-metal, but not VT-D and VM Ware. I will try with a Hyper-V host, but I suspect it's just the combination of Marvell and ESXi that's causing a problem.

 

[grahambunce]

Just to close this off for anyone else... The Marvell has a definite firmware bug with VT-D. In the end I bought an IBM M1015 off e-bay, added some SAS-SATA cables and flashed it as an LSI with IT. It didn't work, until the flash... then it worked fine.

Thanks for the input anyway.