Freenas install on ESXi VM to VT-d Passthough to USB connected Directly to ESXi Host BIOS Boot prob

eilz

Cadet
Joined
Jul 20, 2012
Messages
5
This may sound a little confusing, but please bear with me (I have exhausted all searching, trust me). Setup is aimed at using only 1 machine with Passthrough of most components (VT-d), Currently :

1) I boot ESXi from a 8GB USB Device (works fine)

2) I then passthrough the second USB Bus channel and connect a second 4GB USB Device to a free USB Slot on the same ESXi host/PC

3) I create a VM on the ESXi Host, with no disk, but add the "PCI Device" (passthrough device, which is actually the second 8GB Lexar USB Device mentioned in point 2) to the VM settings>add.

4) I start VM, and attach the ISO of Freenas to boot and install freenas onto this second passthough Lexar 8gb USB Drive, it recognises the key ok, as a "lexar usb" on "da0" and seems to install Freenas ok.

5) Once completed, I cannot boot to this Passthough'd USB Lexar 8gb PCI device although Freenas did install onto it) >>I found ESXi does not support booting from a VM via its bios to a passthrough USB device that's connected directly to the ESXi host.<<

6) I then tried to run PLOP iso to boot the VM, and select run from USB, but PLOP does not find PCI Device (aka Lexar 8GB USB) which has the Freenas install. BUT if I boot this USB Stick directly from the physical PC (so esxi doesn't load) it runs the Freenas startup, so the install on the USB key seems ok.

7) If I go back and boot the FREENAS iso Install again, while loading it sees the PCI Device usb key as Lexar USB on "da0" but from the install ISO, the only option on the FREENAS setup is to install/upgrade or shell, reboot etc, I want an option to run Freenas but from the identified "da0" device, is this possible?

I was hoping to be able to select the "Shell" option, and somehow run the Freenas installation from da0 (the PCI Device Passthrough Lexar USB which has the Freenas install setup) Is this possible, can I do some kind of mount and run from "da0" while in shell?

I know the install has worked on the USB Key, so I guess what I am after is like a Freenas Live CD but with an option to boot from "da0" and continue to load Freenas, is this possible?

I hope this makes sense to anyone out there. In case your wondering why?

I want one Physical Machine to run both Esxi from USB (which works), then one VM to run Freenas from another USB key via ESXi passthrough (so I still haven't used any physical harddrive yet)..... I then want to Passthrough all my Physical Drives to the Freenas VM and setup ZFS, and afterwards configure ESXi to map a datastore (either iscsi or NFS) via the Freenas VM but still allow Freenas full direct control of all the drives by passthrough these also as PCI Device(s).

So just to clarify:
-one Physical PC, with ESXi on USB bootable (ok)
-one VM with Freenas running on a Passthrough USB key (need way to boot to Passthrough PCI Device)
-Freenas configured with say 6 drives (ZFS) physical Drives using PCI Passthrough via ESXi
which is then configured to connect to the Freenas VM datastore (created from th 6 drives) either via ISCSI or NFS. This would then allow direct access, so if I wanted to pull out 6 drives and connect them to a dedicated Freenas box PC, it would be able to read the data ok (as no vmdk files are created and everything is Passthrough configured)

I hope this is clear. I understand the complexity, but I want to also adapt this to a laptop setup to make it portable, and I think I am almost there, if I can get the Freenas Passthrough USB to boot.

p.s. - I don't want to use RDM/vmdk file or connect the USB on any other machine other than the Esx HOST directly, I have also tried the formating a USB key with VMFS and tried to map to ESXi but that is unsupported and not working either. I tried all version of ESXi 5.0 al the way to 5.5 with all updates/patches with no luck. I also read all the article suggesting no the install Freenas as a VM, but remember as everything is passthrough its actually all physical though somewhat risky I know.

Appreciate any help on this.

Regards
Eilz
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It is fine to use a VMware virtual disk as your FreeNAS "USB stick."

Your ESXi needs a datastore of some sort to host the VM configuration files for FreeNAS, so it can hold the VMware vmdk for that virtual disk as well.
 

eilz

Cadet
Joined
Jul 20, 2012
Messages
5
I trying to avoid vmdk creation or installation on harddrive and/or esxi datastore, trying to setup both ESXi and Freenas purely on two Passthrough USB-Drives.
Then passthough all harddrives directly to the VM Freenas is install on (which would be a USB key in passthough as well). Aim is to not create a VMFS datastore but map to the NFS or Iscsi directly from the Freenas VM with the install running off the USB passthough device.
Hope that makes sense
Thanks for quick reply though.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Can't be done. You NEED a datastore of some sort to store the FreeNAS VM files on, even if you are able to use a passthru USB device to boot from.

VMware has helpfully decided that a datastore cannot be a USB flash drive so ... crap.
 

eilz

Cadet
Joined
Jul 20, 2012
Messages
5
I understand the ESXi limitations, but if the Freenas has installed fine to the passthough USB, If I use utility like plop it does not identify the PCI Device (which has the usb key with freenas installed fine via passthough), but if I boot the Freenas install CD it does pickup the device and allows a full freenas install ok.

So I don't see why I can't create some kind of boot disk iso (similar to plop) to then run the freenas install from the stick rather than get the install cd for Freenas that only lets me install again and again.

Surely if I can identify the device on the "shell" setting, then hopefully I can run the Freenas(boot) off the USB device and make this work? (i guess i am looking for a backdoor on this, but I think the only thing I'd need it a boot disk that picks up and run from the usb in passthough outside of the vm bios which is why I was trying to use plop)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I don't understand the goal here.

Where are you storing the VM files for the FreeNAS VM? ESXi must have a datastore available in order to launch a VM.

If you have a datastore available, then it would seem using a few GB of it for a virtual disk to act as the FreeNAS boot device would seem to be an easy solution.

I do not see why you would want to make such a convoluted and complicated bootstrap procedure when it does not appear to gain you any tangible benefit.
 

eilz

Cadet
Joined
Jul 20, 2012
Messages
5
Main reason is to be able to dedicate two ssd drives for passthrough to freenas which would also boot from a usb device pass'd through (say the device was a laptop with vt-d support but only able to connect two ssd drive to its sata port). I dont mind using a boostrap plop style iso if it can boot freenas to usb pass though via esxi freenas vm. I dont want to store vm files for the freenas vm, theres no disk attached to the vm settings in esxi for the freenas vm. I just want to boot and write directly to pass through usb key to read/write. If the freenas can install to it surely I can boot to it somehow. Basically, on a laptop I would aim to mirror two ssd drive in the laptop using freenas mirror raid having full direct controll to pci bus, so could pull out the two ssd and attached to physical box pc and be able to read contents on a desktop. Hope that makes sense.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Okay, you're ignoring my big question here:

WHERE DOES THE VMX FILE FOR THE FREENAS VM RESIDE?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
You know, this isn't the ESXi forums. You should take your question to them.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
The idea was to use something like the below link, by attaching a USB drive and allow ESX to recognise it as a Datastore to store the vmx files onto.

http://nybbl.blogspot.co.uk/2013/08/vmfs-formatted-usb-sticks-in-vmware.html

OK.. so let me ask these questions...

1. Do you really think that trying to twist ESXi to bend in the ways you want is a good idea for your server's reliability?
2. How about your virtual machine's reliability?
3. Are you about to rely on a feature that suddenly might change with an update and not do what you are wanting to do? (Hint: The author of that link is hoping it might change for the better.. so the answer is an implied "yes")
4. Are you sure that once you stop the usbarbitrator you're ever going to be able to pass through ANY USB stick? (From what I'm reading you either get to passthrough USB devices or you do your "hack" to make no USB device of any kind can be passed through.)

This stinks so badly of being a "bad idea" I'm going to unsubscribe to this thread. I really don't need to hear more about how a bad idea is just getting worse(if that is even possible)...

Good luck in your endeavors. If you do this I hope to freakin' God you have thorough and complete backups... Or maybe not... I do make good money recovering destroyed pools...
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The real problem here is that he's got really hung up on this idea of passing through the USB key. If this is going to work at all, find an older large USB key, make a datastore on it, and then put a FreeNAS VM and a vmdk for the FreeNAS boot device. There isn't an alternative; if you are killing off usbarbitrator to make your hacky solution work, then you cannot have it both ways - you cannot give a VM a USB device AND also have usbarbitrator not running.

Generally I would say this is an exercise in determining how far you can take stupid and dangerous simultaneously. That you can tease something into working is almost never the same thing as being able to have a high confidence level in what you're doing. Put your two SSD's on a frickin' RAID controller, mirror them, install ESXi on that, and be done with it. Pass through your spinny rust and you're done. You can even create a VMware virtual disk on top of that and export it to FreeNAS for use as a SLOG if needed. There's no doubt that this works without dodgy hackery involving cajoling ESXi into accepting a marginal datastore.

Be sure to read the sticky on virtualization ("absolutely must virtualize freenas") for the way to do this right.

As for adapting it to a laptop, that also sounds like an exercise in badness. ESXi isn't designed for that. Use VMware Workstation and call it a day (he says, typing on a nice 16GB laptop that has VMware Workstation installed).
 
Top