Register for the iXsystems Community to get an ad-free experience

Virtualize FreeNAS on VMWare ESXi 6 on HP Microserver Gen8 for home NAS

Not open for further replies.


Apr 5, 2016
0. Overview
There are many posts and articles about this topic with detail tutorials and guides already; however, in this post, I just want to address the specific topic for those who want a simplest hardware installation like me.

The hardware that you want to install is:
- HP Microserver Gen8 with 16GB RAM and a CPU support VT-d
- No extra RAID controller
- A PCIex network card with 1 or 2 ports
- 2 or 4 HDDs for storage
- SSD or other 2.5" HDD for installing ESXi (since I don't trust the quality of USB stick, I want to stick with HDD/SSD)

1. Why virtualization?
I'm not satisfied with all the home modems/routers firewall features so I need a "soft" firewall with more powerful features. I also need a good separated web server for other in-house applications. So, I need more than one machine. Also, for home NAS, it doesn't make sense to have such a powerful CPU for just NAS, I want to exploit it anyway.

Yes, I do read all the articles about NOT virtualizing FreeNAS and I acknowledge the risks but this topic is not to answer the question of should or shouldn't virtualize FreeNAS. This topic is just for solving some problems that you have when trying to install a home server similar to my usecase.

2. Hardware preparation
I bought a Xeon E3-1220v2 (support VT-d) for cheap price on eBay to replace my Celeron G1610T (doesn't support VT-d). The original heatsink is designed for 35W CPU only, however, I got the CPU fan from a Dell Precision M60 (which I bought for 20 Euro to make a DiY digital frame) so I installed it there. And as the result, the normal temperature with full 4 cores fullspeed is around 40C. In full load, it's around 55C which is far too good.

The schema for my situation is:
- Boot ESXi from SSD drive
- Make the rest of SSD a local storage for installing FreeNAS VM, firewall VM, web server VM.
- Passthrough all other HDDs in front bays to FreeNAS VM
- Passthrough 1 internal NIC to firewall VM as WAN connection
- The other internal NIC as management interface for ESXi
- 2 ports on PCIEx NIC as LAN to connect to home network e.g wireless AP

The problem comes with the onboard SATA/RAID controller. The server can't boot from the ODD SATA connection if you install HDDs in the front bay. Since you want to passthrough the SATA/RAID controller to FreeNAS VM, you can't create a second logical drive for the ODD connection. You can install ESXi to SSD with logical drive schema, but when ESXi booted up, it can't see the drive anymore (you passthroughed it already) so you have no local storage to install FreeNAS on. And I just come with an idea to use a SATA-to-USB converter to convert the SSD to USB drive and use the internal USB 2.0 connector. Yes, I do know about wasting the performance of SSD when downgrade it to USB speed but in this case, it's only slow down when booting ESXi (how many reboot a year that you do with your ESXi?), FreeNAS and the firewall. However, as many people booting ESXi and FreeNAS from USB sticks, it would not be the big problem.

So, everything is set up. Let's start.

3. Installation
Start install ESXi from DVD (if you have one) or from USB stick. Right after you press Enter to choose Install ESXi (the boot menu), press Shift + O to open the boot option command line, then add:

runweasel formatwithmbr

(the runweasel is already there, you just need to add space and formatwithmbr). The reason why we have to use MBR instead of GPT because simply the HP Microserver Gen8 doesn't support booting GPT from USB device. Then follow the screen guide for installation, please be sure to choose your SSD/HDD to be installed.

Now, start your ESXi and config its management interface so that you can access it from your computer. Either using vSphere Client or using the web UI by accessing:

https://<your management interface IP>/ui

You can see that in the Storage, there is no datastore. I don't know why when installing to USB device, ESXi doesn't automatically create the rest of your HDD/SSD a local datastore as when installing directly to the SSD/HHD. So we have to do it manually. Enable SSH access to your ESXi and make an SSH session there.

First we have a look at the current partition schema by using partedUtil. Although fdisk is said to use with MBR but in this case, fdisk will destroy your HDD as it cannot read the partition table correctly.

[root@localhost:~] partedUtil getptbl /vmfs/devices/disks/<device name>

In my case:

[root@localhost:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba32\:C0\:T0\:L0
14593 255 63 234441648
4 32 8191 4 128
1 8192 1843199 5 0
5 8224 520191 6 0
6 520224 1032191 6 0
7 1032224 1257471 252 0
8 1257504 1843199 6 0

I guess, you will have the same schema as me. So the info shows us that the disk is format with MBR (msdos). The second line are the quantity of cylinders, quantity of heads, quantity of sectors per track and quantity of sectors. Pay attention to the quantity of sectors because we will want to use up this number.

Now, create a new partition number 2 which fill up the free space you have. To create new partition, the command is:

partedUtil set "/vmfs/devices/disks/DeviceName" ["partNum startSector endSector type/guid attribute"]


partedUtil set "/vmfs/devices/disks/DeviceName" "1 128 500000 251 0"

It means, create partition number 1, starting from sector 128, ending at sector 500000 with the type vmfs datastore (251), no additional attribute. In my case:

partedUtil setptbl "/vmfs/devices/disks/mpx.vmhba32:C0:T0:L0" msdos "1 8192 1843199 5 0" "4 32 8191 4 128" "5 8224 520191 6 0" "6 520224 1032191 6 0" "7 1032224 1257471 252 0" "8 1257504 1843199 6 0" "2 1843232 234441647 251 0"

Note that, you have to enter the whole schema here so re-enter the existing schema and add the partition you want to create because each time you run the setptbl command, it will delete all the current partition schema. Then, verify it:

[root@localhost:~] partedUtil getptbl /vmfs/devices/disks/mpx.vmhba32\:C0\:T0\:L0
14593 255 63 234441648
4 32 8191 4 128
1 8192 1843199 5 0
5 8224 520191 6 0
6 520224 1032191 6 0
7 1032224 1257471 252 0
8 1257504 1843199 6 0
2 1843232 234441647 251 0

Now we format it:

vmkfstools -C vmfs5 -b BlockSize -S DatastoreVolumeName /vmfs/devices/disks/ DeviceName:Partition

In my case:

vmkfstools -C vmfs5 -b 1m -S LocalDS /vmfs/devices/disks/mpx.vmhba32\:C0\:T0\:L0\:2

Voila, now come back to the UI, wait a little bit and refresh the storage to see the local datastore appears. Then, click on Manage > Hardware, check on the SATA controller and the internal NIC that you want to set passthrough and click on Toggle Passthrough button. Restart the server for changes to take effects.

Create new FreeNAS VM, give it at least reserved 8G of RAM (because we will use passthrough, you need to reserve it), store the vm disk on the local datastore and add the PCI SATA controller device. Now, you will see the error when trying to start the VM:

Power On virtual machine:The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at xx:xx.x

It's a bug in the UI when adding passthrough device (I hope VMWare will fix it soon). SSH to your ESXi server and issue this command:

lspci -v

And looking for your SATA Controller device. In my case:

0000:00:1f.2 SATA controller Mass storage controller: Intel Corporation Cougar Point 6 port SATA AHCI Controller [vmhba0]
Class 0106: 8086:1c02

Note the number 8086:1c02. Now edit your VM setting by editing:

[root@vmHost] vi /vmfs/volumes/LocalDS/HomeNAS/HomeNAS.vmx

Note: LocalDS is my local datastore created recently and HomeNAS is my FreeNAS VM. In that file, find the vendor ID and device ID and change it to the correct one if any of them are not correct. = "00000:000:31.2"
pciPassthru0.deviceId = "0x1c02"
pciPassthru0.vendorId = "0x8086"

Now start the VM and install FreeNAS as usual. I ran FreeNAS on this server for a year already. After install the FreeNAS VM, restore the config and all of my data is back there again.

I was thinking about Openmediavault instead of FreeNAS and tried it. However, I missed all the nice features from FreeNAS so I decided to find a solution for this case. And for the firewall, in my opinion, you should try the Sophos XG Firewall Home Edition - the most generous free firewall ever.

This solution fits to my case and I don't know if it is the best practice to use or not. So I am welcome to any feedback or comments to make it a better solution.



Resident Grinch
May 29, 2011
It'll be entertaining to recover the system when you have a problem. There's a reason we advise strongly against this. That box makes a reasonable FreeNAS, but not virtualized. You really need to have a competent datastore of some sort available for ESXi, and creating a big bodge to get a datastore is hazardous.
Not open for further replies.