ESXi with TrueNAS vs Proxmox

[badincite]

So currently I'm running ESXi 6.7 update 3 and I was looking to setup a 5 drive raidz datastore. My first route was thinking to completely move away from ESXi and switch to proxmox. But I started seeing post about running TrueNAS as a VM on proxmox which made me think why not just do it on ESXi. Looking at all the dates on post an documentation this looks like it 's something that just started happening. Are there any quirks with with running it virtualized? Playing with proxmox I'm having issues with write speed dropping off inside VM's and not being able to add the full disk as a drive. Planning on using this datastore as my plex database.


Current Hardware
MB: EP2C602-4L/D16
CPU: Dual E5-2680's
GPU: GTX 1060 Passthrough to VM
MEM: 128GB ECC
HBA: Dell Perc H200 ~ I will be flashing this into IT mode
HD: 5x8TB Drives ~ 2 WD Red Pro's & 3 WD Gold

 

[sretalla]

You should read this in either case:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

No passthrough of individual disks if you care about your data, you need an HBA/discrete SATA controller to pass through to the TrueNAS VM in its entirety.

 

[danb35]

you need an HBA/discrete SATA controller to pass through to the TrueNAS VM in its entirety.

...and it seems that this technology, while available in Proxmox, just isn't very mature there.

 

[badincite]

Yep I have a H200 HBA currently so shouldn't been a problem. Definitely feeling this option more than proxmox now.

 

[homer27081990]

I don't know about Dell, but my HP DL380P Gen 8.1 from that era with its HP "smart" Array SAS-SATA-III 8-port controller could not show drives in ESXi correctly (or, maybe, ESXi could not see them). I would suggest to use test drives if possible first, unless you currently do not have a plex database.

 

[badincite]

I don't know about Dell, but my HP DL380P Gen 8.1 from that era with its HP "smart" Array SAS-SATA-III 8-port controller could not show drives in ESXi correctly (or, maybe, ESXi could not see them). I would suggest to use test drives if possible first, unless you currently do not have a plex database.

Currently using the H200 HBA for a couple of the drives ESXi 6.7 recognizes it no problem its on Vmware's compatibly list. But I would be switching this to a passthrough direct to the TrueNAS vm. Right now my database is across multiple drives that's what I'm trying to get away from to have a single drive that I can replace a drive when it fails.

 

[homer27081990]

The problem is not

Currently using the H200 HBA for a couple of the drives ESXi 6.7 recognizes it no problem its on Vmware's compatibly list. But I would be switching this to a passthrough direct to the TrueNAS vm. Right now my database is across multiple drives that's what I'm trying to get away from to have a single drive that I can replace a drive when it fails.

Hi! I totally agree with the ZFS approach. ESXi is what needs to be viewed with caution. They favor enterprise standard solutions that are heavily standardized, meaning, if a network architect wanted to use HBA controllers, they wouldn't try to configure the "standard" ones, they would just buy the servers with different controllers, build only for that. And, no surprise, that is the attitude of ESXi. Its a dog of a hypervisor for complex situations, but it is not too "patent-y" or, if you will, MacGyver-ish. Just an observation, nothing more.

 

[HoneyBadger]

Currently using the H200 HBA for a couple of the drives ESXi 6.7 recognizes it no problem its on Vmware's compatibly list. But I would be switching this to a passthrough direct to the TrueNAS vm. Right now my database is across multiple drives that's what I'm trying to get away from to have a single drive that I can replace a drive when it fails.

Are you using the H200 with the stock Dell IR firmware for your datastore(s) currently? It won't be able to read any RAID volumes once flashed to IT mode.

If the H200 is to be passed through to a TrueNAS VM, you'll still need a storage controller that's "left alone" for ESXi to use for its datastores (including storing the TrueNAS VMX file) - I'm assuming this would be the onboard C602 SATA controller for your motherboard, but performance and redundancy might be lacking there if you intend to run a lot of intensive VMs on ESXi that aren't stored on the TrueNAS disks.

 

[badincite]

Defiantly going this route Got TrueNAS setup in no time and created a raidz pool with iscsi and tied to my vm.

Right now I just have 4 VM's Plex, Web Apps,Photon OS, and one other not doing much. Just picked up a H310 HBA for $25 bucks of ebay may toss that in there also.

 

[badincite]

Anyone had a issue with the iscsi drive not remounting once the server is rebooted? I set the boot order for TrueNAS to be 1st I've even tried to start the windows vm once I knew TrueNAS was fully booted,

Actually keep's the VM from even starting

 

[homer27081990]

I would bet that ESXi doesn't play nice with one of the drives. In my case, it showed sata ssd drives as sas.

 

[homer27081990]

I would bet that ESXi doesn't play nice with one of the drives. In my case, it showed sata ssd drives as sas.

Nevermind. Too tired from troubleshooting, forgot that you dropped the ESXi.

 

[badincite]

The drives show up fine in TrueNAS I just have to force esxi to rescan the sofware iscsi. I'm actually just adding the raw iscsi disk to the vm.

 

[HoneyBadger]

The drives show up fine in TrueNAS I just have to force esxi to rescan the sofware iscsi. I'm actually just adding the raw iscsi disk to the vm.

Have a look at the script repository created by @Spearfoot here that automates some of the chicken-and-egg processes required when doing a virtual TrueNAS solution, including the HBA/VMFS rescan after boot.

Utility scripts for FreeNAS and VMware ESXi

I've created a GitHub repository containing the scripts I've written for use with my FreeNAS-on-ESXi All-In-One systems: https://github.com/Spearfoot/utility-scripts-for-freenas-and-vmware-esxi These might be handy for folks who haven't already...

www.truenas.com

 

[Alex_K]

Your system was OK short of boot drive for ESXi + TrueNAS VM setup. You needed small boot SSD connected to motherboard, Or a RAID card for ESXi boot drive+datastore (with power loss protected write cache). Separate of H200 in IT mode you passing to TrueNAS.

If you want to run a datastore from ESXi boot HDD you would see very bad write performance on write-cache-less controller.

If you want more reliability and run a datastore from outside TrueNAS, get 1 FBWC RAID card for ESXi, and passthrough 1 HBA through to TrueNAS.
If simplicity, boot SSD. And backup ESXi configuration