Home all in one server - ESXi with freenas as a VM or freenas with VMs?

[Herr_Merlin]

Hi all,

I am looking to setup a home all in one server with old leftover parts.
The question is now virtualized freenas or bare metal with VMs?

Hardware I got:
Supermicro X9DRW-CFT
2x Xeon E5-2680 v2
16x16GB DDR3 ECC reg
4x1TB NVMe SSD on a PCIe Card with PLX
1x Vega 56
1x AMD GPRO 4300
1x HP P212
1x HP P420 in IT Mode (This is basically a 9211-8i with 2008 chipset)
10x6TB SAS
2x100GB Samsung SSDs (old enterprise with SATA and PLP)
2x800GB SAS SSD from HGST
all within a Supermicro 2U case with a 12bay

What I want to run on this thing:
-gateway vm
-about 10 windows VMs
-plex with GPU
-Terminal Server for game streaming with the vega
-nextcloud

So the question, go with freenas as the host OS and use jails or go with vmware?

Pool layout would be if I go for virtualized (8Cores + 64GB RAM):
10x6TB as RAIDZ2 or RAIDZ3 with 2x 100GB as SLOG (if going ESXi if going native I would not need those?) on the HP P420 passed to the VM
4xNVMe RAIDZ1 for VMs (Should still be fast enough) passed to the VM
2x 800GB SAS on the P212 in RAID1 for Host OS, Gateway VM and freenas boot disk

My question would be if going for freenas as bare metal, can I pass through the GPUs?
Is there a convertor for vmware VMs I already setup at another host, which I want to decommission?

Hardware Update:
Supermicro X9DRW-CFT
2x Xeon E5-2680 v2
16x16GB DDR3 ECC reg
4x1TB NVMe SSD on a PCIe Card with PLX
1x Vega 56
1x AMD GPRO 4300
1x Adaptc 5805 with BBU für OS RAID1
1x HP H240 HBA (IT Mode) SAS 12G
10x6TB SAS
2x480GB Samsung SSDs for HW RAID1 for ESXi OS
2x800GB SAS SSD from HGST
all within a Supermicro 2U case with a 12bay

Hardware Update 2.0 now kinda working.. NVMe issues see below
Supermicro X9DRW-CFT
2x Xeon E5-2680 v2
16x16GB DDR3 ECC reg
4x1TB NVMe SSD on a PCIe Card with PLX
1x Vega 56
1x AMD GPRO 4300
1x Adaptc 5805 with BBU für OS RAID1
1x Supermicro AOC-S2308L-L8i IT Mode
10x6TB SAS
2x480GB Samsung SSDs for HW RAID1 for ESXi OS
2x800GB SAS SSD from HGST
all within a Supermicro 2U case with a 12bay

 

[Yorick]

My question would be if going for freenas as bare metal, can I pass through the GPUs?

Assume no. You can pass them through but the BAR placement will be off, you very likely won’t be able to actually use them.

Is there a convertor for vmware VMs I already setup at another host,

Sure. Take vmdk, convert to raw. Create zvol. Dd the raw file into the zvol. Create bhyve VM to use the zvol.

go with vmware

I’d go with the idea of VMWare as the main OS and passing an HBA through. Some folk also had to pass a NIC through, your mileage may vary.
Bhyve is not remotely on par with other hypervisors. For running a single-core Ubuntu just because, sure. A Win10 VM without GPU, sure. Running bunches of VMs, particularly if you want GPU passthrough - that’s not bhyve.

 

[Yorick]

Oh one more: VMWare 7 dropped support for the 2008. You can still pass it through, but it seems you have to redo the passthrough after every reboot. Check the thread we had around here about that. If you are going VMWare 7, consider a newer model HBA for an easier time.

 

[Herr_Merlin]

already started a few hours ago with the vmware road. went for 6.7 after a short try of 7.0. 7.0 was so slow and randomly freezing. 6.7 booted up like a charm and runs now super fast and stable.

having some issue here.
windows / linux VMs no issue.
freenas.
passing all NVMe drives and creating a pool. worked perfectly
passing the HP P420 in IT mode -> I am not able to find the drives after freenas booted up. During installer it listed all drives fine. tested to install on one of those - worked. after bootup from a disk connected to the P420 in IT mode I see all drives... but I want to install on the vmware volume for various reasons...
So replaced the P420 with a LSI 9341-4i flashed to LSI 9300-4. freenas vm won't finish booting. stuck at that error ( https://hardforum.com/threads/howto...-and-multiple-cpus-and-multiple-nics.1679233/ ) tried the mentioned solution but it did not work... any ideas?

 

[Spearfoot]

...but I want to install on the vmware volume for various reasons...

May I ask what you mean by this?
And another question: where are you installing the FreeNAS VM?

 

[Herr_Merlin]

I am installing the FreeNAS OS at an vmdk located on the hardware RAID1.
Passing the controller and want to setup the disks as mentioned above

 

[John Doe]

I think I might have a similar setup to your future one.
I went also with ESXi6.7 and it runs very stable. haven't had a single freeze or crash (running since a year or so)

on the esxi boot drive I installed pfsense and freenas, just to make sure, the basic fountain is quickly available. everything else is installed on the pool, freenas provides and passed to esxi via nfs

"back in the days" there was a lot movement. Freenas corral was released and stopped, jails have changed and I got mad on the changes and the maintenance the jails required (at least it was my subjective feeling).

so i thought a bit more independence would be good.

however i definitely can recommend ESXi6.7 with freenas VM. But maybe it is also worth going with freenas as hypervisor. I got the feeling, that there is less movement on jail "backend" now days

 

[Yorick]

that there is less movement on jail "backend" now days

Yeah iocage will likely be it. But that’s containers not VMs. Great for running FreeBSD native stuff, that they are.

 

[Evertb1]

Bhyve is not remotely on par with other hypervisors.

I am with you there. I really tried for my lab environment 2 years ago but when I needed a Windows server VM I decided to go with ESXi (6.7) for my lab. I am very happy with running a FreeNAS VM on it. I passed trough an Dell PERC H310 HBA and an Intel NIC to it and that works like a charm as well. Never had a problem for more then a year now.

 

[Herr_Merlin]

As I still had issues with the HBA I went on a short journey with Server 2019, ReFS and Hyper-V..
Performance wise it was "better" ReFS during this short test really impressed me.
But when wanting to use DDA to pass PCIe devices to VMs it did not support my old hardware. A feature which worked easily with ESXi.
As I still had the issue with the HBA running and the freenas VM, I tried an HP P240 HBA.. which worked like a charm..
That HBA I am using at the two work freenas units.
Long story short, back to ESXi and HP P240 HBA ordered..
The PERC H310 I have has the DELL Firmware on it for HBA mode.. and stops at this option ROM and freezes the system with a warning message.. if I disable it's option ROM the flash tool won't see it..

 

[Evertb1]

The PERC H310 I have has the DELL Firmware on it for HBA mode

I have a couple of those and when I bought them they also contained the Dell firmware. I reflashed them according to this guide: SAS HBA crossflashing Dell Perc H200/H310 . Just read trough it carefully and you won't have any trouble. There are other guides out there but I liked this one the best.

 

[Herr_Merlin]

did not work :(

SOLVED - ESXi with freenas -> freenas does not list HBA Disk - Changed HBA -> solved

Hi all, I've setup ESXi with a freenas VM. Within the VM during Setup I do see all the disk attached to the HBA and the NVMe storage I've passed to the VM. I've installed Freenas on a VMDK at the hardware RAID1 where ESXi is installed. What I do see within the VM: -during Setup I do see...

www.ixsystems.com

still stock with the HBA issues

 

[Herr_Merlin]

Seems like a doomed setup.. after the HBA I have issues with the NVMe.

NVMe Z1 Pool - best practice ? Ultra slow writes

I've setup an ESXi with FreeNas. Now I got 4 NVMe drives, which I passed to the freenas VM. The performance is, well I would expect the same performance out of a floppy Z1 volume.. So clearly I missed some tuning and setup. Are there guidelines, I couldn't find any, for NVMe pools? Using 4 of...

www.ixsystems.com

 

[Evertb1]

7.0 was so slow and randomly freezing. 6.7 booted up like a charm and runs now super fast and stable.

I have a homelab with an older Asrock motherboard upgraded to ESXi 7.0.0 and it boots even faster then it did with 6.7. And over all it's running great. I think that much is depending on the support of the available hardware.

 

[Newfoundland.Republic]

ESXi 7.0.0 and it boots even faster then it did with 6.7

@Evertb1 - I noticed that 7.0 and 6.7 both boot faster than 6.0....

 

[joeschmuck]

@Evertb1 - I noticed that 7.0 and 6.7 both boot faster than 6.0....

Hum, I'm running 6.7 without issue but it does boot up slow, but once running all is good. Need to read on the 7.0 free limitations before I take the plunge.

The question is now virtualized freenas or bare metal with VMs?

I've used ESXi on bare metal and then created a VM for FreeNAS and passed through either the hard drive controllers or individual hard drives via RDM. I prefer passing through the controllers. I only run Plex on FreeNAS in a jail, all other virtualizations are directly on ESXi.

 

[Evertb1]

Hum, I'm running 6.7 without issue

I noticed that you are running ESXi on a Supermicro motherboard as well. Is that the slow one during boot? I have been running ESXi 6.7 on a X10SL7-F motherboard and it booted noticeably slower then on my lab computer that was build on a consumer grade Asrock motherboard. Though after the boot it was running without any issue. At the moment I am saving for an new system because I want to run ESXi on my main server as well. The Supermicro X11... series motherboards are on my mind. Though I would love it when I was sure about the usability of the Ryzen CPU's. All those cores are nothing to sneeze at.

 

[Evertb1]

7.0 free limitations before I take the plunge.

Take care that you get a new license before you take the plunge. I just upgraded (from the command line) without thinking and was suddenly running ESXi without a license. It's not hard to get a new license though.

 

[joeschmuck]

I noticed that you are running ESXi on a Supermicro motherboard as well.

Yes, running on the Supermicro X11SSM-F. It's only slow during a few file executions, not sure exactly what it's doing (cold be something with RAM or Swap File) but once it's booted up, i see no delays in anything. So I'm happy with ESXi 6.7 but now I would like to try version 7.0 to see what it offers and/or pitfalls.

Take care that you get a new license before you take the plunge. I just upgraded (from the command line) without thinking and was suddenly running ESXi without a license. It's not hard to get a new license though.

Thanks for the heads up. I'll log in today to get a new license before I download it and install it.

 

[Evertb1]

I download it and install it.

I just SSH'd into the esxcli with Putty and performed the update from there. Worked like a charm.