Virtualizing FreeNAS/TrueNAS with Proxmox

[samuel-emrys]

Hi all,

I'm seeking some feedback on an idea to move my current setup (FreeNAS bare metal) to a VM managed under Proxmox. To provide a bit of context, my use case for a server has changed recently. In the past I've been content with FreeNAS as a file server, and use it to manage a range of largely passive services in jails. This has worked well, but I now have a need to run a variety of VMs for productivity reasons. This would include a Windows Server VM to expirment with AD, and a Windows VM for gaming with GPU passthrough, but also access to tools like the Adobe Creative Suite, as my daily driver is now a Linux system. I've attempted to virtualise things before using Bhyve, and my experience was that it was very unstable, and the tools for VMs available in FreeNAS currently are not everything I'd hope for. I realise Proxmox has ZFS capabilities, but I've no desire to re-create the wheel. Storage is something that FreeNAS does well, and it automates a lot of the functionality that I would have to manually implement in Proxmox. I could do this, but I have no interest in it.

Another obvious, and more supported alternative is ESXi. I'm a fan of the open source nature of Proxmox, and it seems like it has greater functionality at the free tier than what ESXi does. That said, I'm not completely opposed to considering this more closely if it truly does provide a tangible benefit, but I'd rather try Proxmox first before exploring that. To avoid questions about it later, I've read the following threads of note:
- https://www.truenas.com/community/t...-not-completely-losing-your-data.12714/page-6
- https://www.truenas.com/community/t...ative-for-those-seeking-virtualization.26095/
- https://www.truenas.com/community/threads/freenas-on-proxmox-whats-the-current-state-of-play.84287/
- https://www.truenas.com/community/threads/freenas-virtualized-on-proxmox.55849/

My server configuration is as follows:
- Supermicro X9-SRi-F
- Intel Xeon E5-2660 V2 (10c/20t)
- 6x WD Red 4TB (RAIDZ2) - named 'vault'
- 2x Samsung 860 Evo 500GB (Mirrored) - named 'jailhouse'
- 8x16GB (128GB) Samsung DDR3-1866 Registered ECC DIMMS

Currently, I've got two pools; one for the WD Reds which contains the bulk of my storage, and one for the SSDs, which is used for jails - this typically sees the most I/O.

So, with all of that said, I'm trying to think about what a migration to Proxmox would require. My initial thought is that I could migrate the SSD pool to Proxmox's control, and use it as the installation and VM storage pool (this may/will probably involve wiping it and re-creating the mirror; I don't mean migrate in the strict sense here). I could then either migrate my existing set of jails to 'vault', or transition the services provided in those jails to LXC containers. I would also need to make the 'vault' disks directly available to a FreeNAS vm, so I would need to pass through the on-board disk controller. Running pciconf -lv on my existing FreeNAS host yields the following:

Code:

user@helios:/mnt/vault/private/user# pciconf -lv
...
ahci0@pci0:0:31:2:      class=0x010601 card=0x062b15d9 chip=0x1d028086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'C600/X79 series chipset 6-Port SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
...
isci0@pci0:6:0:0:       class=0x010700 card=0x062b15d9 chip=0x1d6b8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'C602 chipset 4-Port SATA Storage Control Unit'
    class      = mass storage
    subclass   = SAS
...

My read of this is that I have two disk controllers on my motherboard; one with 6 SATA ports, and one with 4 SATA/SAS ports. I could connect 'vault' to the 6 port disk controller, and my SSDs to the 4 port disk controller, then pass the 6 port controller directly through to the FreeNAS VM. My plan was to allocate 3x vCPUs to the FreeNAS VM and 16GB of RAM, leaving me 17x vCPUs and 112GB of RAM for other services and productivity VMs. I'll obviously play this by ear wrt performance. Any further drive/pool expansion would require the purchase of a HBA card.

One of the assumptions that I have here is that I will be able to create an NFS/SMB share from my FreeNAS VM to make the storage pool available to other VMs. I also have FreeNAS set up as a UPS master node, with my router configured as a slave node; will it be possible to maintain this with FreeNAS virtualised? My guess is that I would need to pass through the USB port being utilised for communication to the FreeNAS VM?

So, I have redundant VM and boot media, as well as have provided the FreeNAS VM with direct access to the disk controllers. Is there anything else I need to consider? Are there any holes that can be poked in this plan? Any suggestions or feedback are appreciated.

Cheers.

 

[samuel-emrys]

@jgreco I saw you were active in other threads and had some opinions on this; do you have any feedback you could provide me?

 

[joeschmuck]

While I'm not a Proxmox user I think I could offer some advice with respect to the virtual CPUs. You have a 20 thread CPU and that means you have lots of wiggle room for allocation. You should be able to allocate 2 vCPUs to FreeNAS with no problem, 3 if you find out you actually need it, and then as your other VMs are created, assign them whatever they need. if they are a multithreaded application then 2 vCPUs, if they really need more then add more. Do not worry too much about over-provisioning, it will stress a person out. If you plan to run up some services in a VM that really do require dedicated CPU cores, I would imagine that Proxmox has a feature to lock the cores to the VM like ESXi does, but I'd only do that if performance is affected.

I'm sure someone else will be able to give you better advice but basically you will need to jump in with both feet and see how it goes.

Good Luck and if you don't like Proxmox, give ESXi a chance. Less bells and whistles but solid performance.

 

[samuel-emrys]

Cheers Joe, some good advice.

 

[seanshankus]

@samuel-emrys I see its been a few months, which way did you go and what were the results? I'm consider the same for the reasons you mentioned (although i don't have as much power as you do).