Yeah, I considered that, but I have experience with Debian (and like Debian very much) which is what Proxmox is based on. Proxmox allows me to run LXC containers which I have grown fond of and I like to place each service I am running in a seperate LXC instance that way if something goes pear shaped, I simply have to spin up a new instance of Ubuntu Server and reinstall the service. It does warrant further investigation though. Does vSphere use LXC instances?
No, vSphere doesn't provide LXC or anything like it. FreeNAS, of course, supports jails which are a very similar concept for FreeBSD containers. So you could run your services on those instead, but you would have to get used to FreeBSD, and in particular its packaging system.
I suspect you're dead set on Proxmox - something I certainly understand. For home use, I think it’s wonderful and its community edition beats free ESXi into a cocked hat. If so - and you want to do hyperconverged virtualisation and storage on the same box - you may be better off dropping FreeNAS and just using ZFS directly. ZFS on Proxmox has been a first class citizen for some time. It will natively support creating zvols for VM disks, nested file systems for containers, ZFS snapshotting/cloning, and even - with pve-zsync - scheduled snapshotting and replication. There are suggestions in the manual that at some point in the future there will also be GUI support for setting the latter up and doing VM recovery. All very neat and clean.
What you will lose, of course, is all of FreeNAS's awesome support for graphically managing pools, volumes, shares, snapshots, scrubs, replication etc., very good alerting, and easy configuration backup/restore. Samba with web based management can be done relatively easily by bind-mounting storage into the provided fileserver LXC appliance. NFS is hard to run from a container so must be done from the host, although you can use ZFS's sharenfs properties to easily export shares. ZFS alerting can be done through setting up zfs-zed on the host (SMART alerting to the Proxmox root user's email already works out of the box). Scrubs must be manually scheduled via crontab. To replicate non-VM shares, you need to install one of the many existing scripts for managing snapshots and replication to the host. But all in all, you need to be comfortable doing storage admin at the command line.
FreeBSD/FreeNAS has historically been a more mature ZFS environment than ZFS-on-Linux due to license compatibility issues. However, ZoL's been around a while now and is becoming a more mainstream concern since Ubuntu 16.04 decided to ignore the licensing risk and bundle it in as a first class citizen anyway; Proxmox uses an Ubuntu-derived kernel with a Debian userland. Harder to ignore is the wealth of experience on this forum on running FreeNAS based ZFS, which is subtly different. The Proxmox/ZoL forums don’t have the same level of knowledge.
I’m currently trying to work out which option’s best for me, as I want to slim down from running a two machine Proxmox cluster (with a lightweight Proxmox node on FreeNAS VirtualBox for cluster quorum) accessing FreeNAS storage over NFS. With a backup FreeNAS box to take storage replicas, this is four physical servers. I want to run a hyperconverged ZFS virtualisation setup on two larger servers. As well as being simpler and more efficient in hardware terms, it removes network limitations on accessing storage - 10Gbit looks too expensive and fiddly for me, I don’t have the spare slot for the cards on my ZFS servers, and even with it there’s still a fair bit of protocol overhead from iSCSI/NFS.
Summarising below what I think the pros and cons are at the moment for me - I would appreciate corrections + feedback on things I have missed. I’ve ruled out FreeNAS under Proxmox and under Xen, as they don’t seem very frequently used and potentially problematic.
It’s not an easy decision; if FreeNAS had decent native virtualisation it would be the easy leader, but even in v11 it seems extremely basic.
Cheers,
Martin
****
FreeNAS with native virtualisation
Pros:
* First class, mature ZFS NAS
* No abstraction layers (NFS/iSCSI) over ZFS VM volumes
* Support for all ZFS features - e.g. low cost VM snapshot/clone, thin provision
* Boot off mirrored USB to maximise drive bays for pool storage
* Really easy host backup/restore
* Integrated management/alerting
* Recommended configuration suitable for production use (is this actually the case for the new virtualisation features, or are they still viewed as experimental?)
* Easy snapshot replication
Cons:
* Very immature virtualisation - bhyve pretty new with limited features, UI basic and usability not good
* Somewhat picky about hardware
* No native Linux containers with storage bind-mount
* Can only buy official support on iXsystems certified hardware
****
ESXi/FreeNAS all-in-one (with HBA pass-through)
Pros
* First class, mature ZFS NAS
* Mature ESXi virtualisation
* Fair number of people doing it
* Integrated management/alerting
Cons
* Limited ZFS feature support via VAAI
* VMware somewhat crippled in free ESXi version (e.g. storage vmotion, can you do snapshots nowadays as you couldn’t when I last used it?), vSphere expensive for home use
* Choice between NFS (easy to manage but significant overhead) or iSCSI (more complex management requirements and still some overhead) additional layers over ZFS VM volumes
* USB boot, but ESXi datastore needs to reside on disk, burning a bay/bays
* No software resilience on USB boot device or data store; additional hardware adapter required for RAID of latter
* Very picky about non-enterprise hardware
* Not really recommended for production, not possible to buy official support
* Complicated setup, difficult to get things like controlled UPS shutdown working properly
* No native Linux containers with storage bind-mount
****
Proxmox with native ZFS
Pros:
* First-class, non-crippled virtualisation
* No abstraction layers (NFS/iSCSI) over ZFS volumes
* Support for all ZFS features - e.g. low cost snapshot/clone
* Software mirroring (ZFS or mdadm) for boot drives
* Native Linux containers with storage bind-mount
* Wide hardware support
* Recommended configuration suitable for production use (albeit with some customisation required of Debian based VM host for ZFS snapshotting, zed, NFS sharing)
* Fair number of people doing it
* Reasonably priced support for home use
Cons:
* No built in UI NAS functionality - lots of command-line required
* Have to roll/integrate your own alerting/snapshot replication for ZFS
* Less mature ZFS platform than FreeNAS
* Less mature virtualisation platform than ESXi
* No USB boot, burning a bay/bays
* No appliance-like host backup/restore