jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,626
[---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file sharing (small office, documents, scratch space). THIS post is aimed somewhat more at people wanting to use FreeNAS to manage lots of storage space. ----]
[---- 2022/11/19: It's been a long struggle, but finally someone at iXsystems has said it, for the doubters, here it is ----]
FreeNAS is awesome. FreeNAS can and will run as a VM. That does not make it a good idea.
[---- 2022/11/19: It's been a long struggle, but finally someone at iXsystems has said it, for the doubters, here it is ----]
We don't recommend running any TrueNAS virtualized in general,
FreeNAS is awesome. FreeNAS can and will run as a VM. That does not make it a good idea.
- FreeNAS is designed to run on bare metal, without any clever storage systems (UNIX/VMFS filesystem layers, RAID card caches, etc!) getting in the way. Think about this: ZFS is designed to implement the functionality of a RAID controller. However, its cache is your system's RAM, and its processor is your system's CPU, both of which are probably a lot larger and faster than your hardware RAID controller's cache!
- Without direct access to the hard drives, FreeNAS lacks the ability to read SMART data and identify other developing problems or storage failures.
- A lot of the power of FreeNAS comes from ZFS. Passing a single virtual disk to ZFS to be shared out via FreeNAS is relatively safe, except that ZFS will only be able to detect and not actually correct any errors that are found, even if there is redundancy in the underlying storage.
- There is a great temptation to create multiple virtual disks on top of nonredundant datastores in order to gain "MOAR SPACE!!!". This is dangerous. Some specific issues to concern yourself with: The data is unretrievable without the hypervisor software, the hypervisor might be reordering data on the way out (which makes the pool at least temporarily inconsistent), and the hypervisor almost certainly handles device failures non-gracefully, resulting in problems from locked up VM to unbootable VM, plus interesting challenges once you've replaced the failed device.
- Passing your hard disks to ZFS as RDM to gain the benefits of ZFS *and* virtualization seems like it would make sense, except that the actual experiences of FreeNAS users is that this works great, right up until something bad happens, at which point usually more wrong things happen, and it becomes a nightmare scenario to work out what has happened with RDM, and in many instances, users have lost their pool. VMware does not support using RDM in this manner, and relying on hacking up your VM config file to force it to happen is dangerous and risky.
- FreeNAS with hardware PCI passthrough of the storage controller (Intel VT-d) is a smart idea, as it actually addresses the three points above. However, PCI passthrough on most consumer and prosumer grade motherboards is unlikely to work reliably. VT-d for your storage controller is dangerous and risky to your pool. A few server manufacturers seem to have a handle on making this work correctly, but do NOT assume that your non-server-grade board will reliably support this (even if it appears to).
- Virtualization tempts people to under-resource a FreeNAS instance. FreeNAS can, and will, use as much RAM as you throw at it, for example. Making a 4GB FreeNAS VM may leave you 12GB for other VM's, but is placing your FreeNAS at a dangerously low amount of RAM. 8GB is the floor, the minimum.
- The vast majority of wannabe-virtualizers seem to want to run FreeNAS in order to provide additional reliable VM storage. Great idea, except that virtualization software typically wants its datastores to all be available prior to powering on VM's, which creates a bootstrap paradox. Put simply, this doesn't work, at least not without lots of manual intervention, timeouts during rebooting, and other headaches. (2013 note, ESXi 5.5 may offer a way around this.)