Virtualizing TrueNAS just for the GUI?

[oguruma]

I need to build a new box.

I'd like to condense my main storage and all of my VMs/containers into one box. At present I have a Proxmox VE and a TrueNAS Core box.

I'm considering using Proxmox, but installing TrueNAS SCALE (since I'm more familiar with Linux) in a VM just to have a nice UI to manage the shares.

Has anybody done anything like this?

 

[artlessknave]

check out the forum rules about posting info requirements.
virtualizing TrueNAS is not encouraged. TrueNAS is intended to run on the hardware directly.
what does proxmox do for you that truenas scale doesnt?
also note that any storage on a virtualized truenas will not be available to the virtual host until the VM completely initializes, and so using it for VM storage is far more complex, since the truenas VM would have to be on persistant storage, and then the VM would have to start and only THEN can other VM's start.
since you HAVE to pass an HBA through to TrueNAS, as well, it takes up extra connectivity; you need, at a minimum, 2 storage controllers, one for the hypervisor, and one for TrueNAS
proxmox is also not considered as good a host for virtualizing TrueNAS.
EDIT: proxmox does not have the same track record for TrueNAS VM stability as the recomended VM platforms, and so YMMV. virtualizing TrueNAS is not a supported configuration, and is done at your risk. there is a right way to do it, and many, many wrong ways. assistance for unsupported configurations that go wrongcan be spotty, as many of those here who can help best when something goes wrong consider virtualized TrueNAS a waste of effort to try to fix.

 

[oguruma]

check out the forum rules about posting info requirements.
virtualizing TrueNAS is not encouraged. TrueNAS is intended to run on the hardware directly.
what does proxmox do for you that truenas scale doesnt?
also note that any storage on a virtualized truenas will not be available to the virtual host until the VM completely initializes, and so using it for VM storage is far more complex, since the truenas VM would have to be on persistant storage, and then the VM would have to start and only THEN can other VM's start.
since you HAVE to pass an HBA through to TrueNAS, as well, it takes up extra connectivity; you need, at a minimum, 2 storage controllers, one for the hypervisor, and one for TrueNAS
proxmox is also not considered as good a host for virtualizing TrueNAS.

TrueNAS adds a really nice UI for managing shares/backups.
Proxmox has a really nice UI for managing VMs/containers.

Of course, Proxmox has a larger community and user base, by virtue of being around much longer, and it's obviously more of a mature piece of software, as well.

 

[KrisBee]

The idea that "proxmox is also not considered as good a host for virtualizing TrueNAS" is the received wisdom for TrueNAS CORE which has a proven track record only with VMware ESXI. But, linux based TrueNAS SCALE may change this. I would have expected SCALE to be well behaved as a VM running in Proxmox with the proviso that it is properly confugred and resourced and a HBA is used in pass through mode for the data drives. It's the stability/reliability of the HBA passthrough that is crucial. Personally, I'd stick with two boxes for now.

 

[artlessknave]

"proxmox is also not considered as good a host for virtualizing TrueNAS"

I didn't like this wording and changed it.
stability is absolutely the focus, and hours of error free run time is usually the bar used.
as @jgreco points out for LSI HBA's "Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error".
just because you got it running doesn't mean you can know that it will run well, reliably, or not mangle your data when something does go wrong, and so this is a risk factor that must be considered.

just yesterday I had the displeasure of telling a new TrueNASer that it looks like their data is just gone. I'd rather people know the risks BEFORE that happens, not after. if you then choose that risk, you aren't gonna be coming here with high expectations, though sometimes recovery is possible.

 

[jgreco]

But, linux based TrueNAS SCALE may change this. I would have expected SCALE to be well behaved as a VM running in Proxmox with the proviso that it is properly confugred and resourced and a HBA is used in pass through mode for the data drives.

Linux based SCALE does not change this. The problem isn't SCALE or CORE. Both FreeBSD and Linux are extremely mature operating systems. From a guest OS point of view, they are roughly equivalent, as the emulated or virtualized devices used by these operating systems are both ridiculously well-exercised.

The problem is the relative immaturity of PCIe passthru on Proxmox, and the fact that people try to run this on random bullsheyt hardware. Both the hardware and the PCIe passthru implementations have to be BULLETPROOF. We ran into this with VMware ESXi back in the ESXi4 days. Both the Supermicro X8 (Westmere/Nehalem) boards and also the implementation of passthru in ESXi 4.0 were only about 99%, which led to numerous instances of people getting cocky because it worked fine for an hour, but then went south after weeks.

Anyone who has talked with me knows I am not a huge fan of ESXi. It is the Cadillac of hypervisors. It's idiotic to argue the point. But it's a paid product, extremely limited in its free form. I can't fix that, but I'm happy to see bhyve, Proxmox, and Xen taking it on. I am happy to see the alternatives. I am hoping that they shake out as rock solid. You can do your part here by NOT trying to run Proxmox on some ex-gaming overclocked jankPC and instead running it on hardware known for its solid virtualization. We eventually moved beyond the X8 gear and lots of X9, X10, and more recent gear are unquestioned virtualization champs. This makes it a lot easier to judge Proxmox when run on those platforms. We do need people willing to run TrueNAS under Proxmox. I'm not in a great position to being able to do that right now, but I'm happy to talk to those who are.

The thing that will get me to consider Proxmox a true contender is lots of examples of well-designed systems successfully running it without incident. There will always be the counterexamples. We had lots of this in the early days, people running ZFS on AMD 4GB APU's "because they could", and it ended badly for many. That doesn't make ZFS bad. It just makes that setup bad. So for Proxmox to be successful, not a stretch at all. What will alarm me is if I see well-designed systems that have catastrophes.

 

[KrisBee]

I have proxmox 7 installed on a asrock ITX intel 1150 server m/board. Limited to 16GB, it is just for tinkering and I prefer to use zfs direct on the host rather than use a HBA in the small enclosure. FreeBSD and pfsense are well behaved, no problem with starts, shutdowns etc and "host" cpu type can be used. FreeNAS/TrueNAS CORE always had problems with clean shutdowns when using "host" cpu type, whereas TrueNAS SCALE behaves as any other linux distros with "qemu-guest-agent" pre-installed. No problem using proxmox as a testbed for TrueNAS SCALE, even with nested virtualisation. Production use is, of course, another question.

 

[jgreco]

FreeNAS/TrueNAS CORE are running what should be an identical ACPI setup to FreeBSD or pfSense. That doesn't make a whole lotta sense offhand, but it could be something obscure. I know I put a crapton of time into dealing with the finer points of OS installs for a different version of FreeBSD that is most often used as an appliance in VM configurations. The number of minor quirks you have to deal with is infuriating... I did notice a bunch of weirdnesses went away in FreeBSD 12 and 13 that were there in 11 (which dates from back in 2016). It might be interesting seeing if there's any recent changes. Sometimes they don't put slightly heretical changes into the older versions.

 

[artlessknave]

AMD 4GB APU's

I started on this. though I had 8GB. I even had a backup system. on the same platform. e-450 and e-350 iirc.
the boards even still work, though on one the pci/e ports are buggered somehow. I suspect it might have been from my fake LSI cards.... but who knows. maybe it was mutually assured destruction.
they get like 300 passmark marks though. barely enough to turn on.