TrueNAS Scale's KVM instead of Proxmox - Seeking Hands-on Experiences

LuckyGreen

Cadet
Joined
Sep 10, 2021
Messages
5
I am looking for hands-on personal experience reports from beta testers that tried using TrueNAS Scale with KVM as the host OS instead of Proxmox as the host OS and running TrueNAS Core on top of Proxmox. Which is what I am doing at the moment.

Is in your experience the current TrueNAS Scale beta ready to take Proxmox's place as as the general-purpose single-system VM host OS for home lab purposes? If you tried it and found something missing in TrueNAS Scale that you used in Proxmox, what was it that you were missing?

Your thoughts are much appreciated. I have to make an architectural decision and I have to make that decision before TrueNAS Scale is likely to go GA.
 

warllo

Contributor
Joined
Nov 22, 2012
Messages
117
So far the latest beta of scale has been stable for in my experiences from a KVM / Virtualization standpoint. The biggest feature that I personally miss is the ability to pass through a single USB device rather than an entire USB controller.

If you need to use non UEFI bios / boot type I believe you are unable to use the VNC web viewer, not a huge deal just be prepared for it.

The only other area that I feel Proxmox is vastly superior is the UI around taking and restoring snapshots. Proxmox's is just simple and easier to use. In the end, TrueNas Scale in my opinion can replace Proxmox as a single system VM host OS for home lab purposes.

Please note these are just my opinions and my personal experiences.
 

LuckyGreen

Cadet
Joined
Sep 10, 2021
Messages
5
The biggest feature that I personally miss is the ability to pass through a single USB device rather than an entire USB controller.

Thank you tremendously for your detailed response. Not being able to pass through individual USB devices to a VM, I would miss as well.

Do you happen to know if there exists an existing feature request and/or roadmap item for single-device USB passthrough? I searched on the TrueNAS site, but have not found such a request. Since I am new here, I am certain that I just didn't look in the appropriate section, since someone must have filed this request by now.

I may have found one more challenge. It appears that iXsystems offers paid support only in conjunction with iXsystems hardware. If so, that would be unfortunate, since I typically sign up for paid support when offered.

Thanks again for taking the time to write your very helpful reply.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@LuckyGreen Rather than give you a long list of reason why I think the opposite to @warlio, why don't you configure your Proxmox server for nested virtualisation and test TrueNAS Scale yourself? You'll get a better idea of the features & UI for VM support within TrueNAS Scale which IMHO are far inferior to Proxmox.
 

LuckyGreen

Cadet
Joined
Sep 10, 2021
Messages
5
@KrisBee Thank you for providing another perspective! To answer your question directly, the reason why I haven't (recently) tried TrueNAS Scale myself is because I invested well over 30 hours spread over a weekend testing the alpha back in January, concluding that the alpha wasn't ready to be fielded for any of my use cases. One of which is that general purpose home lab server.

To expand upon my question slightly, allow me to inquire as to your views for more limited use case. Do you believe the TrueNAS Scale beta to be suitable to serve as the host OS for a high-volume Plex server using a dedicated nVidia GPU for transcoding? The server has the adjunct requirement of being able to run a low-volume traffic pfSense VM on the same hardware.

The server is currently architected using Proxmox as the host OS, with a TrueNAS Core VM and an Ubuntu/Plex VM and a pfSense VM as guests.

It wouldn't mind simplifying the architecture by removing Proxmox out from underneath, while running Plex on top of TrueNAS Scale instead of in a separate Ubuntu VM. Moving the pfSense VM on top of TrueNAS Scale's KVM. Thanks!

(Asking to save myself a weekend of testing, if possible).
 

gator96100

Cadet
Joined
Sep 11, 2021
Messages
2
I was running a similar Proxmox configuration, TrueNAS Core as VM and pfSense as VM with Plex directly running on the Proxmox host OS.

I am in the middle of moving to TrueNAS Scale. The Plex application does work nicely with hardware transcoding. I don’t have a dedicated GPU for Plex.

I’m still stuck on configuring pfSense on TrueNAS Scale. The problem I did run into is that I cannot pass the ethernet interfaces that are on my motherboard into the VM, as they are not in separate IOMMU groups. On Proxmox I did solve this by setting “pcie_acs_override=downstream”, I don’t know if there is the option to set this on TrueNAS Scale.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@LuckyGreen

A brief follow-up. I'm not a plex user, but obviously using plex in TrueNas Scale means dealing with their implementation of Kuberentes and satisfying yourself that the version, plugins and GPU transcoding all work as you need. Whether you want to be dependent on Truechart APPS is a question for you. It might be worth reading various releveant threads here and at r/truenas about that. For example: https://www.truenas.com/community/t...xmox-how-are-you-finding-truenas-scale.94662/

The problem @gator96100 has is a reminder that TrueNAS Scale essentially turns a server into an appliance. There's no "apt update" or "apt install". Nothing simple like modifying grub.cfg to add kernel boot params, This begs the question about what happens re: security updates.

There might be some complex network stuff under the hood to support kubernetes and Truechart apps, but for virtual machines you're stuck with one bridge per interface. Unlike, Proxmox, there's no ifupdown2 for vlan aware bridges and no option to use openvswitch. Nor is the standard debian /etc/network/interfaces config in use.

The VM UI in TrueNAS Scale is little more than a port of the UI from Core. For instance, what relevance does the "device ID" and device "order number" have in a VM's device list? There's a problem with "qemu guest agent" support which means you'll not get auto scaling for a VM with a graphical interface, etc (see this thread: https://www.truenas.com/community/threads/21-08-qemu-guest-agent-in-vms.95195/ ). How much of the underlying code used for bhye in CORE has been re-written to take full advantage of KVM?

I've reached the stage where my "home lab" - separate TrueNAS core and Proxmox servers - is used for little more than hobbyist tinkering, but frankly, you would get far more useability from installing virt-manager on a debian dekstop for VM's than using TrueNAS Scale.
 

speedtriple

Explorer
Joined
May 8, 2020
Messages
75
Running ESXi today, with TrueNAS Core and Home Assistant as VM.
Would be perfect if TrueNAS Scale instead could be ’main‘ running Home Assistant as VM.
 

LuckyGreen

Cadet
Joined
Sep 10, 2021
Messages
5
@KrisBee

Thank you so much for your detailed and well-referenced post. As much as my current architecture of having to virtualize a large TrueNAS Core instance just so I can squeeze a tiny pfSense VM onto the same hardware bothers me, it sounds like TrueNAS Scale still has a ways to go before it will become the approach producing fewer headaches.
 

Ixian

Patron
Joined
May 11, 2015
Messages
218
Beta 21.08 release 1 is stable as far as KVM goes but nowhere near replacing Proxmox as a hypervisor host even if you were willing to look past the "beta" part.

Run Proxmox and TrueNAS Core in a VM with the disks passed through if you want TrueNAS's ZFS tools/management/etc. on a VM host.
 

FreeNASftw

Contributor
Joined
Mar 1, 2015
Messages
124
Can I interject simply to ask why it seems that many people are running TN Core under ProxMox? Are you all just testing TN Core or are you actually using it as a VM?
 

LuckyGreen

Cadet
Joined
Sep 10, 2021
Messages
5
Can I interject simply to ask why it seems that many people are running TN Core under ProxMox? Are you all just testing TN Core or are you actually using it as a VM?

I suspect there are several reasons why folks might run TrueNAS Core under Proxmox. I can only speak for myself, but I believe that I've read enough posts and watched enough videos on this particular topic to have a pretty good sense.

Speaking for myself:
* I wish to host a personal Plex server on a Dell R730xd that is stuffed to the gills with drives at a colo. I will have plenty of bandwidth, plenty of IPv4 addresses, but only get rack space for one system. The system also needs to be able to handle some related ancillary traffic that amounts to negligible bandwidth consumption. To make managing that traffic maintainable, I want pfSense on the same physical box. Two operating systems, one physical box. Therefore, I need a hypervisor. I've used VMware since the day it shipped. I am reasonably proficient with Hyper-V and use it for my production systems. Proxmox is where the hypervisor ball is at the the moment and of all the hypervisors, the PCI passthrough in Proxmox appears to be the most reliable to me. (The very concept sucks, of course).

* I have a numerous home lab servers, of which all but one are Intel NUCs. I simply do not have the physical space or the noise tolerance for a bunch of big boxes. The one big box, an ultra-quiet build, will have to house the NAS for the NUCs, host the backup server, and a few other tasks. Which can't all be accomplished easily and cleanly within one OS. Again, I need a hypervisor. And again that means virtualizing TrueNAS. Unless I can reverse the stack and have TrueNAS Scale be the hypervisor, which I was hoping for, but it doesn't sound like that's imminent.

* Not speaking for myself, simply observing based on the number of YouTube videos on this topic that I have seen, many hobbyists that have only or two servers run Proxmox as the host OS to be able to try out any number of things in VMs. Yet they need somewhere to store their data. At present, the choice for NAS is overwhelmingly TrueNAS Core. Again, the end result often is virtualizing TrueNAS Core on top of Proxmox. For some this works without issues, for others it will never work right. Seems to be unpredictable how the dice will fall.
 
  • Like
Reactions: vpv

JOduMonT

Dabbler
Joined
Jan 27, 2015
Messages
29
I'm a long time user of Proxmox and ESX/ESXi for business and personal purpose. At personal level; just switched to TrueNas Scale no matter what I will encounter.

Before switching to TN Scale, I was running OPNsense, which is barebone for now, until I studied the PCI passthrough with TN Scale, and I had a Windows 10 with USB + GPU passthrough; now thanks to a real KVM implementation this windows works through SPICE (which is not very available under Proxmox because of their wrapper). Since a long time I abandoned to run any LXC for k3s, so that was a big win for me when I saw TN Scale is k3s to run the kubernetes part.

Proxmox is still great if
- you need to passthrough USB and PCI (at least for now)
- you need CoroSync for the VM, because at the container level you could easily make a cluster with k3s
- you use LXC, a Proxmox Backup and how don't want to use ZFS

TrueNas Scale
- still in Beta, but sound very stable for me
- it seems to use a pure KVM instead of wrapping in like Proxmox do with KVM and LXC
- I'm please to see k3s running directly on the host instead of in a VM like they tried a few years ago with RancherOS
 

chruk

Dabbler
Joined
Sep 4, 2021
Messages
27
Can I interject simply to ask why it seems that many people are running TN Core under ProxMox? Are you all just testing TN Core or are you actually using it as a VM?

To me it is quite simple, I am not running bare metal for everything I do its inefficient. I also prefer using things specialised for a task rather than a "bit of everything".

Proxmox specialises in VM's, that's what I use it for, its a very mature product.
TrueNAS Core aka FreeNAS, is built up on FreeBSD which unlike Linux supports ZFS officially out of the box, ZFS is very mature on the FreeBSD platform. TrueNAS specialises in storage, which is what I use it for.

I know some people have issues with TrueNAS been run in a VM, maybe that's why the scale product exists, so to capture those who are using Proxmox or ESXi and to get them to use TrueNAS for everything bare metal.

My TrueNAS core VM is configured with two QEMU mirrored virtual disks for booting, and I passthrough a AHCI controller with disks attached for my storage disks. I have configured the host so it doesn't even see the disks attached to the controller, TrueNAS has complete exclusivity access to them.

I am surprised how many are using TrueNAS Scale, a beta product for storage, I always feel don't take chances with storage, hence for me it would be year's before I considered using it.

I just hope Core is not abandoned at a later date.
 

JOduMonT

Dabbler
Joined
Jan 27, 2015
Messages
29
I am surprised how many are using TrueNAS Scale, a beta product for storage, I always feel don't take chances with storage, hence for me it would be year's before I considered using it.

I totally get the scarcity and your point of view;
but from what I understand; the only and in BETA part is the portage of the middleware from BSD to Linux.
For ZFS; I was already using it with Proxmox support it since few years.

Personally I only see the BETA part as stage...
In the Linux world, distros and companies such, as Ubuntu and Canonical, are build on BETA packages (Debian testing)

...because all part independently are pretty solid.
- The middleware proposed by Free/True NAS is there since forever and pretty stable
- ZFS is more and more taking over in the Linux world (the non adopting part is mainly around the licence, not the stability)
- Debian is also pretty stable and a long-term player
and at the end; with TrueNas it is pretty easy to backup and restore data.

At the end, if you look at any of their Release Note, you could see we are a bunch of Enthusiasts dwelling to expect and report bugs.
1633772732502.png
 
Last edited:

cannfoddr

Dabbler
Joined
Nov 28, 2021
Messages
12
I am looking to migrate my current synology NAS + HP Microserver GEN8 to a single platform. I have been testing SCALE as the main host OS on the new hardware and using some Apps from Truecharts.

I am starting to wonder if I should instead be looking to run Proxmox as the host with SCALE running as a VM, I am currrently constrained to 32GB of ECC RAM - looking to go to 64GB as the next increase.

I have little experience of ProxMox at this stage and would appreciate some recommendations?
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@cannfoddr TBH, I can't imagine you'd get a rush of people recommending Proxmomx over TrueNAS on a TrueNAS forum. But to run SCALE as a VM in anything but test node will require the use of drives attached to HBA passed through to the SCALE VM. Running SCALE as a VM under Proxmox will required 16GB for the VM leaving just 16Gb for anything else.

Your best choice right now depends on your use case and aims, and the limitations of your new hardware, which is?
 

cannfoddr

Dabbler
Joined
Nov 28, 2021
Messages
12
@KrisBee its an ITX based Ryzen 5 System 32GB or ECC DDR-4 in 2 slots and the option to add another 64GB. I have 2 NVME drives and 5 (up to 10) 3.5" HDD and 1 (up to 2) 2.5" SATA.
Plan is replace 10 bay (5+5) Synology 1513+ NAS box and an ageing Microserver Gen8 ESXi + docker server with a Single Box.
My initial focus has been to just use TrueNas Scale - I like the idea of K8S and am comfortable with that as a solution for my containerised apps. I do run a few esxi VMs from time to time - mainly development and project related.
 
Top