Virutalize Truenas or virtualize on Truenas ?

[Dr.Pepper42]

I’m currently running a Truenas Scale Bluefin build with the intention of having a monolithic design that consolidates data storage, vms and containers in one node.

I’ve come to realize (as many others) that the intentions of iX systems and my goal seem to diverge quite a bit. One example for this is the lack of a proper way to specify and deploy containers without using the rather limited UI, K3s instead of K8s, etc.

While Truecharts attempts to mitigate some of these issues, I’ve come to realize that their approach is not transparent at all and instead I fear for my configs and data when updates are released.

This leaves me with the option of either:
(1) setting up a vm with docker+docker compose on Truenas Scale
or
(2) virtualize Truenas entirely on Proxmox or ESXi.

It seems rather obvious to choose the first option since users like @jgreco have heavily advocated against virtualizing Truenas in countless threads on this forum.

However, when setting up a vm in Truenas it is not possible for it to connect to the host by default. There seems to be a “hack” to fix this but this gives me the impression that it is not intended by the devs and therefore not supported. What is the "proper" course of action here ?

I’m also unsure how mapping resources from Truenas to the docker VM and on to containers will impact performance?

(I’m currently researching a new build based on an Dell R730+HBA330+RTX A4000+ that is suitable for ML tasks)

 

[sretalla]

One example for this is the lack of a proper way to specify and deploy containers without using the rather limited UI, K3s instead of K8s, etc.

Maybe will one day become official, but there's this if you're really concerned about running your own stuff:

Use a custom "Jail" in SCALE? (Possible? Alternatives?)

Being less familiar with Docker/Kubernetes and containers, while having good results with Jails under TrueNAS Core, I'm curious if it's possible to do the following in SCALE: Have a standalone "from scratch" container with specific CLI tools, mount points, and the ability to SSH into such a...

www.truenas.com

There seems to be a “hack” to fix this

If you mean a documented process, then yes: https://www.truenas.com/docs/scale/scaletutorials/virtualization/accessingnasfromvm/

I’m also unsure how mapping resources from Truenas to the docker VM and on to containers will impact performance?

you could try both the "default"/documented method and the one I linked in the first answer above and compare performance...

As I understand it, the first linked one would perform better.