What is the future of TrueNAS CORE?

[Kris Moore]

Please let that be LXC. That would bode well to win over some of the CORE users (such as myself) that are not looking forward to having to move away from jails if we want to stick with TrueNAS.

We opted for now to use systemd-nspawn, which is much lower level like Jails are. It means you can run LXD/LXC, Docker, or $OTHER container tools within. Granted if we decide to build something directly into the product further we'll look at all options when the time comes.

jailmaker/docs/incus_lxd_lxc_kvm.md at develop · Jip-Hop/jailmaker

Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn! - Jip-Hop/jai...

github.com

 

[Kris Moore]

yes, FreeBSD (and the other BSDs for that matter) are all niche now... but honestly the entire (server) market becoming Linux is a huge issue, we need other valid options otherwise the Q/C of Linux will drop.

And only slightly related. IMO Linux just doesn't have the rock solid storage stack that BSD has, and I wouldn't be saying that if I didn't spend an annoying amount of time chasing down 4kn race conditions on Linux based servers (that have at least been fixed), or dealing with 4kn USB disk support (which still doesn't work on Linux, but does on basically every other contemporary OS). this is one issue but there are many that I've dealt with over the years like it.

I would add that please nobody interpret what I'm saying as "BSD Sux - Linux is sooo PERFECT!". I'm not that much of a fanboy for anything. All systems are a mixture of the good, the bad and the downright ugly.

 

[Jailer]

I would add that please nobody interpret what I'm saying as "BSD Sux - Linux is sooo PERFECT!". I'm not that much of a fanboy for anything. All systems are a mixture of the good, the bad and the downright ugly.

Anybody who knows even just a little bit of your history knows better than to think something like that. You'd have be pretty ill informed to think that you would take a negative stance against BSD.

 

[victort]

There are absolutely pros and cons to either option. FreeBSD is stable, solid, and reliable. Linux has more devs working towards containers etc… Uptime-Kuma is one example of FreeBSD just not being supported, but fully supported on Linux/Containers.

Simple facts have been laid out I think. You won’t win by working on something that no one uses. But at the same time, the fact that no one uses it does not make it a bad product.

Reality is reality I suppose. I appreciate the transparency in this thread. It helps us make informed decisions on our way forward.

I will probably opt to install SCALE, have a FreeBSD VM, and my jails hosted there if I have to.

 

[victort]

Ummm… Do you read minds? @Jailer

 

[Jailer]

Ummm… Do you read minds? @Jailer

Occasionally....but it's just a hobby.

 

[RetroG]

I would add that please nobody interpret what I'm saying as "BSD Sux - Linux is sooo PERFECT!". I'm not that much of a fanboy for anything. All systems are a mixture of the good, the bad and the downright ugly.

this wasn't my intent, sorry if it seemed that way.

but I really do think we need all the systems as you said, with their good and bad. Linux was just the example I used as it is the current dominant server OS.

 

[winnielinnie]

It will be based upon FreeBSD 13.3 and will provide a way to keep running jails and upstream packages for some time to come.

Why not leapfrog to 14.x? This will yield support for jails, and their respective software, for the next 4 to 5 years.

This is true, regardless of maintenance (or lack thereof) from iX.

 

[Kris Moore]

Why not leapfrog to 14.x? This will yield support for jails, and their respective software, for the next 4 to 5 years.

This is true, regardless of maintenance (or lack thereof) from iX.

The amount of enterprise stability fixes and hardening we do every time we jump to a new .0 of FreeBSD is a bit ridiculous. It can be done, but we'd prefer to stay on 13.x for the time being. This is bigger than just jails functionality, since we support Enterprise customers there as well. We've dialed in 13 a LOT to make it very stable with HA and our Enterprise functionality. That kind of stability doesn't come for free.

 

[Patrick M. Hausen]

Good question. And I think its fair that K3's / Helm isn't for everybody. We're taking steps to provide our Jail users alternatives in the upcoming Dragonfish BETA in a few weeks.

I am aware of that project but reluctant to use it because jails just work so well. Why re-invent the wheel?

I appreciate all your open replies in this thread, but please, you did not address my main concern with any app/plugin ecosystem - to cite:

What's your idea for app life cycle management? This has been a sore point in CORE plugins for years and as I pointed out in no way rooted in the underlying technology stack. Users (rightfully IMHO) expect that if an app/plugin like Nextcloud exists and is touted as "official", after initial installation there will be timely and painless updates to it. Even more so in this particular case when the app's entire raison d'être is to be facing the public Internet.

If you want SCALE and apps to be a success, this is crucial, IMHO. And it's not about FreeBSD vs. Linux, not at all.

Kind regards,
Patrick

 

[Patrick M. Hausen]

The amount of enterprise stability fixes and hardening we do every time we jump to a new .0 of FreeBSD is a bit ridiculous. It can be done, but we'd prefer to stay on 13.x for the time being. This is bigger than just jails functionality, since we support Enterprise customers there as well. We've dialed in 13 a LOT to make it very stable with HA and our Enterprise functionality. That kind of stability doesn't come for free.

Understood. That's why if I was to fork the project I would undo all the enterprise features and try to create a clean and working build on top of a stock FreeBSD. For single server without HA only, but with up to date jails, bhyve, ZFS etc.

The middleware and UI could even be made a port to install on top of a stock FreeBSD.

 

[victort]

I am aware of that project but reluctant to use it because jails just work so well. Why re-invent the wheel?

I appreciate all your open replies in this thread, but please, you did not address my main concern with any app/plugin ecosystem - to cite:


If you want SCALE and apps to be a success, this is crucial, IMHO. And it's not about FreeBSD vs. Linux, not at all.

Kind regards,
Patrick

Agreed. For me it’s about usability. Not being able to manage configs without having to edit an app is a big one for me. Hence why I created scripts. They do the basic work, then allow you to continue the config editing from the jail, and have the config persist.

Also the ability to store data files outside the jail, database included.

Stock FreeBSD plus a minimalistic GUI to manage jails would be ideal. My 2 cents…

 

[Kris Moore]

I am aware of that project but reluctant to use it because jails just work so well. Why re-invent the wheel?

I appreciate all your open replies in this thread, but please, you did not address my main concern with any app/plugin ecosystem - to cite:


If you want SCALE and apps to be a success, this is crucial, IMHO. And it's not about FreeBSD vs. Linux, not at all.

Kind regards,
Patrick



What's your idea for app life cycle management? This has been a sore point in CORE plugins for years and as I pointed out in no way rooted in the underlying technology stack. Users (rightfully IMHO) expect that if an app/plugin like Nextcloud exists and is touted as "official", after initial installation there will be timely and painless updates to it. Even more so in this particular case when the app's entire raison d'être is to be facing the public Internet.

Sorry, I missed that originally. This is part of the reason for us doing what we are doing in SCALE today. People may not like K3s, but the Helm charts we used for our App system have been pretty darn good. Its allowed us to for the first time ever properly support apps and their lifecycle in a much more controlled manner. Since SCALE debuted back in Angelfish days, I've personally been using it to manage my humble App "farm", and things have just tended to work. I have yet to have a single Plex update (Now Jellyfin) go bad, I just click the update button and away it goes. Back up in < 1 min or so. Granted its not perfect, we're always going to be fighting issues if upstream projects reorganize things too wildly with their containers, but for the first time it is possible to run an "Official" app, such as Nextcloud, straight from the source themselves, NextCloud. Those tend to "just work" as long as you don't go too nuts customizing them beyond what the container maintainers allow for.

Granted, I can't vouch for 3rd party chart catalogs, they will have their own issues based on how they decide to manage their own catalogs. But in our "official" catalog, we have Enterprise apps now, such as Minio and Syncthing. They have to "just work" with painless updates, which we've managed to do with our carefully controlled repo. Our Charts repo even uses automation to automatically open PR's to Github when upstream containers are updated, so we get updates at a much more rapid pace than before. Additionally when a bug has been found, we've been mostly successful in resolving it for the user on the chart side, so that the App doesn't have to be fully re-deployed. That said, no system is perfect, and we're continuously looking at options to see what we can do to make it even more simplified and stable, so that App updates are 100% painless and allow us to get back to our busy lives as well. Speaking for myself, once i took the plunge and migrated my Plugins/Jails to my pretty standard Apps, it has been a huge timesaver.

 

[Patrick M. Hausen]

So you are using some upstream Nextcloud chart to deploy Nextcloud? I wasn't aware that exists. I install Nextcloud from source, it's just PHP after all. NginX, PHP-FPM, MariaDB - done.

Do these charts have some mechanism that does the mandatory occ upgrade?

 

[Kris Moore]

So you are using some upstream Nextcloud chart to deploy Nextcloud? I wasn't aware that exists. I install Nextcloud from source, it's just PHP after all. NginX, PHP-FPM, MariaDB - done.

Do these charts have some mechanism that does the mandatory occ upgrade?

Yes. Using Nextcloud as an example, we pull from the upstream container / chart and use it for the TrueNAS App we present in the UI. We don't go around hand-rolling container images for everything just for grins, we always try to use the "official" or "semi-official" container images provided by the upstream projects themselves. Plex, Jellyfin, Nextcloud, Immich, Minio, Syncthing, etc etc. Nearly all projects have some sort of Linux container image they provide for this exact purpose. On the FreeBSD side, not so much. We had issues even getting upstream to look at problems we reported due to the fact its was run / compiled on FreeBSD. Many of them are Linux-only shops, which makes sense.

Some references for you:

GitHub - nextcloud/docker: ⛴ Docker image of Nextcloud

⛴ Docker image of Nextcloud. Contribute to nextcloud/docker development by creating an account on GitHub.

github.com

charts/charts/nextcloud/app_versions.json at master · truenas/charts

TrueNAS SCALE Apps Catalogs & Charts. Contribute to truenas/charts development by creating an account on GitHub.

github.com

Container | Jellyfin

Install as a container using Docker, Podman and others.

jellyfin.org

OCC Upgrade in official container:

docker/docker-entrypoint.sh at master · nextcloud/docker

⛴ Docker image of Nextcloud. Contribute to nextcloud/docker development by creating an account on GitHub.

github.com

 

[Patrick M. Hausen]

 

[Whattteva]

FreeBSD, Bastille, Ansible ... hey, even iocage exists and "just works" if you want to migrate your TrueNAS jails to stock FreeBSD. Migrating to more active and better architected Bastille can be done afterwards.

Plus one for Bastille.

I've been using Bastille for a dozen jails I manage on a vanilla FreeBSD VM now. I particularly love the "Templates" feature and you can even use the same config file on the host with "Rocinante". I think they've built a robust simple system and it doesn't require ZFS the way iocage does.

 

[Whattteva]

On the FreeBSD side, not so much. We had issues even getting upstream to look at problems we reported due to the fact its was run / compiled on FreeBSD. Many of them are Linux-only shops, which makes sense.

I've actually had good luck with a number of them. I opened issues with Jellyfin and jellyseerr maintainers and the devs were very responsive.
In fact, the dev fixed Jellyfin port that got broken with FreeBSD 14.0 (due to OpenSSL upgrade) fairly quickly. I do understand that the Jellyfin port is unfortunately not maintained by the official Jellyfin devs.

 

[morganL]

Thanks, @Kris Moore. What's your idea for app life cycle management? This has been a sore point in CORE plugins for years and as I pointed out in no way rooted in the underlying technology stack. Users (rightfully IMHO) expect that if an app/plugin like Nextcloud exists and is touted as "official", after initial installation there will be timely and painless updates to it. Even more so in this particular case when the app's entire raison d'être is to be facing the public Internet.

The strongest argument for running TrueNAS over every other, possibly commercial, NAS platform is that it is also an infinitely versatile application platform. I don't trust SCALE and this helm/k3s/containerd stack enough to use it for real world workloads, yet. Take snapshots, rollback, replication and restore. Dead simple with jails.

Kind regards,
Patrick

I think this post and Kris's answer captures the core (pardon the pun) issues that users face.

As Kris pointed out Linux containers provide better App lifecycle management where the App developer has more control and responsibility. App developers prefer this model. Most users enjoy this simplicity.​

​

However, FreeBSD Jails are simple and powerful, if the (more technical) administrator wants to manage their own App lifecycle.​

Unfortunately, its one or the other at the moment. That's not something we planned, it's just reality.

We are working with nspawn containers to see if there is a middle ground, but it's early days. Jails are still a major strength of TrueNAS CORE.

The good news is that 13.1 is in process.. now planned for Q2 with latest FreeBSD 13.x , Samba and ZFS updates. So this debate and comparison can go on for many more months/years. Jails are supported.

What is being under-discussed is that TrueNAS does make sidegrades from CORE to SCALE relatively simple and enables consistent APIs and storage pools. Only Jails/Apps are different. The goal is to provide choice....users can stay on CORE until they see advantages of moving to SCALE. Most of our commercial users will stay on CORE/13.x for many years. iX plans to consult and listen as we make future plans.

 

[Davvo]

What is being under-discussed is that TrueNAS does make sidegrades from CORE to SCALE relatively simple and enables consistent APIs and storage pools. Only Jails/Apps are different.

I'm not sure sidegrading to SCALE today would allow me and my system for a relatively simple moving of my jails with mounting points without data loss, but that might just be me not having looked deep enough in the process... as I don't even know whether mount points are even possible in a VM under SCALE (which as of today looks to be the cleanest way of moving to SCALE).

That being said, being able to hop from one to the other is indeed a great thing. None here undermines the great work you have done on SCALE.