What is the future of TrueNAS CORE?

[Patrick M. Hausen]

FreeBSD, Bastille, Ansible ... hey, even iocage exists and "just works" if you want to migrate your TrueNAS jails to stock FreeBSD. Migrating to more active and better architected Bastille can be done afterwards.

 

[Ericloewe]

EuroBSDCon 2023 in Coimbra, Portugal.

EuroBSDCon was in driving distance for a change and I missed it? Damn it.

 

[Patrick M. Hausen]

EuroBSDCon was in driving distance for a change and I missed it? Damn it.

It's the single most important conference for me and my professional life every year. This year in Dublin. Sláinte!

 

[victort]

FreeBSD, Bastille, Ansible ... hey, even iocage exists and "just works" if you want to migrate your TrueNAS jails to stock FreeBSD. Migrating to more active and better architected Bastille can be done afterwards.

I would prefer Bastille to be honest. I really enjoy having ZFS and all the SMART checking, replication, monitoring, and so forth from the GUI. Stock FreeBSD would work, but there's a lot of manual work required to get things streamlined. SNMP server, SMART checking, etc...

I could set all this up manually, but that's one reason I like TrueNAS. It has the GUI to do all that. I've had too much grief with SCALE. It near impossible to set up something like Meshcentral and services that are "config-heavy" and require a config file to manage properly. Especially once you need to make a change after having it set up. And the Truecharts apps (they are improving) haven't gained enough stability IMO.

For now the 13.2-RELEASE is ok, but my time is limited to certain months of the year, when the rest of my commitments are not as time demanding.

 

[Patrick M. Hausen]

I've had too much grief with SCALE. It near impossible to set up something like Meshcentral and services that are "config-heavy" and require a config file to manage properly.

And they still have not answered the very fundamental question of how to address updates that involve complex config changes and manually invoked commands. Like Nextcloud.

If you deploy Nextcloud now via a SCALE app and in a year or two decide to update that - because you just didn't care in the meantime and "hey, it's working, right?" - who will do the mandatory database scheme migrations? Of which I am not entirely sure from the top of my head if they will work at all if you skip major releases.

That was frequently named as a problem with CORE plugins - and it is! But it is in no way caused by the underlying container technology. It's a matter of taking responsibility for the entire lifecycle of an application.

We do that for our hosting customers. We have a "patch day" every month and we update everything. Yes, that means coding every couple of months for things like Nextcloud, but hey, two days of Ansible scripting per month to be able to keep >1000 customer instances running and up to date? That's more or less my job.

February patch day is drawing near and I have to upgrade every PostgreSQL instance from 13 to 15 ... yeah!

 

[Ericloewe]

Apropos migrations... There is one thing I know handles those properly: GitLab, specifically their "Omnibus" package. It's a Faustian deal of sorts, you surrender all control over packages and give up on scalability - but they do everything for you. At the cost of a monster image. And you still can't skip arbitrary versions, you have to look up their byzantine upgrade paths (fortunately, they recently created a wizard, because the tables were getting out of hand) and download another monster to run it for half an hour while it migrate things.

My point is that these things are very hard problems, fundamentally because the castle (house of cards?) is being built on quicksand, inside a sandbox made of paper floating in a swamp, on top of a tectonic fault.

 

[victort]

Then again, life is too short to try to always have your way. Like @Jailer mentioned, it could very well be that some of us will be dragged into the world of Linux. All the services I run can also be run on SCALE, so who knows where I will ultimately choose to go. I do know that if it is too much trouble, I will choose the thing that is easiest to manage because again, life is short, so let's make the most of it.

 

[victort]

You might have heard the phrase "Who's gonna bell the cat?"

A group of mice got together to solve the issue of having their number continuously decrease due to the presence of a household cat. Everyone piped up and gave their opinions. Then, one young, up and coming mouse piped up. "Let's put a bell around it's neck so that whenever the cat is nearby, we will hear the bell and be able to escape." Everyone applauded this wonderful advice. Then one old scraggly looking mouse got slowly to its feet, looked around, and said, "Yes, this is a good idea. But I have one question. Who's gonna bell the cat?"

 

[Kris Moore]

I'll try to do my best to answer this as transparently as I can. I realize there are quite a few FreeBSD die-hard fans here in the audience. I myself came from that world as well, so it is something I have sympathy for. Its a very comfortable, clean and in many ways an elegantly simple system. It doesn't change very rapidly, which also means its easy to become expert and then no need to re-learn a lot of new things every few releases. However that is also its Achilles heal for long term viability. We (and many other vendors of FreeBSD based solutions, some still keeping quiet) have shared a concern for years now about its long term viability in the wider marketplace of solutions being based on newer and faster paced technologies. In our own very specific storage space, nearly all the other vendors historically based on BSD have already moved their primary product line to Linux. Then to top it, we've seen a trend towards upstream vendors moving away from supporting it directly, or giving it less and less priority vs a decade ago. It is quickly becoming more and more of a footnote in relevance among the data-center in general. I realize many of the fans on this thread don't want to hear that, but that is the facts.

Take a look at all the work Intel is pouring into Linux for up and coming technologies like CXL, which will be highly relevant to all of us very soon. Does a vendor like iX want to duplicate that effort into the FreeBSD stack, often at a very high cost of time and resource that could have been used for other efforts? Or if we do for that one tech how about others? Pick any other random feature which already has a perfectly viable equivalent on the Linux side of the fence with a very active user and developer base. You have to know where to best spend your (very limited) time and resource, and re-inventing wheels without some very tangible benefit (not just "caching up") isn't a good strategy. That said, I'm not one of those "FreeBSD is dying" doomsayers, I think it won't just "die", its going to just become more and more of a hobbyist and academia focused OS, and that's not necessarily a bad thing. But I think the odds of it having a serious resurgence at this point in wider industry relevance is very very low. Of course it could still happen, and maybe I'll eat my words someday, but just looking at the facts right now in 2024, I'd be hard pressed to place any real bet on that.

This is just a peek behind some of my thoughts into the entire BSD vs Linux situation in 2024. That said, TrueNAS CORE hasn't been deprecated, and 13.1 is planned to start making a showing in Q2. It will be based upon FreeBSD 13.3 and will provide a way to keep running jails and upstream packages for some time to come. It is still a rock-solid NAS and we're expecting to support it for a long while for that use-case. But the numbers also don't lie, SCALE has been the fantastically growing edition (not even close) for 2+ years now. While its still new and plucky, with a lot of maturing still to do, we're seeing that happen at a pace even I didn't expect originally when we started the project.

I will leave you with a parting thought that a lot of internal iXers have heard me say. We live and work in an industry defined by rapid change. Technology moves forward at a rapid pace. The worst thing for an engineer is to get stuck on one particular piece of technology for emotional reasons. We always have to be pushing forward, learning and embracing the new where it makes logical sense for our own use-case. Even Linux itself will be dethroned someday when the next better thing comes along and we should always be mentally prepared to accept and embrace it.

 

[sretalla]

Sad, but fair. Thanks @Kris Moore.

I guess the trick will be finding a way to match the driver reliability for Intel/LSI cards under Linux that were reached in FreeBSD.

 

[Kris Moore]

Sad, but fair. Thanks @Kris Moore.

I guess the trick will be finding a way to match the driver reliability for Intel/LSI cards under Linux that were reached in FreeBSD.

Yes, and that is the kind of thing where we're going to do our best to ensure it is as reliable or better long term. Right now I'd say for "core" NAS technology, the two editions close to on par. Performance similar, ahead in some, behind in some depending on the benchmark being run. The first BETA.1 for 24.04 Dragonfish will launch in only a few weeks, which also addresses the ARC size limitations on SCALE. It will use the same memory % calculation as FreeBSD did now, and that was one last major hurdle for us to cross to reach parity in some of those things. But that said, we're working hard to make sure in the end, the Storage and user experience isn't just "on-par" but better in every way. Will there be bumps? Of course. Its software made by humans. Takes time to polish and get rid of all those rough edges. But we feel we're well on our way at this point :)

 

[Davvo]

@Kris Moore that's absolutely fair, but I think what left us a bad taste in our mouths was the PR around it. And I also believe that the great success of SCALE was at least in part due to iX's commitment not to fix certain things in CORE; I'm not saying that's the main factor, but a concurrent one.

My lack of optimism about SCALE is due to how Linux works in general and its integration of ZFS.

 

[Patrick M. Hausen]

Thanks, @Kris Moore. What's your idea for app life cycle management? This has been a sore point in CORE plugins for years and as I pointed out in no way rooted in the underlying technology stack. Users (rightfully IMHO) expect that if an app/plugin like Nextcloud exists and is touted as "official", after initial installation there will be timely and painless updates to it. Even more so in this particular case when the app's entire raison d'être is to be facing the public Internet.

The strongest argument for running TrueNAS over every other, possibly commercial, NAS platform is that it is also an infinitely versatile application platform. I don't trust SCALE and this helm/k3s/containerd stack enough to use it for real world workloads, yet. Take snapshots, rollback, replication and restore. Dead simple with jails.

Kind regards,
Patrick

 

[Kris Moore]

@Kris Moore that's absolutely fair, but I think what left us a bad taste in our mouths was the PR around it. And I also believe that the great success of SCALE was at least in part due to iX's commitment not to fix certain things in CORE; I'm not saying that's the main factor, but a concurrent one.

Thats fair. I'm sure we can look back and find ways to have improved communication around this.

My lack of optimism about SCALE is due to how Linux works in general and its integration of ZFS.

True, and that was one of our biggest hurdles as well to overcome. Good news is, "Linux" is really just the kernel. The rest of the integration for ZFS is what we make of it in the userland side. Even back before we started, we already noticed OpenZFS beginning to adapt and shift, with the majority of new feature work coming from the Linux contributors among the project. This I think shows the relative health of OpenZFS support on Linux, moving from OpenSolaris to Linux as its primary developer base. It made that transition well and quick, really before anybody even noticed :)

 

[Kris Moore]

Thanks, @Kris Moore. What's your idea for app life cycle management? This has been a sore point in CORE plugins for years and as I pointed out in now way rooted in the underlying technology stack. Users (rightfully IMHO) expect that if an app/plugin like Nextcloud exists and is touted as "official", after initial installation there will be timely and painless updates to it. Even more so in this particular case when the app's entire raison d'être is to be facing the public Internet.

The strongest argument for running TrueNAS over every other, possibly commercial, NAS platform is that is also an infinitely versatile application platform. I don't trust SCALE and this helm/k3s/containerd stack enough to use it for real world workloads, yet. Take snapshots, rollback, replication and restore. Dead simple with jails.

Kind regards,
Patrick

Good question. And I think its fair that K3's / Helm isn't for everybody. We're taking steps to provide our Jail users alternatives in the upcoming Dragonfish BETA in a few weeks. Granted, its still early and not a fully supported UI feature, but its a first step that will allow you to use a technology you may be far more comfortable with for your use case. If the usage trends support it, we may do some additional development in this space to turn it into a full featured aspect of the software.

PD-995: Sandbox (SCALE Jails) article by Mrt134 · Pull Request #2445 · truenas/documentation

Adapt internal notes into a small Sandboxes (Jails) article with cautions about support and being a community-only feature. Thanks for contributing to TrueNAS documentation! By opening a Pull Reque...

github.com

 

[victort]

Excellent discourse! I appreciate your openness about it. It's the best answer I've heard so far, and to be honest, has helped me see your reasons for going the route you are going. Appreciate it greatly.

 

[victort]

What is your opinion on having someone create a fork and potentially keeping up the CORE line?
What about the FreeNAS name?

Is there a solution where both parties can end up getting what they desire, be it in a fork, or something similar?

 

[Jailer]

We're taking steps to provide our Jail users alternatives in the upcoming Dragonfish BETA in a few weeks. Granted, its still early and not a fully supported UI feature, but its a first step that will allow you to use a technology you may be far more comfortable with for your use case

Please let that be LXC. That would bode well to win over some of the CORE users (such as myself) that are not looking forward to having to move away from jails if we want to stick with TrueNAS.

 

[Kris Moore]

What is your opinion on having someone create a fork and potentially keeping up the CORE line?
What about the FreeNAS name?

Is there a solution where both parties can end up getting what they desire, be it in a fork, or something similar?

Its Open Source, so of course forks happen. But I would caution, its a HUGE commitment. Speaking from experience, all the resources required to maintain, build, release, troubleshoot, etc. Never mind any new feature work.. Its a very non-trivial project at this point. We're talking multiple people working as full time engineers and full-time support kind of commitment required, otherwise the quality would greatly suffer over the long run. If the reason is only to maintain its base on FreeBSD, I don't see the payoff personally. Even as much as I loved FreeBSD, that's not something I could do anymore for my own passion projects like PC-BSD or TrueOS (Both FreeBSD). I needed to have a life as well. But that's just my 2C on the situation :)

 

[RetroG]

yes, FreeBSD (and the other BSDs for that matter) are all niche now... but honestly the entire (server) market becoming Linux is a huge issue, we need other valid options otherwise the Q/C of Linux will drop.

And only slightly related. IMO Linux just doesn't have the rock solid storage stack that BSD has, and I wouldn't be saying that if I didn't spend an annoying amount of time chasing down 4kn race conditions on Linux based servers (that have at least been fixed), or dealing with 4kn USB disk support (which still doesn't work on Linux, but does on basically every other contemporary OS). this is one issue but there are many that I've dealt with over the years like it.