WARNING - don't upgrade your TrueNAS CORE jails to FreeBSD 13.3 just yet

[Patrick M. Hausen]

EDIT: FreeBSD 13.2 will be supported until early June 2024 and jails with 13.2 run just fine on the current version of TrueNAS CORE. So just don't upgrade your jails to 13.3 until after the release of TN CORE 13.1.

EDIT2: If you have a jail already upgraded to 13.3 and neither a roll back from snapshots nor a complete reinstall with a 13.2 jail is an option for "reasons" you can try this from the TrueNAS host system: cp -p /usr/sbin/daemon /mnt/<pool>/iocage/jails/<jail>/root/usr/sbin/

Hi all!

With FreeBSD 13.3 just released and considering that one should run up-to-date and supported software I upgraded all my jails from FreeBSD 13.2 to FreeBSD 13.3.

All jails that use the daemon(8) facility to start their services failed to do so.

Investigation revealed that while in theory no breaking changes to the ABI should happen in a FreeBSD -stable branch like releng/13, in this particular case a new system call, kqueuex(), was merged into releng/13 and releng/13.3. I just sent a message to the -stable mailing list to verify.

For the moment this is not a big deal - I just rolled back to 13.2 and all was well.

We, the entire community, will face a major road block in April, though. Starting with the next quarter packages will be built for 13.3 instead of the EOL 13.2. Which means we will probably not be able to reliably deploy new jails by then.

This is exactly the situation we have been in multiple times in the past and which I predicted for some time in 2026 because TrueNAS CORE always lags behind FreeBSD releases.

Looks like I will be jumping to the nightlies rather sooner than later. April the very last.

Kind regards,
Patrick

 

[winnielinnie]

This is one of those posts that I wish I could give a "thumbs up" and a "frowny face".

 

[Davvo]

I'm not quite sure I'm getting the issue... why shouldn't we able to upgrade from 13.2 to 13.3?
Any way to get around this? I'm running standard pkg trains on almost all of my jails, except for a few where i run latest... anyway, gonna pause the crontabs for now.

 

[winnielinnie]

I'm not quite sure I'm getting the issue... why shouldn't we able to upgrade from 13.2 to 13.3?

My understanding is, the sooner we're graced with TrueNAS Core 13.3 (which will be based on FreeBSD 13.3), the better for those of us using jails.

The reason being, there's a "lag window" between iX and upstream FreeBSD. During such a window, you can't safely upgrade your jails to 13.3 (since TN Core is still based on FreeBSD 13.1, combined with the issue @Patrick M. Hausen mentioned above.) But the double-whammy is that keeping your jails on FreeBSD 13.2 isn't feasible, since it is EoL and hence cannot receive pkg updates.

EDIT: It's pretty much switch to SCALE or a "fork" of Core after 2026.

 

[Patrick M. Hausen]

The daemon(8) tool in 13.3 relies on a system call that is simply not present in 13.1. The latter is the kernel and OS version of TN CORE 13.0.

Many "smaller" services rely on the FreeBSD builtin daemon(8) for setup: closing file descriptors, opening log and pid files, dropping privileges, putting the process in the background. This is commonly called "daemonizing", hence the name of the tool.

While the "large" products like NginX, MariaDB, FPM, etc. all have their own routines for that, so e.g. my Nextcloud jail started just fine.

The services using daemon(8) which failed to run were for me: Grafana, Gitea, and Vaultwarden.

Even though these packages are still 13.2 binaries, it is the daemon(8) tool itself which in the 13.2 version works on TN CORE 13.0 and in the 13.3 version doesn't. So you upgrade your jail - boom.

What happens to the "large" products when the FreeBSD package build cluster switches from 13.2 to 13.3 one can only guess. It all depends on the question if the particular software uses the kqueuex() system call or not. Possibly every piece of software will, because it's in libc - then again possibly only some tools.

tl;dr 13.3 introduces a new feature that is not present in 13.1 and so TN CORE. All software will fail to run when it is built for 13.3 and uses that feature.

 

[victort]

Well that's not very convenient. Assuming CORE will be based on 13.3, will things work at that time?

 

[Patrick M. Hausen]

Assuming CORE will be based on 13.3, will things work at that time?

Of course. That's what I tried to explain.

 

[victort]

Of course. That's what I tried to explain.

Sometimes we are just to lazy to read. I apologize.

 

[winnielinnie]

Can confirm: This broke my Jellyfin jail. Had to rollback the jail (via the automatic safety snapshots) to 13.2-RELEASE-p10.

 

[Kris Moore]

And now people don't need to wonder why I made comments about FreeBSD "ABI Compat" not really living up to it's name in practice. The real fun begins when you upgrade kernel to 13.3 and find out some older jail doesn't work properly as well. I got some grey hairs over that at least a few times ;)

 

[Patrick M. Hausen]

The real fun begins when you upgrade kernel to 13.3 and find out some older jail doesn't work properly as well.

I never had a problem with an older jail on a newer kernel, but YMMV as always.

I just sent a message to the -stable mailing list to verify.

Confirmed. Addition of new system calls is explicitly permitted according to Alan Somers. After all older binaries never use them so no ABI breakage this way round.

 

[Davvo]

Was under the impression that smaller releases (13.1, 13.2) tried not to introduce breaking changes (12, 13).
My lack of familiarity with FreeBSD's production cycle showing off here.

 

[winnielinnie]

Was under the impression that smaller releases (13.1, 13.2) tried not to introduce breaking changes (12, 13).

I was under the same impression, which I found applaudable as a benefit of FreeBSD over Linux.

 

[Patrick M. Hausen]

Basically they revisited the daemon breaking things.

Well ... not quite. At least Alan argued that a newer (13.3) kernel does not break anything for older (13.1, 13.2) binaries and also jails. The problem was trying to run a newer binary (/usr/sbin/daemon from 13.3) on an older (13.1) kernel.

I can follow that. But that means the slow pace at which TN CORE gets updates is even more of a problem ...

When TN 13 was announced I read somewhere that iX would track the releng/13 branch for that. So I fully expected every minor update to also come with an up to date version of FreeBSD 13 from the -stable branch. Not necessarily a release, but who cares. releng/13 from 4 weeks ago very probably runs 13.3 binaries just fine. releng/13 from 13.1 release time obviously doesn't.

 

[Davvo]

Anyway, commented out my crontabs while waiting for CORE 13.2... but now I do wonder if there is a need to do so? Or should I just not upgrade the jails?
Thread should be pinned.

 

[winnielinnie]

Anyway, commented out my crontabs while waiting for CORE 13.2.

I just rolled back my jails (via snapshots) to 13.2-RELEASE-p10 again. At least we have until April, and Core 13.3 shouldn't be too far out. (Hopefully?)

 

[Davvo]

I dont know by which logic, but I got the message that we should not pkg update && okg upgrade until we get to FreeBSD 13.3. I'm losing the chess battle with myself here.

iirc estimates were for second half of 2024?

 

[winnielinnie]

My fear is that if iXsystems essentially abandons FreeBSD after 2026 (and thus, "TrueNAS" is synonymous with "SCALE" / "Linux"), then it will be a double-whammy against FreeBSD and software availability on FreeBSD.

Here's my train of logic:

1. A sizable portion of "FreeBSD" users are in fact FreeNAS / TrueNAS Core users. (They wouldn't be "using" FreeBSD, otherwise. Myself included.)
2. If they stop using FreeBSD and jails (under TrueNAS), then the expectation and demand for software support/availability on FreeBSD will decrease.
3. Less of an incentive for maintaining existing FreeBSD ports; and less incentive of making Linux software accessible and ported to FreeBSD.
4. All users of FreeBSD are affected by this.

How did I come to reason this way? Arch Linux.

A massive portion of "Arch Linux" users do not in fact use vanilla Arch Linux, but rather desktop-centric and "user-friendly" derivatives, such as Garuda, Endeavor, and Manjaro. This increases the overall userbase for "Arch Linux", and hence, its packages. This larger pool of users also increases the amount of potential package maintainers and even AUR packagers. (Think of the AUR as a "Makefile repository" for software not available in the official repositories.)

I, myself, being a package maintainer of some AUR packages, can say with certainty that if Garuda, Endeavor, or Manjaro didn't exist, I wouldn't be a PKGBUILD maintainer of these applications.

In other words, when TrueNAS Core ceases to be, FreeBSD itself will take a huge hit in the wider market.

 

[Patrick M. Hausen]

Sometimes we are just to lazy to read. I apologize.

No need to - all good.

 

[Patrick M. Hausen]

@winnielinnie That's why the FreeBSD foundation and other interested parties are talking about the future of the project. But let's not throw out the baby with the bath water just yet.