Possible to upgrade Kernel?

[BirdBare]

I recently bought a 5700g APU to use with Scale so I can get my VM working.

5700G temperature monitoring is supported with the 5.15 Linux kernel.

Is it possible to update the kernel from within Scale or do I need to rebuild Scale with a newer kernel version? If I have to rebuild, do you have a guide I can follow?

 

[sretalla]

Is it possible to update the kernel from within Scale or do I need to rebuild Scale with a newer kernel version? If I have to rebuild, do you have a guide I can follow?

TrueNAS is an appliance which packages together an OS and some services. If you mess with any of those components, it isn't the same thing anymore.

If you want to do a custom-compile, go right ahead, it's open-source, but you won't be Running TrueNAS anymore. Documentation is already out there on how to compile.

If you want a supported appliance, wait for the kernel to be updated in the main project (or even request it on Jira to perhaps make it go faster).

 

[BirdBare]

Fair enough. Thanks!

 

[anodos]

We are sticking to LTS kernel branch. If you decide to build a new kernel, you'll need to take our kernel patches as well, then rebuild ZFS, samba, and py-libzfs. Also if you do this, please don't file any bug reports :)

 

[nstiurca]

Did you ever try a different kernel or otherwise get your APU & VM working? I think I'm probably wanting to do the same thing as you if I've guessed your intent right.

 

[anodos]

Did you ever try a different kernel or otherwise get your APU & VM working? I think I'm probably wanting to do the same thing as you if I've guessed your intent right.

IIRC we'll update kernel version for BlueFin (next major release), but not before that. Upgrading kernel in an unsupported manner in the appliance will most likely result in breaking ACL support and other things.

 

[Arwen]

Their is reason to stay off of 5.15 even though it's now been listed as LTS, (Long Term Support). Basically the 5.10 kernel releases are going to be supported til Dec, 2026. While the 5.15 is only listed as being supported til Oct, 2023. That is less than 2 years away.

Reference:
Linux active kernel releases - Long term support

Of course, the expected EOL date of 5.15 may change.

 

[cap]

Their is reason to stay off of 5.15 even though it's now been listed as LTS, (Long Term Support). Basically the 5.10 kernel releases are going to be supported til Dec, 2026. While the 5.15 is only listed as being supported til Oct, 2023. That is less than 2 years away.

Reference:
Linux active kernel releases - Long term support

Of course, the expected EOL date of 5.15 may change.

The newer the kernel, the more hardware is supported.
As far as I know, you need the latest kernel for i225-V (2.5GBe) support, for example.

 

[cap]

We are sticking to LTS kernel branch.

Linux Kernel 5.15 is LTS!

IIRC we'll update kernel version for BlueFin (next major release), but not before that.

Does this mean that there will be a newer kernel only with BlueFin (so probably in 2023) and Angelfish will stay with 5.10?

 

[Arwen]

The newer the kernel, the more hardware is supported.
As far as I know, you need the latest kernel for i225-V (2.5GBe) support, for example.

Yes. And 2 years might be more than long enough for a NAS release, before going to a newer kernel.

At home for Linux, I stick to LTS kernels. That way I get some updates, but don't have to go through the mess of dealing with a new kernel.

 

[truecharts]

Linux Kernel 5.15 is LTS!


Does this mean that there will be a newer kernel only with BlueFin (so probably in 2023) and Angelfish will stay with 5.10?

We've also previously discussed this with iX developers (along with a few more requested updates)
The problem is that 5.15 was released when Anglefish was already in Release Candidate and it's best-pracice not to update kernels during RC.

So no, Anglefish will not have 5.15 and will stay on 5.10.
Kernel update for Bluefin is likely, not sure if a definative call for it is made yet considering the current state (early draft) of bluefin plans.

 

[cap]

OK. Thanks for the answer.

 

[whoschek]

IMO, would have been better to delay the Scale release by a few weeks than to end up with an LTS kernel that's outdated by the time of the scale release and having to bear that burden for years. The "it's an appliance" formula tends to be too simplistic for the real world.

In any case, I'm on very recent hardware - Alder lake, USB-to-2.5GbE Ethernet adapter using Realtek RTL8156, etc. Kernel 5.10 is too old for such recent hardware. I really need to get onto 5.15 or 5.16. So what's the suggested way to upgrade the kernel for scale?

This doesn't work (on adding bullseye-backports):
apt install -y -t bullseye-backports linux-headers-5.15.0-0.bpo.2-amd64 linux-image-5.15.0-0.bpo.2-amd64

I'm thinking to rebase the changes in https://github.com/truenas/linux/commits/SCALE-v5.10-stable onto 5.15 and run https://github.com/truenas/scale-build to build a new ISO.

Thoughts?

 

[jgreco]

The official word is that you are not expected to be able to upgrade the kernel, certainly not a significant version bump. Already answered in this thread, in fact.

IIRC we'll update kernel version for BlueFin (next major release), but not before that. Upgrading kernel in an unsupported manner in the appliance will most likely result in breaking ACL support and other things.

 

[Arwen]

@whoschek - That is a Linux only problem. In the case of Solaris, I was routinely able to run hardware drivers from earlier releases on later OS releases.

This is one of my biggest complaints about Linux, the EXTREME need to re-compile drivers for every single kernel change. It's Linux official policy that kernel interfaces can change with every release.

So your comment about 5.10 being too old, and the release of TrueNAS SCALE should have been delayed to support the latest branch of LTS Linux kernel, could apply to any kernel. And for any new bizwang new dodad.

This Linux kernel issue is actually something that RedHat and other Linux vendors are experiencing. RedHat dropped support for BTRFS because they would have to backport every single update to the kernel driver for BTRFS back to the stable kernel RedHat used for their release. Eventually RedHat said no.

---

@cap - As for i225-V (2.5GBe) support, it's already baked into 5.10.52, (the version I checked), or later, (SCALE is using .82 if I remember correctly).

 

[whoschek]

@Arwen 5.10 is too old to effectively run current gen Intel CPUs (i.e Alder lake) which is what a lot of folks will be asking for to be supported. Also, people need to be able to install urgent kernel security updates without waiting for Truenas releases. In practical terms, the decision to not delay the SCALE release for 5.15 and to not suggest straightforward means to upgrade to a more recent kernel will force folks to look elsewhere. Release mgt always involves a lot of tradeoffs, but this seems like an obious poor business decision to me.

 

[jgreco]

this seems like an obious poor business decision to me.

That's a disgusting choice of words, in my opinion.

Here you have a company, who is giving you their NASware for free, and you are calling it a poor business decision that they're not going out of their way to support the very latest hardware.

iXsystems has a goal of being able to run TrueNAS on the hardware platforms that they sell as TrueNAS Enterprise. They have already been incredibly generous by allowing you to run Core or Scale, for free, on the hardware of your choice. They are not currently selling TrueNAS systems with Scale, so, for them, maintaining stability by sticking to the kernel that they've been using probably makes a lot of sense.

We would all be better off remembering that the developers of this stuff do not work for free, and that until we are paying customers, perhaps we should not be hypercritical of the choices that are being made. They are trying to develop a product which will in turn bring in their paychecks. They have taken a huge gamble on the direction of the product. We should not be demanding the absolute latest kernel to support the absolute latest bit of hardware possible. It is much better to have a thoroughly vetted, very well tested platform.

 

[whoschek]

@jgreco What I'm saying is that with this business model the company is unlikely to attract Truenas SCALE customers and thus additional revenue.

 

[jgreco]

business model the company is unlikely to attract Truenas SCALE customers

There is no such thing as a "TrueNAS SCALE customer".

The FreeNAS and TrueNAS software communities primarily serve as bug testing and feature development resources for the developers at iXsystems.

iXsystems generates revenue through hardware sales, and it is safe to say that they do not sell hardware that is not THOROUGHLY vetted and qualified to work with TrueNAS.

There is also a certain amount of free advertising for the product that results from the free appliance versions, much like T-shirts handed out at a trade show.

However, companies intent on finding a hyperconverged product to purchase are not likely to be basing their choice on whether they were able to get a green-colored VMware T-shirt at the trade show. I got a light blue one. Wasn't my first choice. Doesn't affect my buying decisions. Similarly, anyone who is basing buying decisions for expensive hyperconverged storage costing tens or hundreds of thousands of dollars isn't going to be obsessing over the kernel version of the free SCALE product. They're going to be comparing the feature sets of Nutanix vs VxRail vs VMware vs Proxmox vs TrueNAS.

Heaven forbid, they're not GIVING AWAY TRUENAS with a NEW ENOUGH version of the kernel, it's going to scare everyone off.

Mmm. Yup.

 

[Arwen]

@Arwen 5.10 is too old to effectively run current gen Intel CPUs (i.e Alder lake) which is what a lot of folks will be asking for to be supported. Also, people need to be able to install urgent kernel security updates without waiting for Truenas releases. In practical terms, the decision to not delay the SCALE release for 5.15 and to not suggest straightforward means to upgrade to a more recent kernel will force folks to look elsewhere. Release mgt always involves a lot of tradeoffs, but this seems like an obious poor business decision to me.

Yes, 5.10 maybe too old to run the newest Intel CPUs. (Or for that matter, newer AMD CPUs! Or ARM CPUs too!).

However, their is a reason LTS kernels get a third set of numbers. Those can be security updates, or new hardware drivers. I checked my Linux distro, and it's up to 100 sub-sub-releases, as in 5.10.100. (The most recent per "kernel.org" for 5.10.) Exactly what was added or fixed, I generally don't care too much for my home systems. But, here is the proof that at least one Linux distro has security fixes to the older 5.10 kernel;

Gentoo Linux - kernel 5.10.100 source changes

I do agree when installing newer hardware, it's generally best to install later Linux kernels.

However, your suggesting that iXsystems fix a LONG standing Linux kernel problem, by releasing more recent kernels goes against what Linux distros that charge MONEY do. RedHat certainly does not release the latest. RHEL 6 uses 2.6 kernels, RHEL 7 uses 3.10, (if I remember correctly), and RHEL 8 is still stuck on 4.x kernels.

I am not trying to be argumentative, but this Linux kernel issue will exist on every single distro. It's just the way Linux is designed, fast paced and moving target. Having a vendor, (or distro), create a weekly kernel update and verify it works reasonable well, is a bit too much unless you are paying them. RedHat certainly does not release weekly kernel updates, (for paid Enterprise Linux... Fedora or CentOS are totally different).