As a noob, I am a bit concerned about migrating to Scale, can someone calm my nerves?

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
I'm in no way saying that everyone would experience it. But this is absolutely a problem specific to SCALE, because k3s process does not exist in CORE. YOU personally may not experience it, but I can point to a bunch of threads of people experiencing this problem, even on the project's own GitHub (and that is just one of many threads I saw there), which they refuse to fix. This person even wrote a blog post about it.

I'm happy for you that you don't experience it, but saying that your experience somehow means that this is not a problem is just factually not true and YMMV from person to person. On the contrary, CORE will NEVER suffer this problem because it does NOT (and probably never will) have k3s process.
On the other hand, merely posting that because you read some people having a specific problem does not mean that the problem is what you say it is either. It is indeed an issue or problem perhaps. In this very thread, someone who migrated to Scale from Core had a higher CPU usage, which you then immediately assumed was k3s (could have been, or may not have been). Maybe the migration has a bug, maybe if he did a new install it wouldn't. In that case, the actual prolblem is the migration right? I can cherry pick problems people are having on freeBSD and state that freeBSD has some issue too as a few users had some similar sounding problem. That wouldn't make it valid. I can find thousands of issues posted here and make broad claims. That is what I feel you are doing here. I have heard few people complain of that, the last time someone did they found out that it was one of their apps, just last week, the thread was titled something about high CPU usage. Problem was solved. High cpu usage can be all sort of things, you want to associate that with Scale and k3s. But yes, indeed, having one more layer is going to add something to resource usage, we all know that, and that will be worse on lower powered machines. Undoubtedly. But Windows has more resource usage than dos too. As things progress, more power is needed but it doesn't make it a problem per se.

The linked to blog post is light on details. The github post is about small (embedded) systems by the description, and many of the linked to cases are as well. Not very compelling except if your point is k3s will take more resources than no k3s, well, yes, that's true of any software. Stating things like high load average is conflating what load average is vs cpu percent, etc. Note the issue is not reproduceable indicating it's something on the specific machine too. The github issue is full of Rapsberry Pi users (I use one) and that is not surprising at all. Yes, k3s undoubtedly uses > 0 resources and may be too much for a Pi. Note the very last post on github, the guy discovers it was a specific helm chart. It's hard to say what the actual problem is for all the other folks. The most compelling of your links was actually to a truenas user with lots of k3s processes. That was clearly a bug of some sort with Scale.

In any case, what this is about is someone is having some sort of problem with cpu usage. What that is is unknown except for the solved cases on github that claimed this and turned out to be something else. It could be k3s, but then you would expect far more posts about that in my view here on the truenas forums which I've seen very few of (yes, more than 0). And on github, people will chime in and say they have the "same problem" when it is totally different, this is not surprising or unusual. High CPU by itself means nothing as far as determining why.

Yes, I could also find a process on Core/freeBSD that does not exist on Scale and make some claim about how that problem would never appear on Scale, but that's stretching things. I used to use freeBSD and found it quite good actually.

I realize there is a lot of hatred (or bias against) here for/against Scale/Linux and I don't want to discuss that but many posts in these forums have that. Core is a great product and undoubtedly it's still more reliable, I believe we can agree on that. That will change over time. When you say things like "this is why I will never install Scale", Scale is Beta, etc. that kind of indicates where comments are coming from. When I read things like that, my first natural impulse is to think I will reduce the value of the entire posters comment due to bias. That's the natural inclination, but then I realize I shouldn't really.

Really to me, Scale is more complex to use than Core. I don't mind that as a retired It guy as tech is trivial for me, and thus my Scale machine runs perfectly with no issues really. But so many home users who have limited docker / Linux knowledge is definitely going to create a lot of posts with various issues. That will take a while yet to prevent.

I'm glad you are here volunteering your time, you have a lot of good posts and knowledge to bring to the community in general. I just see this differently but appreciate your view. I just wish there was less of Core vs Scale in these forums. And that's what I saw your comments as. I could well be wrong about that. But pretty sure I'll get a lot of blowback on this post!
 

William Bravin

Contributor
Joined
Mar 16, 2016
Messages
195
Hello all

I truly tip my hat to all who contribute to this discussion

I'm not an IT guy, although i did manage very large Business intelligence and DW implementation projects.

I'm like the person who goes to the flour mill and comes out all dusted in flour. I understand how flour is made but i can't make it.

Therefore i'm willing and happy to learn how to make it

I truly loved scale gui design. Can this design since developed be ported to core?

Are the upgraded functionalities of scale be ported to core?

Will there be a painless migration path for individuals like myself to benefit from these updates?

Should it be?

why have 2 solutions. This is more expensive, time and resources consuming and yes will increase post like this thread ad infinitum

Be all well
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
Are the upgraded functionalities of scale be ported to core?
That depends on which functionalities you mean:
  • ZFS improvements? Yes, that's the plan.
  • UI improvements? I hope so, but I don't recall hearing either way from iX
  • Clustering? AFAIK, there's no plan or intention to do this.
  • Apps? Same.
  • VMs? Not really possible for the most part.
why have 2 solutions. This is more expensive, time and resources consuming
...which is why many of us suspect that iX plans to drop CORE entirely. As yet they deny this.
 
Joined
Oct 22, 2019
Messages
3,641
UI improvements? I hope so, but I don't recall hearing either way from iX
SCALE's GUI has skewed (and improved) so much from what still exists on Core, that they would have to just yank the rug and replace it with another. Not just the design and layout, but the tools as well.


If not for "Apps" (i.e, k3s), and let's pretend the GUI layout and tools are otherwise exactly the same, why would someone migrate to SCALE?

Surely not because of better memory/ARC management. Surely not due to the underlying ZFS. Surely not because they totally love the difference in "ada0" vs "sda1" naming scheme.


To ask another way: Without invoking "Apps", how would you convince a Core user to migrate to SCALE? (If you say "improvements in the GUI and other fixed bugs", then you're implying that such improvements could not have been implemented in Core, or even a "Community Core Edition" to spare enterprise customers from "risky changes in code".)


So, it's "Apps". Isn't it? That's the big driver. And since that's the driver, all the work is being done on the SCALE platform, leaving Core to remain as something that inherits "FreeBSD updates, ZFS updates, and major security bugfixes". Nothing more.
 
Joined
Oct 22, 2019
Messages
3,641
ZFS improvements? Yes, that's the plan.
But to be pedantic, it's not that ZFS improvements are being backported to Core from SCALE. Upstream OpenZFS code is where they pull from. If SCALE didn't exist, Core would still receive the latest and greatest ZFS features. (Perhaps not exposed in the GUI. Or perhaps indeed exposed in the GUI, since their focus would be on Core, and Core alone. :wink:)
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
If not for "Apps" (i.e, k3s), and let's pretend the GUI layout and tools are otherwise exactly the same, why would someone migrate to SCALE?

Surely not because of better memory/ARC management. Surely not due to the underlying ZFS. Surely not because they totally love the difference in "ada0" vs "sda1" naming scheme.
You forgot: Surly not for k3s using up to 10% CPU at idle.
And surely not for the pleasure of having to fix GRUB whenever it loses its marbles because some drives were swapped.

In short: If one is not using apps/containers there's NO reason at all to migrate from CORE to SCALE.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
If not for "Apps" (i.e, k3s),
These are not equivalent. k3s is a system under which the apps run; it is not the same thing as the apps.
why would someone migrate to SCALE?
So you want to exclude the major (current) differentiator between the two products, and then ask why someone would choose one over the other? That's frankly a silly thing to ask with that constraint. But clustering is still out there, and kvm is a much better hypervisor than bhyve.
Without invoking "Apps", how would you convince a Core user to migrate to SCALE?
I wouldn't have the slightest interest in doing so. I'm not interested in playing the CORE vs. SCALE game.
So, it's "Apps". Isn't it? That's the big driver.
Why act like this is some big revelation? People want to run other software on their NASs, and they want to do it easily. That's why FreeNAS had plugins ten years ago. That's why CORE continues to have plugins, even though we all know they're going away. And they've always sucked. iX believes that docker containers and Helm charts under Linux will result in a much more sustainable ecosystem. Time will tell if they're right, but the hundreds of apps that TrueCharts have built suggest that they are.
But to be pedantic, it's not that ZFS improvements are being backported to Core from SCALE.
OK, they're being backported from Linux to FreeBSD, and much of that is happening upstream. Still doesn't change the fact that SCALE is getting them sooner.
not for k3s using up to 10% CPU at idle.
"up to 10% CPU", or as little as 0% CPU. Whoop-de-****, that process uses some CPU resources.
If one is not using apps/containers there's NO reason at all to migrate from CORE to SCALE.
Apps/containers, or VMs, or clustering.
 
Joined
Oct 22, 2019
Messages
3,641
These are not equivalent. k3s is a system under which the apps run; it is not the same thing as the apps.
"Apps" in double quotes, with a capital "A", and within the specific context of TrueNAS SCALE...

There's a reason I wrote it like that. :wink:
 

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
To me, the next time someone has the high cpu usage with nothing running, someone should ask the poster (if it cannot be determined otherwise) to please submit a bug report. For me, that's the constructive and helpful way to squash whatever the issue is (since obviously it does occur at times).

I seriously doubt IX Systems would approve of a modern system using 30% CPU and doing nothing useful. Right?

IMHO, if someone is running Core, and it works for them, there is no complelling reason to switch. Don't see any reason to encourage it either.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
"Apps" in double quotes, with a capital "A", and within the specific context of TrueNAS SCALE...
Tracking all that. And with all of that given, it's incorrect to say that they are k3s. They use k3s.
I seriously doubt IX Systems would approve of a modern system using 30% CPU and doing nothing useful. Right?
I wonder if the real complaint is that that's how it shows up in htop. And in that case, htop obscures the fact that the CPU% column refers to the percentage of a single core. But since multi-core processors have been the norm for, oh, about 15 years now, that percentage isn't especially relevant.
IMHO, if someone is running Core, and it works for them, there is no complelling reason to switch. Don't see any reason to encourage it either.
Agreed. Other than wanting the features of SCALE, the only reason I can see to switch from CORE would be concern that iX is going to kill CORE off--but even if that's the case, there's no rush.
 

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
I wonder if the real complaint is that that's how it shows up in htop. And in that case, htop obscures the fact that the CPU% column refers to the percentage of a single core. But since multi-core processors have been the norm for, oh, about 15 years now, that percentage isn't especially relevant.
Some of them could be, but who knows. I have not read any thread where someone suggests submitting a bug report, and, actually did. I'd love to see it squashed if there is a bug. I would hope anyone would.

Not only would I wish someone to submit a bug report, I'd love to hear the final answer. Instead of speculating. I don't actually care what the problem is, it should be fixed where there is one. But if it were anywhere remotely close to common, there would be way more posts here. Instead, we get I can't figure out how to passthrough my GPU, or, permissions issues, or what hardware should I buy issues, or, why does ZFS say my drive has failed.
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
Apps/containers, or VMs, or clustering.
CORE has VMs. I grant you "clustering"; but how relevant is it for home users? IMHO, if you're clustering servers you should be using TrueNAS Enterprise, and getting your support directly from iX rather than here.
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
CORE has VMs.
True, but KVM is a much more mature hypervisor than bhyve.
I grant you "clustering"; but how relevant is it for home users?
I find my Proxmox cluster quite useful at home. How well that'd translate to TrueNAS I don't know.
 

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
It's a cost of using k3s: https://github.com/k3s-io/k3s/issues/294

They're well aware of the constant CPU usage, but they justify it.
Incorrect, there are systems (especially Pis) that for whatever reason yet to be determined are having high CPU usage (though in Pis case are rather obvious), could be k3s, might not be the issue, but, that is not the same thing. Sorry, that's an incredible stretch. "They" are not aware and if you read the thread, said so. Not able to be reproduced. To take that with Raspberry Pis and change it to Scale has such issues does not make much sense. You can believe as you wish and frame as you want to but that is not reality. Again, read the last post:


That manifested itself with the supposed k3s issue, but it never was.

Next time just encourage someone to submit a bug report to IX. You KNOW they will not want a system with the purported overhead. Why mess around with preconceived notions and cherry picking bugs to make wild speculation when you could encourage a fix for any that do pop up? That makes little sense if you are trying to help the community (or a user) as a whole. Then it can be put to bed hopefully.

I submitted a bug with replication and system crashes on a non truenas target, and, IX was involved in fixing it even though it was not even Truenas. They can often get things fixed or at least make a better case for it.

Oh, and load is often mis-understood as well, not just cpu usage. A load of 6 on my 32 thread CPU is not exacty the same as a load of 6 on a Pi. And it's really not terribly meaningful in many cases it's used in. It doesn't equate to CPU, it could, but it's not a direct comparison at all.
 
Last edited:
Joined
Oct 22, 2019
Messages
3,641
"They" are not aware and if you read the thread, said so.

From the related bug reports, it was stated very clearly. They are aware of this in regards to k3s, but it's the cost of using such sophistication. (As in "not free" of CPU overhead.)
Brad Davidson said:
Running Kubernetes is not free. It is expected that it will take about 0.5 to 0.75 of a single core on an embedded system like a Raspberry Pi just as a baseline. If you don't want the overhead of constant health checking, reconciliation, and metrics collection you might look at something a little more lightweight like docker-compose.

I am going to close this issue because at this point it seems to just be attracting more folks who want to argue that Kubernetes should run with zero overhead on any arbitrary low-end or embedded system. Anyone who is curious why this isn't feasible can read the conversation above and click through to the offered links, with https://docs.k3s.io/reference/resource-profiling describing a good baseline.

* Emphasis added.


In other words, there’s nothing that iXsystems can really do about it for SCALE.
 
Last edited:

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
From the related bug reports, it was stated very clearly. They are aware of this in regards to k3s, but it's the cost of using such sophistication. (As in "not free" of CPU overhead.)


* Emphasis added.


In other words, nothing that iXsystems can really do about it for SCALE.
I of course am fully aware of that, but you are throwing out examples of usage across all cores, non Pi cores, with Truenas. Pi cores have extremely limited CPU, hardly the same contention. We are talking about Truenas here, not a Pi. So, I fail to see ANY relevance at all. .5 of a Pi core is virtually 0 of a Xeon core. And of course, as I stated earlier, running more layers is always > cpu than running less layers. Hardly a great revelation. I agree complettely with the comment in the guthub case, I would expect that also! .5 of a Pi core would bea way less than .05 of my Xeon core (likely way less but let's just say 10 to 1), and .05 of a Xeon core with 16 cores would be .05/16 = .003. Overhead? yes, but essentially 0.

It's a non issue for Scale barring some other interaction which would need to be discovered. Thank you for making my point.
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
In other words, there’s nothing that iXsystems can really do about it for SCALE.
What is the "it" in question? "k3s uses some CPU"? Of course it does; any running daemon is going to cost some CPU time, and no, iX isn't going to be able to change that fact. But that isn't what's being said. What's being claimed (and what I contend is FUD) is "k3s will use 10% of your CPU at idle." And that's just nonsensical, and ignores the enormous variation in hardware people use to run TrueNAS. The available CPU for a Pentium G860, for example, is quite a bit less than what's available with my dual Xeon Gold 6132s (which cost all of US$30 each--hardly an expensive CPU), and I'm certainly not using top-of-the-line gear.

If you want to say, "users with very limited CPU shouldn't use SCALE," you're making a reasonable point. But "k3s will use X% of your CPU at idle" (for any value of X, really) is both meaningless and FUD.
 
Joined
Oct 22, 2019
Messages
3,641
I went back to edit my post to remove the part where I said k3s will use 10% (or X%) of your CPU while idle, but couldn’t find where I said that. :wink:

Low-end or low power systems, of which there are NAS users, need to be aware of this for SCALE.
 
Last edited:

asap2go

Patron
Joined
Jun 11, 2023
Messages
228
I think there's probably a minimum amount of power needed to cover the overhead of any system. A low-power system that runs slower just takes longer to do the work, and therefore shows more CPU usage. The way around this is with a new system with dynamic clock speeds and instructions that make use of low- power states.

Personally, it seems SCALE on Linux is more robust though less stable and less of a power miser than CORE on BSD that's designed "tighter." With that said, rolling a tight Linux Gentoo system is the best of both, but a lot of work.

My server at idle probably burns more power than yours at full-tilt, BUT it's near impossible to get mine over 15% CPU usage and 10% additional power usage which is ideal for sustained heavy loads, not home use.
Can you give a concrete example of that?
Sounds near impossible to me unless you run an ancient version of OpenBSD.
 
Top