TrueCharts high CPU utilization when cataloging

Bahama Tom

Dabbler
Joined
Feb 24, 2022
Messages
14
I have noticed with TrueNas Scale 22.02.0 that when TrueCharts is cataloging (I assume once a day, I am new to TureNas), the CPU usage maxes out at 100% for a few moments. Is anyone else experiencing this issue? If so what can be done to ensure utilization doesn't spike that high during the cataloging process?

My system:

Fujitsu MX130 S2 microserver
AMD FX-Series 6 cores 6 threads 3.3GHz
16 Gigs of ECC RAM
2x 120Gig SSD's (OS)
2x 4Tb NAS HDD's (Storage)
1x nvme ssd (cache)
PNY NVIDIA Quadro P400 Professional Graphics Board
HP NC364T PCIe 4Pt Gigabit Server Adptr (using port 1)

Currently only running TrueNas Scale 22.02.0, Apps: Plex And TrueScale

Planning to run PiHole and also Home Assistance (Apps)

Thanks for any assistance rendered and or any suggestions, in advance.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
It's 100% normal for a PC to spike once in a while while it needs to process a few thousand files and folders.
 

Bahama Tom

Dabbler
Joined
Feb 24, 2022
Messages
14
It's 100% normal for a PC to spike once in a while while it needs to process a few thousand files and folders.
Thanks for the reply. Another question if I may, since it is only processing the catalog (from what I see in the task manager) is there a way to disable TrueCharts, from what I assume updating it's catalog, and turn it into a manual process?
 

Bahama Tom

Dabbler
Joined
Feb 24, 2022
Messages
14
I had to delete TrueCharts, as the resource usage was insane. I installed TrueNas scale on two other boxes I have laying around here, and all experience the same issue with TrueCharts. Either I have something not configured correctly (I am by far no TrueNas expert), or it is "normal" that an application brings an entire environment down to its knees that it is unusable. I truly am loving to TrueScale concept, but for some reason it is not working for me (like I mentioned it might be user config error).
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
I had to delete TrueCharts, as the resource usage was insane. I installed TrueNas scale on two other boxes I have laying around here, and all experience the same issue with TrueCharts. Either I have something not configured correctly (I am by far no TrueNas expert), or it is "normal" that an application brings an entire environment down to its knees that it is unusable. I truly am loving to TrueScale concept, but for some reason it is not working for me (like I mentioned it might be user config error).

To be clear:
TrueChart is NOT an application, it's an catalog.
It's just a list of Apps to install. Loading and updating all those files takes a few minutes and some CPU usage, but after that it does not use any additional CPU.

you also seem to be thinking it's something that's to do with our project, however: We have no influence on how iX/SCALE loads the catalogs, we just provide them.

Generally speaking, however, 5 minutes is really the max time it takes to (re)load a catalog and yes, that might take some CPU.
 

LarsR

Guru
Joined
Oct 23, 2020
Messages
719
Just for comparisson, my ryzen 3700x spikes to 100% on 4 threads while syncing catalogues, but only for 1-2 Minutes till it's finished. After that it goes back to idle.
 

Bahama Tom

Dabbler
Joined
Feb 24, 2022
Messages
14
To be clear:
TrueChart is NOT an application, it's an catalog.
It's just a list of Apps to install. Loading and updating all those files takes a few minutes and some CPU usage, but after that it does not use any additional CPU.

you also seem to be thinking it's something that's to do with our project, however: We have no influence on how iX/SCALE loads the catalogs, we just provide them.

Generally speaking, however, 5 minutes is really the max time it takes to (re)load a catalog and yes, that might take some CPU.
truecharts,

I was not trying to think anything or bad mouth truecharts. I did mention in my original post I am new to truenas and truecharts, so if I get the verbiage incorrect, there is no need to assume I am thinking in anyway or trying to imply anything.

I also thank you for the clarification, and I truly would love to use the truecharts feature, due to its ease of use. Personally I think it is awesome.

But to my original question if there is anything that could be done that during the catalog process it does not tax the CPU to 100%, such as disabling it instead of having to uninstall it? I would think that is a fair question. And if the answer is no, I can accept it.

The only reason I even started this thread is the fact when truechart is not a part of the catalog, and I add docker images manually, I do not experience no where close to CPU spikes, as when I have the truecharts catalog added. And yes I fully understand that truecharts is a massive catalog and that you have no influence how iX/SCALE loads the catalogs. But my point here is the current catalog (if it is called that) remains quite small compared to loading the entire turescale catalog.

I am just throwing an idea out there and maybe it can be addressed that iX/SCALE actually takes a look at how catalogs are processed and to limit how many resources are used to do so.

Another idea I don't know how feasible it is and or if it already exists, that truecharts has various catalogs with less files and folders. For example: a media only category; or a network only category, that list only the most popular apps.

But again what do I know.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
But to my original question if there is anything that could be done that during the catalog process it does not tax the CPU to 100%, such as disabling it instead of having to uninstall it? I would think that is a fair question. And if the answer is no, I can accept it.

None. It's basically what defines SCALE Apps and only runs infrequently anyway.

iX actually increased resource consumption to ensure catalogs are update faster and there are no users till now complaining about it as 100% CPU load is quote normal under a multitude of cases. CPU spikes should not significantly slow down your system and are pretty normal.

If the CPU spikes cause issues, you might want to look at what you're trying to do. It might be the fact that you're running a relatively weak CPU combined with too many things you're trying to do on a system.


Another idea I don't know how feasible it is and or if it already exists, that truecharts has various catalogs with less files and folders. For example: a media only category; or a network only category, that list only the most popular apps.

We have no plans to split up our catalog into multiple catalogs, besides the splitting some of our Apps up into the current trains.
There is no gain for our users.

If you have issues with the way iX designed the catalog updates, you need to resolve it with them. We are not going to go through the drama of requiring all our users to reinstall due to splitting the catalog for something that is not of primary importance to our project.
 

soupbowl

Cadet
Joined
Apr 24, 2022
Messages
1
Hello,

It is weird to me that Truecharts is taking this so personal. It is not normal for apps or a 'catalogue' to spike your server to 100% cpu. 75% sure, but not 100% maxing out all the fans. I have a Intel(R) Xeon(R) CPU E5-1620 v3 4 core 8 thread cpu, it makes no sense that my web GUI comes to a crawl when a catalogue is updated.

We'll have to send the word out to iX systems that their catalogue system needs some attention.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
Hello,

It is weird to me that Truecharts is taking this so personal. It is not normal for apps or a 'catalogue' to spike your server to 100% cpu. 75% sure, but not 100% maxing out all the fans. I have a Intel(R) Xeon(R) CPU E5-1620 v3 4 core 8 thread cpu, it makes no sense that my web GUI comes to a crawl when a catalogue is updated.

We'll have to send the word out to iX systems that their catalogue system needs some attention.

We're not taking this personal, we just want to be clear it's something that's not (directly) related to our project or within our sphere of influence.
We also have not been able to repeat this behavior, primarily the consequences of slowing down other systems, within our team as of today.

Though the GUI does come to a crawl and require a hard-refresh once in a while, this weird buggy behavior is not related to catalog syncing.
 

Fr33dan

Cadet
Joined
Apr 21, 2022
Messages
5
I tried to use Truecharts a few days ago at a suggestion in another thread. I added the catalog and had to remove it the next day because it was making my server unusable with the GUI coming to a crawl.

I found another feature I wanted so I decided to see if there was a fix so I could add it back and found this thread.
We have no plans to split up our catalog into multiple catalogs, besides the splitting some of our Apps up into the current trains.
There is no gain for our users.
Yes it would, it would allow us to use your apps without locking up the server periodically. I don't need 130+ apps, I need maybe one or two that are specifically to support my local LAN. It seems clear to me that the catalog is much larger than TrueNAS was tested for and the cataloging code is just not optimized enough to handle it on many configurations.

If your catalog were broken down into groups of similar size to the one built into TrueNAS SCALE then I would image it would work fine since that catalog gives me no problems.

The fact is I (and the others in this thread) simply can't use it as it is. I think the proper "solution" would be for iX to fix the catalog system to use less resources but doing this would also solve our problem.

You jump in on nearly every thread on this forum with an attitude of "we fix everything broken with TrueNAS :smile:", so it just seems very weird that for this one thing you've taken a "not my problem" approach. It's such a stark contrast from your normal helpful attitude it makes it seem like you are taking this personally.
Though the GUI does come to a crawl and require a hard-refresh once in a while, this weird buggy behavior is not related to catalog syncing.
I have only been running my server for a month now (and never close my UI tabs because I have a problem), but my GUI was always snappy until I added the TrueCharts catalog so if you are experiencing this, it may actually be related to the added catalog after all.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
I tried to use Truecharts a few days ago at a suggestion in another thread. I added the catalog and had to remove it the next day because it was making my server unusable with the GUI coming to a crawl.


I have only been running my server for a month now (and never close my UI tabs because I have a problem), but my GUI was always snappy until I added the TrueCharts catalog so if you are experiencing this, it may actually be related to the added catalog after all.


The GUI issues are not related so something we build (there is no influence we have there) and not synced with catalog updates (we know because we manually run those thousands of times during testing). It seems to be pretty-much random and also not linked to CPU spikes either.

If your catalog were broken down into groups of similar size to the one built into TrueNAS SCALE then I would image it would work fine since that catalog gives me no problems.

Thats an assumption, considering it's mostly CPU onrelated GUI slowdowns, it could just as well be the size of the gui's our Apps build/use/create and not the size of the list.

The fact is I (and the others in this thread) simply can't use it as it is. I think the proper "solution" would be for iX to fix the catalog system to use less resources but doing this would also solve our problem.

And others can without issues on this front. So that brings up what is actually the problem, because our staff cannot reproduce it to the degree of the complaints here.

To be clear: we're not actively working on fixing this, as we didn't even recieve a single bugreport for it. (we REALLY need actual bugreports to be filed to us directly, as we are not based on this forums byond a minor core-team presence).

Nor are we aware of a quality bugreport towards iX Systems either, which is required for them to pick this up (the dev time does not work from this forum, generally speaking)

You jump in on nearly every thread on this forum with an attitude of "we fix everything broken with TrueNAS :smile:", so it just seems very weird that for this one thing you've taken a "not my problem" approach. It's such a stark contrast from your normal helpful attitude it makes it seem like you are taking this personally.

To be clear: That's not our goal, besides the docker-compose side-gig we did, it just happens to be the case because iX didn't implement many quite-vital features of kubernetes. As kubernetes compatibility/synergy is our goal, it just "happens" to fix SCALE issues.

That being said and ontopic to your actual question:
At the very minimal your request is not something we can do easily, it means demanding ALL our users to somehow backup, migrate and reinstall their Apps.

This is assuming just splitting things into trains solves your issues, which is not guaranteed to be the case.

Which means splitting into multiple catalogs, which would mean completely rewrithing our project, CI and so forth.
It means at least a few hunderd dev hours and "screwing over" all our loyal users.
We're simply not willing to go that far for this issue, which we still have not reproduced anyway nor is technically our responsibility.

You make it sound like it's easy to do something like this, while we barely can keep up with pushing the project forward with our current staff-power, without total rewrites of the whole project.

To be clear: It's not that we have a vast majority of our users requesting this either. You guys are, luckily enough, a non-reproducable minority at this time.

Building a project is making choices, when we can pick things up we try to pick things up. Even when it's not technically 100% our responsibility. But we do this for free with minimal manpower available and we cannot pick-up everything that someone requests.

Also: the only one capable of doing this is Ornias, who is already pre-occupied with more-important reworks till at least 2023.
 

Fr33dan

Cadet
Joined
Apr 21, 2022
Messages
5
The GUI issues are not related so something we build (there is no influence we have there) and not synced with catalog updates (we know because we manually run those thousands of times during testing). It seems to be pretty-much random and also not linked to CPU spikes either.

Thats an assumption, considering it's mostly CPU onrelated GUI slowdowns, it could just as well be the size of the gui's our Apps build/use/create and not the size of the list.
That is true I did assume they were related since they both started for me when I added the catalog, but thinking longer they are probably separate issues since the UI would lock even when cataloging wasn't happening.

It makes sense but hadn't considered that each app will have a configuration panel, the same way that the Plex app (the only one I'm using from the built in app catalog) does and that these could be impacting performance. I never actually had any apps from TrueCharts installed, so if this understanding is incorrect than I'm not sure what the issue could be.
And others can without issues on this front. So that brings up what is actually the problem, because our staff cannot reproduce it to the degree of the complaints here.

To be clear: we're not actively working on fixing this, as we didn't even recieve a single bugreport for it. (we REALLY need actual bugreports to be filed to us directly, as we are not based on this forums byond a minor core-team presence).
Nor do I think you necessarily should, but if I were on your team I would be keeping a close eye on this with a stance of "not working on this or plan to at this time". It's my experience that for every user that file a bug report or went to a forum for help, there are at least 5 more who silently gave up. TrueNAS SCALE has only been out of pre-release status for a couple of months, and I don't know what the install base is like but from looking at this forum 4 people having the same/similar issues within such a short timeframe seems like a lot. I'm guessing this is only going to get bigger.

And I wouldn't expect to see a bug report when this thread exists. I don't know anyone who would file a report without searching about the problem first leading to this thread which in turn discourages filing a bug report with y'all.
Nor are we aware of a quality bugreport towards iX Systems either, which is required for them to pick this up (the dev time does not work from this forum, generally speaking)
This is actually frustrating to learn as the UI dashboard would seem to indicate this is the place to go to get offical support (although reading the wording very carefully they do not actually say from them) and until told/100% sure otherwise I operate as though it's user error and not a bug worth filing a report for. I'm reaching that level of certainty now though and may try adding the catalog again to do a proper report with them.
To be clear: That's not our goal, besides the docker-compose side-gig we did, it just happens to be the case because iX didn't implement many quite-vital features of kubernetes. As kubernetes compatibility/synergy is our goal, it just "happens" to fix SCALE issues.

That being said and ontopic to your actual question:
At the very minimal your request is not something we can do easily, it means demanding ALL our users to somehow backup, migrate and reinstall their Apps.

This is assuming just splitting things into trains solves your issues, which is not guaranteed to be the case.

Which means splitting into multiple catalogs, which would mean completely rewrithing our project, CI and so forth.
It means at least a few hunderd dev hours and "screwing over" all our loyal users.
We're simply not willing to go that far for this issue, which we still have not reproduced anyway nor is technically our responsibility.

You make it sound like it's easy to do something like this, while we barely can keep up with pushing the project forward with our current staff-power, without total rewrites of the whole project.

To be clear: It's not that we have a vast majority of our users requesting this either. You guys are, luckily enough, a non-reproducable minority at this time.
I understand and agree that removing and replacing the existing catalog is likely a non-starter due to the existing userbase. It maybe could have been justified when TrueNAS SCALE got an official release, but since this post was made two days after you would have had to been able to see at least 6 months into the future for that.

I do think there is a possible solution with trimmed down git branches that are kept up to date with the main full catalog by bots. Can't deny the dev hours part though (although the S on hundreds could be debated since I bet you could do it in slightly under 199 dev hours).

Although I'm pretty confident a smaller catalog would have less issues, that should defintely be tested before anyone thinks about trying to do anything. I wonder if I could fork y'alls repos and hack together a version with less apps just for the sake of testing.
Building a project is making choices, when we can pick things up we try to pick things up. Even when it's not technically 100% our responsibility. But we do this for free with minimal manpower available and we cannot pick-up everything that someone requests.

Also: the only one capable of doing this is Ornias, who is already pre-occupied with more-important reworks till at least 2023.
And as it sits now I genuinely don't think you should divert resources from your current plan, I just wouldn't be so dismissive of the idea that you could attempt something later. Just keep an open eye/mind on it.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
That is true I did assume they were related since they both started for me when I added the catalog, but thinking longer they are probably separate issues since the UI would lock even when cataloging wasn't happening.

It makes sense but hadn't considered that each app will have a configuration panel, the same way that the Plex app (the only one I'm using from the built in app catalog) does and that these could be impacting performance. I never actually had any apps from TrueCharts installed, so if this understanding is incorrect than I'm not sure what the issue could be.

Thats the hard part: Figuring out what precisely is triggering something...
Be sure to send as much information as you can about this to iX by filing a Jira ticket and attaching a debug :)

Nor do I think you necessarily should, but if I were on your team I would be keeping a close eye on this with a stance of "not working on this or plan to at this time". It's my experience that for every user that file a bug report or went to a forum for help, there are at least 5 more who silently gave up. TrueNAS SCALE has only been out of pre-release status for a couple of months, and I don't know what the install base is like but from looking at this forum 4 people having the same/similar issues within such a short timeframe seems like a lot. I'm guessing this is only going to get bigger.

And I wouldn't expect to see a bug report when this thread exists. I don't know anyone who would file a report without searching about the problem first leading to this thread which in turn discourages filing a bug report with y'all.

To be clear: we only accept bugreports and offer assistence using our own channels.
Even more so because we would/might need to go into back-and-forth for btoh support and bugtracing purposes and we do not have the staff available to do so on 99 random forums/reddits/youtube/discords.

Not reporting it properly is "not getting fixed" most of the time, the same goes from iX Systems who also generally don't fix bugs that are only reported on the forums. That's even if the right person our team finds the report.

We do not have the manpower to turn each reported issue into bugreports ourselves or keep track of everything reported everywhere.


This is actually frustrating to learn as the UI dashboard would seem to indicate this is the place to go to get offical support (although reading the wording very carefully they do not actually say from them) and until told/100% sure otherwise I operate as though it's user error and not a bug worth filing a report for. I'm reaching that level of certainty now though and may try adding the catalog again to do a proper report with them.

Generally speaking our experience with iX is that they rather have over-reporting of bugs than underreporting :)

I understand and agree that removing and replacing the existing catalog is likely a non-starter due to the existing userbase. It maybe could have been justified when TrueNAS SCALE got an official release, but since this post was made two days after you would have had to been able to see at least 6 months into the future for that.

To be clear:
Our project was started in the end of 2020, by the summer of 2021 we where already "mostly done" with the important/big bits. So even if we would've known about this minor bug (from our side it's minor, as it's not our bug technically speaking), we doubt we would've totally revamped our project at the end of 2021.


I do think there is a possible solution with trimmed down git branches that are kept up to date with the main full catalog by bots. Can't deny the dev hours part though (although the S on hundreds could be debated since I bet you could do it in slightly under 199 dev hours).

We can do a lot with bots, but we have very firmly decided not to rewrite the CI or project structure for this.
We're move important reworks ongoing, this is not a priority for us,

We have actually discussed this option and it was shot down by the core-team already. It's not going to happen, it's not how we want to structure our project, at least not before we see how iX-Systems is going to implement catagories.

Once we know we might(!) split into more trains (not likely in relevant maner though, we do not plan to have 10+ trains), depending on how the catagory things worked out. But we're quite firm that we're not going to do catalog splitting any time soon.

Sorry, but you really need to solve this bug upstream.

Although I'm pretty confident a smaller catalog would have less issues, that should defintely be tested before anyone thinks about trying to do anything. I wonder if I could fork y'alls repos and hack together a version with less apps just for the sake of testing.

And as it sits now I genuinely don't think you should divert resources from your current plan, I just wouldn't be so dismissive of the idea that you could attempt something later. Just keep an open eye/mind on it.

We've voted that out, it's not going to happen.
Sorry, but we're not going to get into 'murican PR. As long as there is no one voting in favor within our core team, it's just not happening. Period.

Sure things change, things change all the time, but we're not going to act like we're keeping an open mind, while we don't have any plans to work on this anytime in the next 5 years.

Please remember:
You're a minority, with a low-priority bug, that's not caused by us ourselves.
You really need to fix any performance issues you got with iX Systems.

If you need any help with any of our stuff, please reach out to our (support)staff directly (mostly on discord).
We're just here to nudge people in the right direction and give answers to where our projects is heading in the broadest of directions.
 

zeroZshadow

Cadet
Joined
May 1, 2022
Messages
1
I just wanted to add some info to help this along.
To start of the issue doesn't lie with truecharts, kinda.

Whenever the python middleware does the catalog.items step, it does a few quick steps, like syncing the git repository the charts are on.
This is all fine.

The issue is that it will then load the details of EACH version for EACH app in EACH train you have into cache.
It does so using this method which will load 5 files from disk, parses them, and store them into the "cache" for 25 hours.
Since this is done per version per app, this is 388 Charts. Loading AND PARSING 776 yaml files, and 1164 markdown files.
This is the bit that seems to be so costly.

So from how I see it, this isn't really Truecharts's fault. It just shows up for them as they have a significantly larger selection of apps and version.

Hopefully in the future TrueNAS only load the minimum required data into cache on updating the catalogs, and load the rest on use.
Especially the questions.yaml for each app version is absolutely massive, and is likely the heaviest bit here.
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
I just wanted to add some info to help this along.
To start of the issue doesn't lie with truecharts, kinda.

Whenever the python middleware does the catalog.items step, it does a few quick steps, like syncing the git repository the charts are on.
This is all fine.

The issue is that it will then load the details of EACH version for EACH app in EACH train you have into cache.
It does so using this method which will load 5 files from disk, parses them, and store them into the "cache" for 25 hours.
Since this is done per version per app, this is 388 Charts. Loading AND PARSING 776 yaml files, and 1164 markdown files.
This is the bit that seems to be so costly.

So from how I see it, this isn't really Truecharts's fault. It just shows up for them as they have a significantly larger selection of apps and version.

Hopefully in the future TrueNAS only load the minimum required data into cache on updating the catalogs, and load the rest on use.
Especially the questions.yaml for each app version is absolutely massive, and is likely the heaviest bit here.

This is indeed correct.
The behavior here has been created just before release because the parsing times of catalogs became extremely long.

However: unless someone files a Jira bugreport about this behavior, it's also not magically comming into iX' their agenda either :(
 

cccode

Cadet
Joined
Jul 4, 2022
Messages
2
I am facing the same problem.
When truenas refreshes the catalog of Truecharts,my server's CPU usage is at 100%, and its fans are spinning, and the web ui is unresponsive.
It was a really bad experience!
 

stavros-k

Patron
Joined
Dec 26, 2020
Messages
231
I am facing the same problem.
When truenas refreshes the catalog of Truecharts,my server's CPU usage is at 100%, and its fans are spinning, and the web ui is unresponsive.
It was a really bad experience!
Hey, read the exact previous post of yours. :)
Especially the last sentence :)
 

MountainMan

Dabbler
Joined
Dec 10, 2020
Messages
42
Are you both running the latest 22.02.2 ? It does have some changes to trim down resources when cataloging. Also there is an unrelated fix coming that may be robbing you of some CPU and responsiveness over time as well. But I agree, it would be nice to be able to limit large catalog runs to late night or something.
 

cccode

Cadet
Joined
Jul 4, 2022
Messages
2
Hey, read the exact previous post of yours. :)
Especially the last sentence :)
I don't understand what you mean. I'm new here and the above is my first reply. :)

Maybe it's better to say "spinning" as "flying". Anyway, what I want to say is 'cpu fan is making a lot of noise'.
Are you both running the latest 22.02.2 ? It does have some changes to trim down resources when cataloging. Also there is an unrelated fix coming that may be robbing you of some CPU and responsiveness over time as well. But I agree, it would be nice to be able to limit large catalog runs to late night or something.
Yes, a new installation, the latest version.
 
Top