Prometheus on SCALE?

SuperQ

Cadet
Joined
Mar 29, 2021
Messages
6
Hey all, I'm a Prometheus developer, and I've been following this project for a little while. I finally got some time to setup a test system.

I saw that the system currently uses collectd to manage metric data storage. I was wondering if there would be any interest in using Prometheus as part of the SCALE setup?

This would fit in nicely, as it's the most common monitoring system used on Kubernetes. All of the Kubernetes components implement Prometheus metrics endpoints, so it would be a natural fit for SCALE.
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,448
@SuperQ - That could indeed be interesting. Would Prometheus need to be loaded into the base system somehow, or can it be deployed containerized in some fashion?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
@SuperQ Good question indeed...

For TrueCharts, Prometheus support is planned for 21.06 (aka: I'm going to start working on it the comming month).
This would include metrics exports for quite a few of our apps :)

That being said: I totally forgot to document it for 21.06 >.<

@Kris Moore Prometheus works fine being containerised, it's used like that quite often ;-)

---

Edit:

Documented the current TrueCharts goals related to Prometheus and metrics in general:
 
Last edited:

SuperQ

Cadet
Joined
Mar 29, 2021
Messages
6
Containerized or not, either works perfectly well.

Prometheus itself will need a persistent volume somewhere, as it needs to store the time-series data (like the collectd RRD does).

The only thing that I don't recommend shipping as a container is the node_exporter. It's far easier to package as a normal systemd unit service.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
@SuperQ Agreed on the adding the ability to export node information to Prometheus.
You could either add it yourself in a PR or open a Jira suggestion for that ^^
 

SuperQ

Cadet
Joined
Mar 29, 2021
Messages
6
Sure, I can open a PR. What would be the right place to open one?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
truenas/webui is for the webui (if it requires any form of user interaction to add, thats where you can add the UI element)
truenas/middleware is for the middleware and configuration (so this repo includes some setup scripting)
truenas/scale-build contains the install/update media (including where you can define packages to install during setup)
 

SuperQ

Cadet
Joined
Mar 29, 2021
Messages
6
Thanks. I've submitted the first PR to scale-build to add the package. I'll look into the middleware/webui next to see how difficult it will be to have it update the defaults file and trigger service restarts.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Thanks. I've submitted the first PR to scale-build to add the package. I'll look into the middleware/webui next to see how difficult it will be to have it update the defaults file and trigger service restarts.
Awesome to see people put their time where their mouth is! :D
I've put some review feedback in for you :)
 

ronnok

Cadet
Joined
Nov 24, 2021
Messages
1
I see a PR go merged into scale-build. But I'm not seeing anything for the UI or any documentation on how to use this functionality. Any references would be appreciated.
 

stavros-k

Patron
Joined
Dec 26, 2020
Messages
231
I see a PR go merged into scale-build. But I'm not seeing anything for the UI or any documentation on how to use this functionality. Any references would be appreciated.
The only PR I can find in scale-build is `Closed` and not `Merged`. Also there is info inside for the reason it did close and not merged :)

If there is still another PR that I'm missing I'd like to see it!
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
@SuperQ We just released our first effort at a Prometheus App for TrueNAS SCALE.

Node Exporter will be added next as well :smile:
 

namnnumbr

Dabbler
Joined
Jan 19, 2021
Messages
14
If I just wanted to use node-exporter to monitor my SCALE box using an existing Prometheus instance, how would I do that?
 

namnnumbr

Dabbler
Joined
Jan 19, 2021
Messages
14
It's included by default on our App.
this is extraordinarily unhelpful. it is not evident to me which app you're discussing (prometheus or node-exporter or something else), which app provider (truenas vs truecharts) or provide any instruction on how to use said unspecified app specifically for node-exporter functionality.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
this is extraordinarily unhelpful. it is not evident to me which app you're discussing (prometheus or node-exporter or something else), which app provider (truenas vs truecharts) or provide any instruction on how to use said unspecified app specifically for node-exporter functionality.

The app would appear to be Prometheus, based on the title of this thread.

The app provider would seem likely to be TrueCharts based on the fact that the comment you're objecting to was posted by truecharts.

Posts #12 and #13 by truecharts indicate that Node-exporter has been integrated.

I don't think it is fair to call this "extraordinarily unhelpful."


In light of subsequent messages, I withdraw my comment.
 
Last edited:

namnnumbr

Dabbler
Joined
Jan 19, 2021
Messages
14
Perhaps "extraordinarily unhelpful" is a bit too far. I am frustrated at the lack of detail or direction. Saying "we merged node-exporter [into prometheus app]" does not particularly answer "how can I run just node exporter". Further, I have done the following:

If I load the truecharts prometheus app, it shows I can deploy Alertmanager and/or Prometheus but does not mention node-exporter. @truecharts posts #13 and #16 do not detail how to use node-exporter with or without the prometheus deployment.

Reviewing the truecharts docs:
There is no mention of "node-exporter" in the prometheus documentation: https://truecharts.org/charts/stable/prometheus/
Searching docs for "node-exporter" returns two container security warnings, but clicking either of them leads to an empty page (ex: https://truecharts.org/charts/stabl...tnaminode-exporter150-debian-11-r0-debian-115)
The changelog shows many chores but no mention of the merge: https://truecharts.org/charts/stable/prometheus/CHANGELOG/

Finally (and this is not necessarily related to the question of how to deploy node-exporter): the prometheus documentation in truecharts is sparse. Prometheus (and Alertmanager) have a ton of fiddly configuration, and it would be nice to have some documentation regarding configuration (if any is possible through the truecharts deployment).
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
Perhaps "extraordinarily unhelpful" is a bit too far. I am frustrated at the lack of detail or direction. Saying "we merged node-exporter [into prometheus app]" does not particularly answer "how can I run just node exporter". Further, I have done the following:

If I load the truecharts prometheus app, it shows I can deploy Alertmanager and/or Prometheus but does not mention node-exporter. @truecharts posts #13 and #16 do not detail how to use node-exporter with or without the prometheus deployment.

Reviewing the truecharts docs:
There is no mention of "node-exporter" in the prometheus documentation: https://truecharts.org/charts/stable/prometheus/
Searching docs for "node-exporter" returns two container security warnings, but clicking either of them leads to an empty page (ex: https://truecharts.org/charts/stabl...tnaminode-exporter150-debian-11-r0-debian-115)
The changelog shows many chores but no mention of the merge: https://truecharts.org/charts/stable/prometheus/CHANGELOG/

Finally (and this is not necessarily related to the question of how to deploy node-exporter): the prometheus documentation in truecharts is sparse. Prometheus (and Alertmanager) have a ton of fiddly configuration, and it would be nice to have some documentation regarding configuration (if any is possible through the truecharts deployment).

First off: We don't offer support here, we gave a quick and short answer as a courtesy.

As stated before node-exporter is included in our Prometheus App.
That's it, there is no configuration options or separate app for it.

If you want to discuss our Apps, improve them or update the docs, I suggest you discuss it with our own community on Discord instead.
 
Top