How to add new scrapers to prometheus from TrueCharts?

[darcos]

Hi,

I'm on Bluefin and have installed prometheus from truecharts.

I can see that it automatically scrapes TrueNAS out of the box and I can see the metrics in grafana work fine.

But now I wanted to add other scrapers so prometheus can grab that data from servers I have running on the internet but I can't figure out how to add scrapers.

I'm probably missing something obvious. Can some kind soul point me in the right direction?

Thanks

 

[mikebenna]

I'm stuck at exactly the same place. I've found /etc/prometheus/config_out/prometheus.env.yaml by poking around in the shell, but from its timestamp I'm guessing it's dynamically generated. I was expecting to be able to mount a config yaml somewhere and have Prometheus pick it up, but I can't find any docs pointing to how to do that. I look forward to further suggestions from the community!

Enabling file based service discovery would be a bonus. :)

 

[dee2me]

The truescale prometheus is not able to scrape custom sources.
I'm also trying to find a good time series db. Nothing yet.
Influxdb would be great but the way to deploy that to scale atm is ridiculous.

 

[MadKillerChicken]

The truescale prometheus is not able to scrape custom sources.

I don't get it, this seems to defeat the whole purpose of log aggregation.
Where did you read about this?
I sure hope there are plans to change this in the future.

 

[dee2me]

Reddit - Dive into anything

www.reddit.com

Tbh, I just stopped with the whole scale app idea for the time being.
It's much easier to use portainer or a linux vm.

 

[MadKillerChicken]

Reddit - Dive into anything

www.reddit.com

Tbh, I just stopped with the whole scale app idea for the time being.
It's much easier to use portainer or a linux vm.

Ah, yes, thank you for the link. I kinda figured something along those lines was the reason.

 

[mobrien118]

I spent days, maybe weeks, trying to solve this same "problem". You can skip to the last post for all you need to know:

TrueCharts Logging Server - Loki + syslog-ng + Grafana?

Hello, Pre-warning - this is a WIP that eventually could be a howto if I figure it out. I'm asking for community support to solve the problem and have already done quite a bit of research. I'll post my current challenges as sub-posts and update the main post as I move along. I'm having an...

www.truenas.com

 

[requird]

Installed Prometheus app. Got stuck. Read this post. Uninstalled Prometheus.

 

[vampirebyte]

One other way to integrate with the truecharts prometheus is using the prometheus-operator (also from truecharts) and deploy a podmonitor manually (or with fluxcd). This way, no prometheus config change is needed but the metrics magically show up and can be shown in grafana too. Works great for us.

 

[benkrejci]

I eventually figured this out for my setup; posting here in case anyone doesn't want to go down the whole Kubernetes/TrueCharts rabbit hole:

Stack:
TrueNAS Scale (23.10.1.3)
TrueCharts App Catalog
TrueCharts Apps:
- prometheus-operator (4.5.6)
- prometheus (16.4.3)
- grafana (10.3.1)

Problem: The problem I was trying to solve was getting hard drive temperatures to Grafana (the prometheus app worked great out of the box for a variety of other TrueNAS metrics via the included node-exporter, but not HDD temps).

Solution: I saw some vague posts like the above mentioning the prometheus operator and CRDs, but had no idea what this meant. Here's how I got SMART attributes to go to prometheus (explanation at the end):

1. Install a custom App from the prometheus-smartctl docker image (thanks to the maintainers of this repo, it works like a charm!)
   1. Click "Discover Apps" -> "Custom App"
   2. Enter "matusnovak/prometheus-smartctl" in the "Image repository" field
      
      
   3. Add port forwarding to 9902
      
      
   4. It's important that you check the "Priveleged Mode" box so that it has access to smartctl
      
      
      
      NOTE: Do this at your own risk! Presumably this effectively gives sudo to the container, so only proceed if you trust the contents of the docker image to have access to everything on your host!
2. SSH into your TrueNAS host
   1. Once the app has started, take note of the name and namespace of your new service so you can tell prometheus how to reach it:
      
      Code:

      sudo k3s kubectl get services -n <namespace>

      
      My namespace was "ix-prometheus-smartctl" (based on the application name) but you can also use `kubectl get services -A` to list all services if you don't know the namespace.
      
      
   2. Create a new yml file somewhere temporary (e.g. `vim /tmp/prom-smartctl-scrape-config.yml`
   3. Insert your namespace and name into the following and save:
      
      Code:

      apiVersion: monitoring.coreos.com/v1alpha1
      kind: ScrapeConfig
      metadata:
        name: static-config
        namespace: <your-namespace>
        labels:
          prometheus: system-monitoring-prometheus
      spec:
        staticConfigs:
          - labels:
              job: prometheus
            targets:
              - <prom-smartctl-app-name>.<prom-smartctl-app-namespace>:9902

      
      
      
   4. Run the following kubectl command:
      
      Code:

      sudo k3s kubectl apply -f <scrape-config-yml-location>

      
      
      
3. That's it! For me, prometheus-operator discovered the new CRD object within seconds and the new `smartprom_` metrics started showing up in Grafana.
   
   
   
   (I had the best success with the `smartprom_temperature_celsius_raw` metric

More info:
This is my best explanation of why this works, but somebody feel free to correct me:

• Kubernetes has a notion of a "CRD" (Custom Resource Definitions) which are just extra things you can you can store which are accessible via all their normal APIs.
• The prometheus-operator defines a CRD for scrapeconfig which other apps can post to in order to register additional configs. You can see the CRD and schema by running: `sudo k3s kubectl describe crd scrapeconfigs.monitoring.coreos.com`
• By adding a resource, prometheus-operator

Next steps:

• I am not sure if this will persist indefinitely (in fact I'm pretty sure it won't) so I may try to roll an app including prometheus-smartctl which registers its own scrapconfig CRD resource so it is linked to the lifecycle of the app.
• Would love pointers on this if anyone has experience contributing to TrueCharts (the docs are pretty vague).