I eventually figured this out for my setup; posting here in case anyone doesn't want to go down the whole Kubernetes/TrueCharts rabbit hole:
Stack:
TrueNAS Scale (23.10.1.3)
TrueCharts App Catalog
TrueCharts Apps:
- prometheus-operator (4.5.6)
- prometheus (16.4.3)
- grafana (10.3.1)
Problem: The problem I was trying to solve was getting hard drive temperatures to Grafana (the prometheus app worked great out of the box for a variety of other TrueNAS metrics via the included node-exporter, but not HDD temps).
Solution: I saw some vague posts like the above mentioning the prometheus operator and CRDs, but had no idea what this meant. Here's how I got SMART attributes to go to prometheus (explanation at the end):
- Install a custom App from the prometheus-smartctl docker image (thanks to the maintainers of this repo, it works like a charm!)
- Click "Discover Apps" -> "Custom App"
- Enter "matusnovak/prometheus-smartctl" in the "Image repository" field
- Add port forwarding to 9902
- It's important that you check the "Priveleged Mode" box so that it has access to smartctl
NOTE: Do this at your own risk! Presumably this effectively gives sudo to the container, so only proceed if you trust the contents of the docker image to have access to everything on your host!
- SSH into your TrueNAS host
- Once the app has started, take note of the name and namespace of your new service so you can tell prometheus how to reach it:
Code:
sudo k3s kubectl get services -n <namespace>
My namespace was "ix-prometheus-smartctl" (based on the application name) but you can also use `kubectl get services -A` to list all services if you don't know the namespace.
- Create a new yml file somewhere temporary (e.g. `vim /tmp/prom-smartctl-scrape-config.yml`
- Insert your namespace and name into the following and save:
Code:
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
name: static-config
namespace: <your-namespace>
labels:
prometheus: system-monitoring-prometheus
spec:
staticConfigs:
- labels:
job: prometheus
targets:
- <prom-smartctl-app-name>.<prom-smartctl-app-namespace>:9902
- Run the following kubectl command:
Code:
sudo k3s kubectl apply -f <scrape-config-yml-location>
- That's it! For me, prometheus-operator discovered the new CRD object within seconds and the new `smartprom_` metrics started showing up in Grafana.
(I had the best success with the `smartprom_temperature_celsius_raw` metric
More info:
This is my best explanation of why this works, but somebody feel free to correct me:
- Kubernetes has a notion of a "CRD" (Custom Resource Definitions) which are just extra things you can you can store which are accessible via all their normal APIs.
- The prometheus-operator defines a CRD for scrapeconfig which other apps can post to in order to register additional configs. You can see the CRD and schema by running: `sudo k3s kubectl describe crd scrapeconfigs.monitoring.coreos.com`
- By adding a resource, prometheus-operator
Next steps:
- I am not sure if this will persist indefinitely (in fact I'm pretty sure it won't) so I may try to roll an app including prometheus-smartctl which registers its own scrapconfig CRD resource so it is linked to the lifecycle of the app.
- Would love pointers on this if anyone has experience contributing to TrueCharts (the docs are pretty vague).