TrueNAS, Prometheus, and app/pod metrics

[Perry The Cynic]

I've installed the included Prometheus app. It works great. Thanks. The one from the built-in catalog, that is.

Now I want k8s metrics for the apps running on the box. Specifically, how much RAM and CPU each of them is actually using at a point in time. This should be easy, right? Yet I can't figure it out.

Using the standard Prometheus kubernetes_sd_configs clauses does not seem to work. It finds nothing (no nodes, no pods). I suspect that's because Prometheus itself runs with the default role account, which of course has no cluster-wide access rights. The Prometheus configuration screens have no option (that I can see) to change that.

I can't find per-app RAM or CPU metrics in the Graphite metrics (via graphite-exporter) either. Am I missing them (where?)

Does anyone have a working solution, preferably without hacking up the system in ways its maintainers don't approve of? It seems odd that Prometheus can't monitor where app memory is being used over time...

Cheers
-- perry

 

[Perry The Cynic]

It turns out that the included Prometheus "app" does not even use the default service account at all, and thus has no ready access to the cluster. I ended up running Prometheus as a custom app instead. (Custom apps do have role account credential access by default...)

Is there any point in providing feedback on how silly this is? And if so, where to?

Cheers
-- perry