Mapping of FreeNAS data sent to graphite_exporter (Part of Prometheus)?

tobiasbp

Patron
Joined
Dec 2, 2015
Messages
238
Hello there...

I'm playing around with exporting data from FreeNAS to Prometheus via the graphite_exporter. It will hold the data for a while (5 minutes by default), and Prometheus will scrape the data periodically. The incoming data can be labeled for Prometheus using a mapping-config file. The syntax of the mapping-config is the same as used by ''statsd_exporter". It is described in more detail here: https://github.com/prometheus/statsd_exporter#metric-mapping-and-configuration
  • Has anyone here written a mapping-config file for the data from FreeNAS?
  • Is the data exported to Graphite by FreeNAS documented somewhere?
I'll share an iocage plugin/jail running graphite_exporter if there is any interest.

Thanks,
Tobias
 
Last edited:

tobiasbp

Patron
Joined
Dec 2, 2015
Messages
238
FreeNAS sends something like this:
Code:
servers.your_server_name.aggregation-cpu-average.cpu-idle

I came up with this mapping configuration:

Code:
mappings:
- match: "servers.*.*.*"
  name: "${3}"
  labels:
    instance: "${1}"
    job: "freenas"
    type: "${2}"
- match: "servers.*.*.*.*"
  name: "${3}_${4}"
  labels:
    instance: "${1}"
    job: "freenas"
    type: "${2}"


I'd still like to see some documentation of the data sent by FreeNAS. Is it documented?
 

cloudycelt

Dabbler
Joined
Dec 30, 2019
Messages
16
Hey @tobiasbp I know it's been just over a year to the day but did you happen to keep up with this? Did you get all the data you wanted out?

I'm currently trying to get some metrics into Prometheus, ideally I'd like to not have to run multiple metric stores (Influx, Graphite, Prometheus) and just stick with Prometheus.

I got node_exporter running in a jail and that works great, even has some ZFS stats, but no information on datasets/volume usage which would help a lot.

I found https://github.com/pdf/zfs_exporter which is older and covers some of those stats, but have yet to try to get it to run in the jail.

This one looks promising too https://github.com/matusnovak/prometheus-zfs and https://github.com/Leidos-CSS-IRAD/ZFS-exporter
 

cloudycelt

Dabbler
Joined
Dec 30, 2019
Messages
16

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
I use the builtin collectd to push data from my FreeNAS to an InfluxDB which can receive data via a Graphite compatible interface. I visualise that in Grafana. I am not quite sure what your problem or precise question in this thread is.

Kind regards,
Patrick
 

cloudycelt

Dabbler
Joined
Dec 30, 2019
Messages
16
Hi @Patrick M. Hausen - thanks for reading!

That approach works if InfluxDB is your metrics database of choice already, but if either:
1- You are in a large standardised environment that has chosen Prometheus (and has expertise in that and not Influx)
or
2- You have a small environment like a home one and don't wish to waste limited resources running multiple metrics solutions

Then it might be desirable to export/send metrics directly to Prometheus, if you are already using it for all of your other metrics.

That said, Prometheus does have a collectd_exporter that you could install into a jail, the pkg name is prometheus-collectd-exporter-0.4.0_1 that I can see, it is some version of https://github.com/prometheus/collectd_exporter - it is supposed to function as a place you can send collectd metrics and it will convert them to Prometheus format.

I'm not sure what would need to be altered to get it functioning - I don't see /var/db/collectd in a jail, but it is available in the main system shell, so I'm guessing that would be a problem, unless there is some other way to have the main system send its collectd metrics to an endpoint I can define.

So far I've only seen the Graphite field at System-->Reporting for this which I have tried, setting the collectd_exporter address as the destination, but it seems to be trying to send over TCP, while the exporter runs on UDP so it's getting `connection refused` in the log.

I actually came across this too just now: https://collectd.org/wiki/index.php/Plugin:Write_Prometheus

This indicates that collectd has a plugin for outputting in Prometheus format already (exciting!) but on FreeNAS it's not enabled, and doesn't seem to be easy to do so:

In `/conf/base/etc/loca/collectd.conf` I see:

`##LoadPlugin write_prometheus` indicating it's possibly not even installed, or just explicitly disabled

and I see

```
#<Plugin write_prometheus>
# Port "9103"
#</Plugin>

```

I tried uncommenting both as well as setting `AutoLoadPlugin true` and restarting collectd but no luck, it doesn't seem to load it.

I also just found your post over at https://www.truenas.com/community/threads/collectd-settings.87659/

I think that if I could customise collectd to enable the `write_prometheus` plugin, that would potentially be sufficient, even if it didn't survive reboots/upgrades since those don't happen every day.

Did you have any luck customising collectd?
 

cloudycelt

Dabbler
Joined
Dec 30, 2019
Messages
16
As an additional set of points I've now found:

- Customising things in `/conf/base/etc/local/collectd.conf` seems to have little/no effect at least as far as `write_graphite` and `write_prometheus` are concerned. I may be missing a flag somewhere.
- Running just the regular Prometheus node_exporter binary straight from https://freebsd.pkgs.org/13/freebsd-amd64/node_exporter-1.0.1.txz.html in the main shell (not jail) *does* produce some of the kind of metrics I want, in that it includes detailed metrics on all filesytems (size, bytes, files, space free etc.) Note: It doesn't include info like zpool status but I think that's because those metrics were only recently added to node_exporter and aren't included in a release yet, see https://github.com/prometheus/node_exporter/commit/519203e3d0a0ecb9f6c09f88c27dce3c558a4a87

So there's a few possible paths here both in the short and longer term, it feels like:

Short term:
- Find a way to keep the node_exporter running on the main filesystem
or
- Find a way to expose details of all the filesystems/zpools to a jail
or
- Find a way to modify write_graphite collectd plugin to try over UDP
or
- Find a way to enable write_prometheus collectd plugin

The last 2 above may give me more metrics than node_exporter does today, like the zpool ones, not sure.

Longer term:
- Properly customisable collectd settings in UI including write_prometheus plugin, especially if this does bring the additional pool stats
and/or
- Proper support for node_exporter natively (such as a UI checkbox to toggle it on)
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
You cannot get at all the host metrics from within a jail. Plus, iXsystems extended the standard collectd package quite a bit to log things like CPU temperature etc. which are not available in the standard port.
So I would treat the TrueNAS collectd as "given" and try to adapt on the receiving side. Currently it is configured to use the Graphite plain text protocol. InfluxDB can act as a Graphite receiver, so my stack works quite conveniently. Can't you get from Graphite to Prometheus somehow? Or can't Prometheus receive Graphite directly?

Edit: ah ... stupid me. The OP already does that. So what's the problem? In Influx I just dump all the data in there, then type "show series" - and I get a list of all metrics received. Looks like the file I attached and is all the documentation I needed ...

My receiver looks like this:
Code:
[[graphite]]
  enabled = true
  database = "graphite"
  retention-policy = ""
  bind-address = ":2003"
  protocol = "tcp"
  consistency-level = "one"

  separator = "."

  templates = [
    "servers.* .hostname.resource.instance.measurement*",
  ]

Looks quite similar to the mapping configuration of the OP.
 

Attachments

  • series.txt
    78.4 KB · Views: 735
Last edited:

Henning Kessler

Contributor
Joined
Feb 10, 2015
Messages
143
Thanks for the work @tobiasbp

Any chance that you cloud share your rc script for the graphite_exporter and your grafana template if existing? :smile:
 
Top