FreeNAS collectd info into Graphite

[cmh]

We're running several FreeNAS servers, and just picked up two TrueNAS units for our backups. Looking at the Reporting page I noticed it's just RRD stuff, so I investigated and discovered it's collectd. We've got a graphite server we use very heavily for gathering metrics, and since the metrics are already being gathered, couldn't help but think that it'd pretty much rock to get that RRD data into Graphite. A wee bit of searching uncovered the write_graphite plugin to collectd, but awww, it's not included in FreeNAS (/usr/local/lib/collectd) No problems, fire up a jail, install collectd, find the write_graphite plugin files, copy them over to the FreeNAS directory. Add the relevant lines to /etc/local/collectd.conf, reload collectd, life is good.

Ran into some issues, however:

• Everyone logging to the graphite host as "localhost" won't do much good. Changed hostname in collectd.conf to the server hostname, but this broke the graphs on the reporting page. Simple fix - move the old "localhost" dir in /var/db/collectd/rrd to something else, and symlink the newly created "hostname" directory in there to "localhost". Everyone's happy.
• Bigger problem that skilled FreeNAS folks already guessed - editing /etc/local/collectd.conf gets steamrolled, and the graphite changes to the file get lost. So I look at /conf/base/etc/local/collectd.conf - but it's a completely different file, it's the same as the sample one that's shipped with collectd on the jail.
• Of course, an upgrade to FreeNAS would very likely hammer the write_graphite plugin files in /usr/local/lib/collectd. Could be restored from the jail again, but that's a bit hackish.

Any thoughts on how to get this working reliably?

 

[Dusan]

Ran into some issues, however:

• Everyone logging to the graphite host as "localhost" won't do much good. Changed hostname in collectd.conf to the server hostname, but this broke the graphs on the reporting page. Simple fix - move the old "localhost" dir in /var/db/collectd/rrd to something else, and symlink the newly created "hostname" directory in there to "localhost". Everyone's happy.
• Bigger problem that skilled FreeNAS folks already guessed - editing /etc/local/collectd.conf gets steamrolled, and the graphite changes to the file get lost. So I look at /conf/base/etc/local/collectd.conf - but it's a completely different file, it's the same as the sample one that's shipped with collectd on the jail.
• Of course, an upgrade to FreeNAS would very likely hammer the write_graphite plugin files in /usr/local/lib/collectd. Could be restored from the jail again, but that's a bit hackish.

Any thoughts on how to get this working reliably?

collectd.conf is generated by /etc/rc.d/ix-collectd (you can run "service ix-collectd start" to regenerate the file). The persistent copy of the script is here: /conf/base/etc/rc.d/ix-collectd.

I customized my FreeNAS install a bit. I never modify the USB stick so that I can replace it without having to worry that I lose my changes. I have a small "system" dataset that holds my changes and a Post Init script that applies them. It modifies things in /etc and restarts services if needed. If I need to change something outside /etc (i.e. some python code) the scripts uses unionfs to attach a directory containing my changed files above the USB stick read-only directory.

Of course, the easiest (but not fastest) approach is to create a feature request (https://bugs.freenas.org/projects/freenas/issues) to get your functionality into the official build. Attaching a github pull request with the implementation helps too ;).

 

[cyberjock]

That feature sounds very cool Dusan. But wow.. that really blurs the line between being an "appliance" and being a fully functional OS that you could potentially install software on. Not sure if I like that idea or not to be honest. Supporting it would be a nightmare.

 

[cmh]

collectd.conf is generated by /etc/rc.d/ix-collectd (you can run "service ix-collectd start" to regenerate the file). The persistent copy of the script is here: /conf/base/etc/rc.d/ix-collectd.

- I came across that script in my travels and thought "ooh, that looks interesting, I should look at that... and then promptly forgot. Thanks!

I customized my FreeNAS install a bit. I never modify the USB stick so that I can replace it without having to worry that I lose my changes. I have a small "system" dataset that holds my changes and a Post Init script that applies them. It modifies things in /etc and restarts services if needed. If I need to change something outside /etc (i.e. some python code) the scripts uses unionfs to attach a directory containing my changed files above the USB stick read-only directory.

Wow, that's pretty hard core. Like cyberjock says, that really does blur the line. Just with my simple mods I'm already wondering if it might not be a bad idea to just install FreeBSD and take it from there.

Of course, the easiest (but not fastest) approach is to create a feature request (https://bugs.freenas.org/projects/freenas/issues) to get your functionality into the official build. Attaching a github pull request with the implementation helps too ;).

It would be pretty slick to be able to add a graphite server from the GUI, but that level might be above my skills at the moment and the target audience may be very slim. Still, I may just take a look. No better way to learn than by trying it out, huh?

Thanks for the info!

 

[fracai]

See this similar request for Librato support: https://bugs.freenas.org/issues/3491

 

[cmh]

Success! Modified the ix-collectd and it's working. Right now it's specific to our config, but I'll probably try to figure out how to generalize it and also handle the localhost vs hostname directory thing.

 

[cmh]

Quite a while down the road but I'm working with Graphite again, and I wrote up the procedure for logging from FreeNAS to Graphite:

http://cmhramblings.blogspot.com/2015/04/logging-freenas-performance-data-into.html

 

[cmh]

Thanks to improvements in 9.3, I've got the integration even more improved and now it's almost what I would call production-ready, at least until native Graphite support is in FreeNAS: http://cmhramblings.blogspot.com/2015/12/update-on-using-graphite-with-freenas.html