extended monitoring, checkMK and libstatgrab

[ggruber]

as I found only old threads by searching the forum:
we run a couple of FreeNAS boxes here und dropped the checkMK (aka CMK) FreeBSD Agent (a simple bash script) and some plugins (i.e. for smart monitoring) on these. All fine so far. Might be included or become a plugin, but anyway.

What I'm missing painfully is libstatgrab / statgrab for statistics of individual disks to find "low performers", as I can do this under other X-ish OS.

Could anybody share expiriences on how to add libstatgrab to a recent TrueNAS?

tnx and best regards

Gerold

 

[jgreco]

Please note that modifying the TrueNAS appliance platform is unsupported and not recommended. TrueNAS is distributed as a monolithic appliance, and despite it looking sort-of like a FreeBSD based system, changes made to the platform that are not made through the GUI are likely to be lost on update, and can interfere with updates and other system processes.

That said, why don't you just untar the appropriate package and install it by hand? This is not in any way, shape, or form advice to do so. It is just telling you how to do what you are likely to end up trying to do anyways.

 

[ggruber]

I'm aware of this opportunity and completely agree with your idea of an appliance.
the idea of this post is also to make iXsystems and the community aware of my ideas.
My silent hope is that a statgrab integration would possibly be considered useful as a standard component of TrueNAS, as a tool for tracking (and debugging) of performance issues.

Nevertheless thanks for your reply, jgreco.

 

[jgreco]

iXsystems generally does not actively source ideas from these forums, and, often, in the past, has had minimal involvement here. If you want to make a request, you probably need to file a Jira ticket. However, in the past, they have not been enthusiastic about this sort of thing, because it is opening up Pandora's box to adding this-and-that to support the hundreds of monitoring and statistics packages out there.

If you really wanted to do this "right" (for values of "right" as likely as we're going to get) and this is based off a shell script and maybe some executables, you might want to rewrite the shell script so that it can run from a dataset, locate all the binaries and libraries in there, and then you end up with a portable thing that isn't touched by updates/upgrades.

 

[ggruber]

my solution for this particular task:

• created a jail
• pkg installed libstatgrab
• extracted the needed binaries, dropped them on the host
• added those files together with the few for the check_mk_agent into a tarball
• deployed this tarball to all our machines

 

[jgreco]

Make sure you keep the jail and the tarball. You are virtually guaranteed to need at least the tarball, and the jail will be extremely helpful for updating the packages.

 

[ggruber]

(nod) I agree but all could be rebuilt very easily as there are less then 10 files (estimated) in the tar.
the step-by-step guide is my gem ;-)

 

[hp.prst]

We are having some problems getting this to work. What we have is:

• TrueNAS Core 13.0-U3.1
• Created a clone jail based on FreeBSD 13.1
• NATted port 2222 on the host to port 22 in the jail
• In the jail properties, set "devfs_ruleset" to "0"
• Installed:
  pkg install ipmitool libstatgrab bash python3 wget
  pkg install sudo
  pkg install smartmontools
  pkg install lsblk
• Installed agent:
  wget -O /usr/local/bin/check_mk_agent https://checkmk.io/3EjKJlt
  chmod 0700 /usr/local/bin/check_mk_agent
  check_mk_agent | more
• Enabled SSHD and set CheckMK server to query using ssh (Setup - Agents - Other integrations - Individual program call instead of agent access)

What we get is:

How can we make zpools on the host visible?
How can we make SMART work so we can monitor the disks?
Can someone give a more detailed description on how to monitor TrueNAS Core 13 using CheckMK via a Jail?

Thank you.

 

[ggruber]

Hi hp.prst,

did you transfer your software you grabbed in the jail to the host system?
I think you will not see the host's zpools from within the jail.
Similar for the smart plugin, but this needs in addition some tweeks to work on TrueNAS.
Possibly I could send you my version.

Regards

Gerold

 

[hp.prst]

Hi hp.prst,

did you transfer your software you grabbed in the jail to the host system?
I think you will not see the host's zpools from within the jail.
Similar for the smart plugin, but this needs in addition some tweeks to work on TrueNAS.
Possibly I could send you my version.

Regards

Gerold

Hi Gerold,

I thought the whole reason for doing things in a jail is that you don't have to modify anything on the host itself. You would loose the changes made on the host in case of a software update.

I am curious about the additional tweaks you mentioned. You can send met your version if that is ok.

Regards,
Hugo

 

[jgreco]

did you transfer your software you grabbed in the jail to the host system?

This is not the correct way to do this. TrueNAS is an appliance OS, and the base firmware is not meant to be modified in this way.

Instead, it would be prudent to allow the jail to ssh into the main system to run commands such as "zpool status", which is going to be much better supported, and will survive updates, upgrades, and even reinstalls. The basic idea is that you turn on SSHD, allow root logins (probably), set up an SSH key in the jail, allow that on the NAS root account, and then from your jail, make sure that you are able to execute "ssh root@mynas.host.name zpool status" and see that it puts out the same results as "zpool status" would on the NAS itself.