Temperature monitoring

[nick779]

Is there any way to get a simple readout of the system/HDD temperatures added to the GUI?

I have a script that runs nightly and emails me with the temps, but Id like to see an easier "at a glance" look through the GUI.

Thoughts?

 

[Ericloewe]

I'm fairly certain there's a feature request for this. Voice your support and hope it makes it into the rewrite that's coming with FreeNAS 10.

 

[DifferentStrokes]

Use the IPMI port on your motherboard.

 

[nick779]

Use the IPMI port on your motherboard.

I do use the IPMI to check. Id just like to see something in the GUI.

 

[DifferentStrokes]

10-4

 

[FatBoyNotSoSlim]

An additional temperatures section in the GUI would be nice. Using the same graphing format, maybe one graph per physical cpu (with per core temps), system temps, and ones for hard drives. (Configurable to one graph all temps, or splitting out would be extra nice.)

...So does anyone have the correct feature request ticket where I can add the above? I couldn't find a specific one.

 

[cyberjock]

I'd imagine if one existed it was probably shot down as "that's not a job for the OS". The problem is there's so many different types of hardware out there that there is a likelyhood of there being incompatibilities (or it not being supported at all for your hardware). The only really good option is to use the IPMI. It's designed to handle that because it's OS agnostic and tied to the hardware.

The right tool for the right job.

 

[cdgonzalez]

Is the argument that the IPMI interface is a better tool for the job? The existing SMART monitoring is done via the OS correct? Would a simply GUI monitor be that much further out of scope?

Edit:
Found https://bugs.freenas.org/issues/1565 - Feature Request - add a "SMART Status" page

 

[Cellobita]

pfSense already does it, for the mobo and CPU cores (it also has access to SMART attributes, so disk temperatures are available) - perhaps it is possible to "borrow" the corresponding code and use it as a starting point.

 

[Visseroth]

I'm sorry to wake this thread up again, but system temps has been on my mind a LONG time. I've done digging and ran across this forum and thought I'd reply.

We have CPU graphs, memory graphs, heck even HDD activity graphs, those are about as necessary as CPU temp graphs. They are for informational purposes! HD, CPU and MB temp graphs, Heck even fan speed shows us how the hardware is doing in the machine, determining whether or not there is enough air flow or if there is some sort of problem, how the health of the hardware and supporting components are functioning, if a fan is going bad or if a heat sink is coming loose, ect. There are many reasons to monitor system wide temps and hardware, they give us insight into the health of the system and can give us a warning that system alerts may possibly miss. For instance if there is a time of day when the room that the system resides in is getting to warm or if under load if the system is having a hard time coping with getting rid of produced heat.
FreeNAS doesn't have the ability to just simply add an app so that someone can monitor the hardware being it uses a web interface and all. The lmsensors is already included in the CLI, why not a graph in the gui to show some history as well?
Yes, we can use things like splunk to monitor system loads, temps, ect. but not all of us are that savy. I've been trying to figure it out and I consider myself literate in a lot of ways, but apparently not enough to get splunk working decently and loading splunk into a VM adds more overhead than a simple graph ever would!

So, are temp graphs necessary? Absolutely!
Some may not think so, but then why the CPU, memory, HD, network, ect graphs? Because they provide valuable information. Another for instance would be, "Is the machine in question getting to full of dust and lint".
Not all of us put our machine is data center clean rooms. I'd love a data center style clean room but I also don't want the bill that comes along with it. I have better things to do with my money and if that means blowing out my machine every once in a while vs knowing it's clean all the time to save hundreds of $$ then I'm ALL over it!

Some may reply, "Well why don't you add it yourself?". Unfortunately I'm not a programmer nor do I have the knowledge necessary to do so, otherwise I would, then I'd write up a how to so others can enjoy the benifits of having system temp/fan speed graphs.
If I could afford it I'd just pay someone to do it. Heck, I wonder how many would donate to have someone write up such a feature.

Bottom line is, system health graphs are necessary. I'd love to have a health graph on everything including myself!

OK, enough of my ranting, but I hope to see this feature in version 10.

Thanks.

 

[Visseroth]

I'd imagine if one existed it was probably shot down as "that's not a job for the OS". The problem is there's so many different types of hardware out there that there is a likelyhood of there being incompatibilities (or it not being supported at all for your hardware). The only really good option is to use the IPMI. It's designed to handle that because it's OS agnostic and tied to the hardware.

The right tool for the right job.

Easy fix would be no detected sensors = no graphs

 

[FatBoyNotSoSlim]

Sometime after 9.3 final came out last year, I dug into the code and was able to get a semi working graph system working. Problem was, it was all in dev as I could never figure out how to build my code into a working image to then mount to a real computer. I'll take another look this week to see if a) I still have my code and b) there was working build instructions available.

From memory, I essentially replicated the entire "system" tab under the reporting, and had it pulling from the correct sensors for temps. Given a few new tabs have popped up recently (targets and more zfs graphs) it shouldn't be too hard to replicate this much again.

(Apologies for poor grammar and spelling, using mobile...)

 

[Seren]

Sometime after 9.3 final came out last year, I dug into the code and was able to get a semi working graph system working. Problem was, it was all in dev as I could never figure out how to build my code into a working image to then mount to a real computer. I'll take another look this week to see if a) I still have my code and b) there was working build instructions available.

From memory, I essentially replicated the entire "system" tab under the reporting, and had it pulling from the correct sensors for temps. Given a few new tabs have popped up recently (targets and more zfs graphs) it shouldn't be too hard to replicate this much again.

(Apologies for poor grammar and spelling, using mobile...)

If you do locate your code, I'd love to see it. I poked around in the source at one point to see how hard it would be to add temperature graphs, but didn't get very far. Seeing what you did would be a nice headstart.

 

[SneakiCow]

would this help? Freenas temp report

 

[Seren]

Thanks SneakiCow. I was hoping for something graphical, so I just wrote my own rrdtool scripts and put them up on github: https://github.com/seren/freenas-temperature-graphing

Currently the scripts record and graph CPU and drive temps at 1 and 5 minute intervals. Here's an example (the colors could probably be improved):

 

[danb35]

I just wrote my own rrdtool scripts and put them up on github: https://github.com/seren/freenas-temperature-graphing

Looks very nice, thanks for the work! I'm going to show my ignorance now and ask where the graphs appear--I'm sure they aren't in the web GUI anywhere (or are they?).

 

[Seren]

Looks very nice, thanks for the work! I'm going to show my ignorance now and ask where the graphs appear--I'm sure they aren't in the web GUI anywhere (or are they?).

No, I didn't take to time to figure out how to integrate them into the main dashboard (and it would be wiped out every time the system was updated). That might a fun addition though.

Currently the graphs and rrd files are written into the same directory that the scripts are run from.

 

[danb35]

Ah, yes, that explains it. Thanks for clarifying. Easy enough to share out that directory and view the graphs from there. I'm running into problems with the scripts, though.

I created a dataset to put them in, did 'git clone https://github.com/seren/freenas-temperature-graphing' in that dataset, and set up the cron jobs as directed, adjusting the paths to suit. But every minute (and twice every five minutes), I get an email saying:

Code:

ERROR: /mnt/tank/scripts/freenas-temperature-graphing/temps-1min.rrd: expected 14 data source readings (got 9) from N

When I run the temps-simple.sh script, I get this:

Code:

[root@freenas2] /mnt/tank/scripts/freenas-temperature-graphing# ./temps-simple.sh
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada0' as root on freenas2.familybrown.org.
ada0 - 
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada1' as root on freenas2.familybrown.org.
ada1 - 
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada2' as root on freenas2.familybrown.org.
ada2 - 
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada3' as root on freenas2.familybrown.org.
ada3 - 
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada4' as root on freenas2.familybrown.org.
ada4 - 
Sorry, user root is not allowed to execute '/usr/local/sbin/smartctl -a /dev/ada5' as root on freenas2.familybrown.org.
ada5 - 
dev.cpu.0.temperature: 57.0C
dev.cpu.1.temperature: 57.0C
dev.cpu.2.temperature: 59.0C
dev.cpu.3.temperature: 59.0C
dev.cpu.4.temperature: 57.0C
dev.cpu.5.temperature: 56.0C
dev.cpu.6.temperature: 57.0C
dev.cpu.7.temperature: 57.0C
[root@freenas2] /mnt/tank/scripts/freenas-temperature-graphing# 

It may also be relevant that my system has a total of 14 drives: mirrored boot SATA DOMs (ada0-1), four more data disks on the onboard ports (ada2-5), and eight more data disks on an M1015 card (da0-7).

Edit: Removing the 'sudo' from the 'sudo smartctl...' line in temps-simple.sh did the trick for that script, but nothing for the others.

 

[Seren]

@danb35 Sorry for the delayed response. Since my system only has 4 drives, there's probably a bug with drives over 10 (since the error mentions 9 data-points). It might have something to do with text-parsing double-digit drive numbers, but that's speculation.

Could you login in as root, change your shell to bash, and run running the components manually? I'd start with the script that reads the SMART values:

Code:

/mnt/tank/scripts/temps-rrd-format.sh

If you get an error, could you run and post the results of this:

Code:

set -o xtrace
/mnt/tank/scripts/temps-rrd-format.sh
set +o xtrace

If the above works swimmingly, then could you do the same with the script that initializes and updates the rrd database:

Code:

/mnt/tank/scripts/temperature-monitoring/rrd.sh /mnt/tank/scripts/temperature-monitoring/temps-1min.rrd

and if there's an error, post the results of this:

Code:

set -o xtrace
/mnt/tank/scripts/temperature-monitoring/rrd.sh /mnt/tank/scripts/temperature-monitoring/temps-1min.rrd
set +o xtrace

Thanks. :)

 

[danb35]

Code:

[root@freenas2] /mnt/tank/scripts/freenas-temperature-graphing# ./temps-rrd-format.sh
44:45:49:49:40:40:42:42:25 0:25 0:33:32:31:32
[root@freenas2 /mnt/tank/scripts/freenas-temperature-graphing]# set -o xtrace
[root@freenas2 /mnt/tank/scripts/freenas-temperature-graphing]# ./rrd.sh temps-1min.rrd
+ ./rrd.sh temps-1min.rrd
ERROR: temps-1min.rrd: expected 14 data source readings (got 9) from N
[root@freenas2 /mnt/tank/scripts/freenas-temperature-graphing]# set +o xtrace
+ set +o xtrace

Edit: If I assume that the CPU temps are given first, and then the drive temps from ada0-ada6, the two drives that are reporting temps of "25 0" are my boot devices, two SATA DOMs. This seems to be confirmed by the output of temps-simple.sh:

Code:

[root@freenas2] /mnt/tank/scripts/freenas-temperature-graphing# ./temps-simple.sh
ada0 - 25C
0C 
ada1 - 25C
0C 
ada2 - 33C 
ada3 - 32C 
ada4 - 32C 
ada5 - 32C 
dev.cpu.0.temperature: 54.0C
dev.cpu.1.temperature: 54.0C
dev.cpu.2.temperature: 52.0C
dev.cpu.3.temperature: 52.0C
dev.cpu.4.temperature: 45.0C
dev.cpu.5.temperature: 45.0C
dev.cpu.6.temperature: 46.0C
dev.cpu.7.temperature: 46.0C
[root@freenas2] /mnt/tank/scripts/freenas-temperature-graphing#