Script to control fan speed in response to hard drive temperatures

[Mr Snow]

The output of camcontrol devlist seems to change the order of passX and adaX for every OS version. I've had to change that back and forth several times to keep up. Now it's correct for my adaX drives but not for my daX drives (on a M1015 adapter). I'll fix that . . . again. Here's my current output, very annoying:

Maybe the easiest way to have a unified script is to add a zone or region setting. ZONE=0 if you have only one zone, like me, and ZONE=1 if there are two zones. Then, when issuing the ipmitool raw command, use $ZONE variable in that place. Would that work?

I did consider doing that. Just have a simple variable you set to be zone 0 or zone 1. I'll update my script in the next couple of days and post it.

I've also got a modification for you on retrieving the device name. I changed to using two grep lines with regex instead of the awk you are using. It could probably be done with one regex, but I couldn't figure it out :D I'll post that once I get home tonight.

Regards,

CJ

 

[Mr Snow]

@Glorious1 here is my mod to your line to retrieve disk names:

Code:

DEVID=$(echo $LINE | grep -Eo '\((.*)\)' | grep -Eo 'a?da[0-9]' )

Regards,

CJ

 

[Jr922]

My supermicro 846 cooks my drives so I'm trying to get this script running but its keeps pushing fans to 100% when max temp is below max allowed temp. I'm not sure if its pulling the HDD temps properly because my log is displaying the temp value which is around 150 when the actual temp (Raw_value) = 40c

Here is the output from the script when debug = 4 and max allowed temp = 45

[root@VeNASflytrap] ~# /usr/bin/fans.pl
/usr/local/sbin/smartctl -A /dev/da0 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 157 157 000 Old_age Always - 38 (Min/Max 25/57)

/dev/da0: 157
/usr/local/sbin/smartctl -A /dev/da1 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 150 150 000 Old_age Always - 40 (Min/Max 21/61)

/dev/da1: 150
/usr/local/sbin/smartctl -A /dev/da2 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 139 139 000 Old_age Always - 43 (Min/Max 19/57)

/dev/da2: 139
/usr/local/sbin/smartctl -A /dev/da3 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 142 142 000 Old_age Always - 42 (Min/Max 25/57)

/dev/da3: 142
/usr/local/sbin/smartctl -A /dev/da4 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 150 150 000 Old_age Always - 40 (Min/Max 23/54)

/dev/da4: 150
/usr/local/sbin/smartctl -A /dev/da5 | grep Temperature_Celsius
194 Temperature_Celsius 0x0002 153 153 000 Old_age Always - 39 (Min/Max 24/54)

/dev/da5: 153
Maximum HD Temperature: 157

drives are too hot, going to 100%

 

[Stux]

The awk line is pulling out the wrong number. Someone better at scripting than me might want to propose an alternative.

I believe my version simply grabs the fifth value.

 

[asmodeus]

The awk line is pulling out the wrong number. Someone better at scripting than me might want to propose an alternative.

I believe my version simply grabs the fifth value.

This works for me (with SATA drives). E.g. for da0:

Code:

smartctl -a -n standby /dev/da0 | awk '/194 Temperature_Celsius/ { print $10 }'

For the script, replace /dev/da0 with "/dev/$DEVID"

 

[asmodeus]

I've got a 4-way PWM splitter cable and want to connect my two HD PWM fans to FANA. So I plugged them into the cable and left the other two sockets empty. Do you guys still get an RPM reading with this setup? I'm not getting any reading for FANA from ipmitool:

Code:

[root@freenas ~/bin]# ipmitool sdr type fan
FAN1             | 41h | ok  | 29.1 | 2000 RPM
FAN2             | 42h | ok  | 29.2 | 1200 RPM
FAN3             | 43h | ok  | 29.3 | 1300 RPM
FAN4             | 44h | ns  | 29.4 | No Reading
FANA             | 45h | ns  | 29.5 | No Reading

 

[asmodeus]

I've got a 4-way PWM splitter cable and want to connect my two HD PWM fans to FANA. So I plugged them into the cable and left the other two sockets empty. Do you guys still get an RPM reading with this setup? I'm not getting any reading for FANA from ipmitool:

Code:

[root@freenas ~/bin]# ipmitool sdr type fan
FAN1             | 41h | ok  | 29.1 | 2000 RPM
FAN2             | 42h | ok  | 29.2 | 1200 RPM
FAN3             | 43h | ok  | 29.3 | 1300 RPM
FAN4             | 44h | ns  | 29.4 | No Reading
FANA             | 45h | ns  | 29.5 | No Reading

Well scratch that. One of the PWM splitter cables is marked as "with fan speed reading"...

 

[Jr922]

This works for me (with SATA drives). E.g. for da0:

Code:

smartctl -a -n standby /dev/da0 | awk '/194 Temperature_Celsius/ { print $10 }'

For the script, replace /dev/da0 with "/dev/$DEVID"

This code works for me thank you.

I was still having the issue because it waasnt just that line, the code above would give me the temp properly when run by itself; but when it was in the script the code to pull the 4th value was still pulling the value and not the raw_value that I needed.
I changed that line to this:
# grab 10th item from the output, which is the raw_value hard drive temperature
$temp = "$vals[9]\n";

this line should work for either command
and it looks good so far.

 

[Stux]

I'm looking at adding a CPU temp control loop.

There would be the one or two zones, but also if the HD zone is independent of the CPU zone, or if you want to increase the HD zone when you increase the CPU zone in response to escalating CPU temps.

My idea is basically to work out the CPU and HD zone duty cycles, and if the CPU zone is greater than HD zones duty cycle AND they aren't independent, then set both zones to the CPU duty cycle, else their own.

By independent, I'm referring to cases where the HDs are in a separate compartment which does not share airflow with the CPU compartment

So, I actually did this thing :)

Hybrid CPU & HD Fan Controller Script

 

[Stux]

I think I've worked out why I'm seeing Optimal changing FANA. Its because it sets FANA to a stalling speed. FANA stalls, it then surges up...

Your fans are probably not stalling in Optimal.

edit: I've been leaving the system in CPU burn mode, in Optimal, FANA fans are surging as optimal dials the fans to 20%, which is a stalling speed for my fans :(

Since I gave up on Optimal mode, and started doing all my testing with Full, and then wrote the hybrid cpu/hd fan controller loop, I have not had any issues with my 120mm Noctuas stalling.

I had edited the default thresholds, so not sure if that contributed, but as it stands I'm now very happy with the fan control on my case!

(it took me a while to get back to it)

:)

 

[Keresian]

What are the chances that this (or a slightly modified version of this) would work on an X8 board?

 

[Glorious1]

Chances are pretty good if you can find or get SuperMicro to tell you the ipmitool commands. Or you could just start experimenting and hope the commands are the same.

 

[Stux]

Chances are pretty good if you can find or get SuperMicro to tell you the ipmitool commands. Or you could just start experimenting and hope the commands are the same.

I don't think the x8 systems support setting Fan Duty, but at least some of them might support setting Fan Mode. So you can at least have a script which fluctuates the fans from 50% to 100% (Standard to Full) based on temp.

The trick is to test the ipmitool commands used in the scripts to see what works.

 

[Kevin Horton]

PID Fan Control Script with Detailed Logs

EDIT: I found when the processor was under heavy load, its temperature spiked. Of course the script only responds to drive temperature. So, like @Stux did for X10 boards, I modified the script substantially to respond to both drive and CPU temperature. My board has only one fan zone, so it is not split as in Stux's script. The relationship is a bit complicated, but I have tested for days with a lot of scrubs and processor-intensive tasks, and it has worked very well. When the drives need cooling, PID is still used. When the CPU needs more attention, the fans simply ramp up with temperature, and there are two settings to regulate that. New scripts attached.
****************************************

I found that with my Noctua NF-F12 fans for the hard drives, a 20% duty cycle would occasionally lead to a fan stalling, which would cause the BMC to put all fans to 100% duty cycle and stay there. I ended up changing the minimum allowable duty cycle to 30% to avoid that issue.

It may be worthwhile to modify @Glorious1's script to have the minimum duty cycle as a variable defined at the start of the script, to make it clear that this may need to be adjusted, and to provide an obvious place to make that adjustment.

 

[Glorious1]

I found that with my Noctua NF-F12 fans for the hard drives, a 20% duty cycle would occasionally lead to a fan stalling, which would cause the BMC to put all fans to 100% duty cycle and stay there. I ended up changing the minimum allowable duty cycle to 30% to avoid that issue.

It may be worthwhile to modify @Glorious1's script to have the minimum duty cycle as a variable defined at the start of the script, to make it clear that this may need to be adjusted, and to provide an obvious place to make that adjustment.

Good call. I guess my duty cycle never got that low, but you must have better cooling or just a cool home in the Great White North :)
I updated the script and replaced it in the post you quoted. Thanks!

 

[Stux]

I found that with my Noctua NF-F12 fans for the hard drives, a 20% duty cycle would occasionally lead to a fan stalling, which would cause the BMC to put all fans to 100% duty cycle and stay there. I ended up changing the minimum allowable duty cycle to 30% to avoid that issue.

I found exactly the same thing :)

 

[Kevin Horton]

Good call. I guess my duty cycle never got that low, but you must have better cooling or just a cool home in the Great White North :)
I updated the script and replaced it in the post you quoted. Thanks!

The fans kept going to full speed during bad blocks testing. I reset the BMC multiple times, shut down and restarted the PID script, but they would only stay slow for a few minutes before jumping to 100%. I eventually shutdown the script and troubleshot by trying various fan duty cycles, while watching the rpm. I found that 20% duty cycle was causing a repeatable fan stall.

I do not understand why the script should have commanded a 20% duty cycle, as a higher value than that was needed to maintain the target HD temperatures, but the problem did not reoccur after I changed the minimum duty cycle.

I'm still suspicious that I haven't fixed the correct root cause, as I cannot explain how this fix worked.

 

[Stux]

The fans kept going to full speed during bad blocks testing. I reset the BMC multiple times, shut down and restarted the PID script, but they would only stay slow for a few minutes before jumping to 100%. I eventually shutdown the script and troubleshot by trying various fan duty cycles, while watching the rpm. I found that 20% duty cycle was causing a repeatable fan stall.

I do not understand why the script should have commanded a 20% duty cycle, as a higher value than that was needed to maintain the target HD temperatures, but the problem did not reoccur after I changed the minimum duty cycle.

I'm still suspicious that I haven't fixed the correct root cause, as I cannot explain how this fix worked.

I found Optimal mode would periodically set fan duty cycle to circa 20%. Max mode does not.

 

[Kevin Horton]

I found Optimal mode would periodically set fan duty cycle to circa 20%. Max mode does not.

The fan mode was set to Full - I checked that multiple times when I was having the issue with fans going to 100%, as I knew that it could be a possible cause of the problem I was seeing.

 

[Stux]

The fan mode was set to Full - I checked that multiple times when I was having the issue with fans going to 100%, as I knew that it could be a possible cause of the problem I was seeing.

Did you adjust the IPMI fan thresholds too?