Script to control fan speed in response to hard drive temperatures

[Glorious1]

Thanks Kevin, that's useful info. Yes, your control is very simple. Do you see the temps overshooting at all, varying in a bit of a sine wave?

 

[Kevin Horton]

No, I see my temperatures varying quite slowly, as long as the fan speed is somewhere close to optimal. Keep in mind that hard drives have quite a bit of mass, so it takes a large amount of heat energy to cause the hard drive temperature to vary by one degree.

If I look at my log of temperatures, I see that the room temperature cools at night, and the hard drive temperatures may go as low as 33°. The furnace comes on in the morning, and the current location of the server is too close to a hot air vent, so the ambient temperature increases quickly, and the hard drive temperatures increase about 3°/hour, until they hit 39° and the fan speed increases to 900 rpm. Then I see the temperatures bounce back and forth between 38° and 39° until the room temperature stabilizes, then they are 37° or 38° for the rest of the day. All in all, I see the temperatures quite stable, except when they cool a few degrees at night.

 

[Bidule0hm]

It's very easy to implement a PID controller in software (and you just need the four basic operations, no need for integral and derivative functions), the most complicated part is to tune it correctly (and drives are slow to change temps so every change in tuning will take time to see if it's better or worse).

I made a post on all of this a while ago, in the second half of the first page of this thread in fact ;)

 

[tmsmith]

Hey Kevin, this thread really has my attention but I'm a little new to scripting. I'm in desperate need of some fan control though, so I'm willing to try some things. My hottest hard drive temperatures are hitting the 50c when running at Optimal Speeds, which isn't going to cut it. At full speed, I hit around 36. I have a few questions though.

Does your script take into account the max RPMs of the specific fan or is it a hard coded RPM? I have the Noctua NF-F12 iPPC 3000 PWM that I would like to run at half speed for the medium to slow range. Is this easily adjustable?

I haven't tried it yet (as this is production) but can I run the following command to set all fans to 50%?

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32

For reference, I'm using a SuperMicro 846 with the Noctua NF-F12 120mm in the front and two 1800rpm fans in the back. Maybe the two exhaust fans aren't working good enough for all the inflow air....?

Thanks for posting this!

 

[Dice]

Can you post the 'updated' version of your script @Kevin Horton please?

 

[Kevin Horton]

Here is the latest version of my script. It controls fan speeds and logs HD temperatures.

Code:

#!/usr/local/bin/perl

# This script works on SuperMicro X9 and X10 motherboards to control case
# fan speed mode in response to hard drive temperatures.
# It should be set as a cron job to run on roughly a three minute interval.

# edit the following values
$number_of_hard_drives = 5;
$hd_designator = "/dev/da";
$min_fan_speed = 400;
$max_fan_speed = 1500;

$LogFile = '/root/HD_TempLog.txt';
$min_fan_speed *= 1.3;
$max_fan_speed *= 0.8;

# edit nothing below this line

use POSIX qw(strftime);
$datestring = strftime "%F %H:%M:%S", localtime;

open (LOGFILE, ">>$LogFile");

$max_temp = 0;

foreach $item (0..$number_of_hard_drives-1) {
  $command = "/usr/local/sbin/smartctl -A $hd_designator$item | grep Temp";

  # print "$command\n";

  $output = `$command`;
  @vals = split(" ", $output);

  # grab last item from the output, which is the hard drive temperature
  $temp = "$vals[-1]\n";

  # update maximum drive temperature
  $max_temp = $temp if $temp > $max_temp;
}

if ($max_temp > 39) {
  # at least one hard drive is 40 deg C or higher
  # set fan speed control to Full
  `ipmitool raw 0x30 0x45 0x01 0x01`
}

elsif ($max_temp == 39 ){
  # maximum drive temperature is 39 deg C
  # set fan speed to 50% duty cycle
  `ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32`
  # set fan speed to Standard
  # `ipmitool raw 0x30 0x45 0x01 0x00`
}

else {
  # all hard drive temperatures are 38 deg C or cooler
  # set fan speed control to Optimal
  `ipmitool raw 0x30 0x45 0x01 0x02`
}

# reset BMC if temps are cool, and fan is not slow
if ($max_temp < 38){
  $command = "ipmitool sdr | grep FAN2";
  $output = `$command`;
  @vals = split(" ", $output);
  $fan_speed = "$vals[2]";

  if ($fan_speed > $min_fan_speed){
    `ipmitool bmc reset cold`;
  }
}

# reset BMC if temps are warm, and fan is not fast
if ($max_temp > 40){
  $command = "ipmitool sdr | grep FAN2";
  $output = `$command`;
  @vals = split(" ", $output);
  $fan_speed = "$vals[2]";

  if ($fan_speed < $max_fan_speed){
    `ipmitool bmc reset cold`;
  }
}

print LOGFILE "$datestring - $max_temp";
close (LOGFILE);

It has been tested on FreeNAS 9.10, on a Supermicro X10SL7-F motherboard. The perl location in the first line would need to be changed to use on FreeNAS 9.3.

 

[Kevin Horton]

Hey Kevin, this thread really has my attention but I'm a little new to scripting. I'm in desperate need of some fan control though, so I'm willing to try some things. My hottest hard drive temperatures are hitting the 50c when running at Optimal Speeds, which isn't going to cut it. At full speed, I hit around 36. I have a few questions though.

Does your script take into account the max RPMs of the specific fan or is it a hard coded RPM? I have the Noctua NF-F12 iPPC 3000 PWM that I would like to run at half speed for the medium to slow range. Is this easily adjustable?

The raw commands to set fan speed are commanding a duty cycle - 100% duty cycle would be full speed. Thus that portion of the script should work for any PWM fan. I recently added a section to check for a stuck fan speed by checking to see if the fan speed is abnormally low or high given a hot or cold disk. This portion uses max and min fan speed constants that the user must set at the start of the script.

I haven't tried it yet (as this is production) but can I run the following command to set all fans to 50%?

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32

Yes, that command should command 50% duty cycle, which would be very roughly half speed.

What motherboard do you have?

The script works for me, but I make no guarantees that it will solve your problem.

I'm spending most of today trapped in an airliner, so I won't be back on the forum for quite a few hours.

 

[tmsmith]

The raw commands to set fan speed are commanding a duty cycle - 100% duty cycle would be full speed. Thus that portion of the script should work for any PWM fan. I recently added a section to check for a stuck fan speed by checking to see if the fan speed is abnormally low or high given a hot or cold disk. This portion uses max and min fan speed constants that the user must set at the start of the script.

Great! That sounds awesome. Do you have any specific instructions for creating a script within FreeNAS? I am very fresh to manipulating the IPMI via commands and would like a small tutorial. Do have one that could go over creating and navigating the script? I know this maybe asking a lot but this is truly the only reason I would need to perform these steps.

What motherboard do you have?

I have a X10SLL-F. I went ahead and ran it against with no issues. Looks like everything is now running at roughly 50%. If I wanted to make minor adjustments to the RPM, how do you know which raw data to manipulate?

I'm spending most of today trapped in an airliner, so I won't be back on the forum for quite a few hours.

Thank you for your help and sharing. Have a safe flight!

 

[Kevin Horton]

In Vancouver now, but my next flight is delayed.

You can set various PWM duty cycles by changing the last portion of that raw command. More info here.

 

[Glorious1]

I've been making progress with a bash script using the proportional and integral terms of PID. It is fully functional and seems to be working well, even during a scrub, but needs more testing.

I discovered something during one test. With the script off, I set the fans to low speed (300/400 rpm) to let the drives warm up before starting the script and seeing how it performed. After about three minutes, the fan speed jumped to 800 rpm, presumably from the BMC's check cycle. I had previously thought that when you set duty cycle with a raw command, the BMC would not control it further.

On the one hand, it's nice to know the BMC has your back if the script fails, but on the other hand, with such a script, you are not the only one attempting to control fan speed. And the logic, particularly when you are adding to or subtracting from current duty cycle, may not work right. Is there a way to tell the BMC to cease and desist?

The alternative is to just use fan mode control, but that is coarse and PID would not be appropriate. As it is, in my readings I can see the BMC and I are having a tug-of-war.

 

[tmsmith]

I apologize in advanced but I'm having some trouble getting the script to work. I have placed the script in one of my volumes and given it 'Everyone" access to hopefully alleviate the permissions issue. I am trying to run the cronjob as root and getting the following error message:

Code:

 not foundl.sh:
: not foundl.sh:
: not foundl.sh:
./fancontrol.sh: =: not found
: not foundl.sh:
./fancontrol.sh: =: not found
: not foundl.sh:
./fancontrol.sh: =: not found
: not foundl.sh:
./fancontrol.sh: =: not found
: not foundl.sh:
: not foundl.sh:
./fancontrol.sh: =: not found
: not foundl.sh:
./fancontrol.sh: *=: not found
: not foundl.sh:
./fancontrol.sh: *=: not found
: not foundl.sh:
: not foundl.sh:
: not foundl.sh:
./fancontrol.sh: 20: Syntax error: "(" unexpected

I tried following the simple guide here, but I'm not having too much luck since I'm so green in the area of FreeBSD and Unix cmds. Any help would be greatly appreciated. Thanks!

 

[Glorious1]

I have placed the script in one of my volumes . . .

What exactly is the statement in the cron job? You can't say ./fancontrol.sh, because it will look in the wrong place. You should give the full path to its location.

 

[tmsmith]

What exactly is the statement in the cron job? You can't say ./fancontrol.sh, because it will look in the wrong place. You should give the full path to its location.

I have "/mnt/volumename/scripts/fancontrol.sh" in there.

 

[Glorious1]

Is it set to be executable?

 

[tmsmith]

Is it set to be executable?

EDIT: Attached image of permissions.

I ran the following command on it: chmod +x fancontrol.sh

 

[Glorious1]

Yes, that's it. I'm not sure what those errors mean, other than a problem in line 20. What happens when you try to run the script manually, being in its directory and typing ./scriptname.sh?

 

[tmsmith]

Yes, that's it. I'm not sure what those errors mean, other than a problem in line 20. What happens when you try to run the script manually, being in its directory and typing ./scriptname.sh?

Same thing.

I doubled checked to make sure nothing got cut out. Would there be any issue with editing from Notepad++ from the CIFS share?

Code:

#!/usr/local/bin/perl
# This script works on SuperMicro X9 and X10 motherboards to control case
# fan speed mode in response to hard drive temperatures.
# It should be set as a cron job to run on roughly a three minute interval.
# edit the following values
$number_of_hard_drives = 24;
$hd_designator = "/dev/da";
$min_fan_speed = 1200;
$max_fan_speed = 2500;
$LogFile = '/root/HD_TempLog.txt';
$min_fan_speed *= 1.3;
$max_fan_speed *= 0.8;
# edit nothing below this line

 

[Kevin Horton]

I apologize in advanced but I'm having some trouble getting the script to work. I have placed the script in one of my volumes and given it 'Everyone" access to hopefully alleviate the permissions issue. I am trying to run the cronjob as root and getting the following error message:

Which version of FreeNAS are you using?

Please show the full version of your script, including line 20.

 

[Bidule0hm]

The output of cat -e the_script.sh would be useful to see if there's isn't any weird characters :)

 

[tmsmith]

Which version of FreeNAS are you using?

Please show the full version of your script, including line 20.

The output of cat -e the_script.sh would be useful to see if there's isn't any weird characters :)

I am on version FreeNAS-9.10-STABLE-201603252134. I used the command you gave Bidule, to give me the full script. I'm not sure why it's showing $ on empty lines. Is that something is shown when copied out of Putty? I don't see those when I edit the script.

Code:

#!/usr/local/bin/perl$
$
# This script works on SuperMicro X9 and X10 motherboards to control case$
# fan speed mode in response to hard drive temperatures.$
# It should be set as a cron job to run on roughly a three minute interval.$
$
# edit the following values$
$number_of_hard_drives = 24;$
$hd_designator = "/dev/da";$
$min_fan_speed = 1200;$
$max_fan_speed = 1800;$
$
$LogFile = '/root/HD_TempLog.txt';$
$min_fan_speed *= 1.3;$
$max_fan_speed *= 0.8;$
$
# edit nothing below this line$
$
use POSIX qw(strftime);$
$datestring = strftime "%F %H:%M:%S", localtime;$
$
open (LOGFILE, ">>$LogFile");$
$
$max_temp = 0;$
$
foreach $item (0..$number_of_hard_drives-1) {$
  $command = "/usr/local/sbin/smartctl -A $hd_designator$item | grep Temp";$
$
  # print "$command\n";$
$
  $output = `$command`;$
  @vals = split(" ", $output);$
$
  # grab last item from the output, which is the hard drive temperature$
  $temp = "$vals[-1]\n";$
$
  # update maximum drive temperature$
  $max_temp = $temp if $temp > $max_temp;$
}$
$
if ($max_temp > 39) {$
  # at least one hard drive is 40 deg C or higher$
  # set fan speed control to Full$
  `ipmitool raw 0x30 0x45 0x01 0x01`$
}$
$
elsif ($max_temp == 39 ){$
  # maximum drive temperature is 39 deg C$
  # set fan speed to 50% duty cycle$
  `ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32`$
  # set fan speed to Standard$
  # `ipmitool raw 0x30 0x45 0x01 0x00`$
}$
$
else {$
  # all hard drive temperatures are 38 deg C or cooler$
  # set fan speed control to Optimal$
  `ipmitool raw 0x30 0x45 0x01 0x02`$
}$
$
# reset BMC if temps are cool, and fan is not slow$
if ($max_temp < 38){$
  $command = "ipmitool sdr | grep FAN2";$
  $output = `$command`;$
  @vals = split(" ", $output);$
  $fan_speed = "$vals[2]";$
$
  if ($fan_speed > $min_fan_speed){$
    `ipmitool bmc reset cold`;$
  }$
}$
$
# reset BMC if temps are warm, and fan is not fast$
if ($max_temp > 40){$
  $command = "ipmitool sdr | grep FAN2";$
  $output = `$command`;$
  @vals = split(" ", $output);$
  $fan_speed = "$vals[2]";$
$
  if ($fan_speed < $max_fan_speed){$
    `ipmitool bmc reset cold`;$
  }$
}$
$
print LOGFILE "$datestring - $max_temp";$
close (LOGFILE);$
$