Script to control fan speed in response to hard drive temperatures

Joined
Dec 2, 2015
Messages
730
I don't have sufficient privileges to create a new thread in Hacking or Useful Scripts, so I'll post here.

My FreeNAS box (currently in hard drive badblocks testing) will be idle most of the time. The hard drive temperatures stay low with the case fans at low speed when the drives aren't being heavily used, but they climbed above 40 deg C when under constant use.

The following script uses ipmitool to set the fan speed on some SuperMicro boards to either Optimal or Full, depending on the hard drive temperatures. It works on my X10SL7-F board, and it should work on other X9 or X10 boards.

If any hard drive temperature is 40 deg C or higher, it sets the fan speed to Full. If all hard drives are 39 deg C or cooler, it sets the fan speed to Optimal. I set it as a cron job to run every five minutes.

Use at your own risk.

Code:
#!/usr/bin/perl

# edit the following values as required
$number_of_hard_drives = 5;
$hd_designator = "/dev/da";

# edit nothing below this line

$case = 0;

foreach $item (0..$number_of_hard_drives-1) {
  $command = "/usr/local/sbin/smartctl -A $hd_designator$item | grep Temp";

  # print "$command\n";

  $output = `$command`;
  @vals = split(" ", $output);

  # grab last item from the output, which is the hard drive temperature
  $temp = "$vals[-1]\n";

  # check for temperature greater than 39 deg C
  if ($temp > 39) { $case = 1; }
}

if ($case == 1) {
  # at least one hard drive is 40 deg C or higher
  # set fan speed control to Full
  `ipmitool raw 0x30 0x45 0x01 0x01`
} else {
  # all hard drive temperatures are 39 deg C or cooler
  # set fan speed control to Optimal
  `ipmitool raw 0x30 0x45 0x01 0x02`
}


This script was further improved by @Stux in post #102 of this thread.

Neither @Stux nor I are still using this type of hard drive fan control. @Stux later developed a more complex and robust fan control script that handles CPU fan as well as hard drive fans. I modified that script to use a PID control loop for the hard drive fan control - that script was posted here.
 
Last edited:

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Good idea; maybe an little improvement: if the drives go back under 40 °C let the speed to full to avoid cycling (I've the feeling it's a very bad idea for the drives' life-span to cycle the temp). And as the drives should stay under 40 °C the script is more like a last resort thing so you can afford to manually reset the fan speed to optimal as it shouldn't trigger under normal conditions.
 
Joined
Dec 2, 2015
Messages
730
Ideally, there would be a way to modulate the fan speed in very small increments, to attempt to keep the drive temperatures constant. I think the next best thing is to modulate them frequently, to keep the temperatures as close to constant as possible. The evidence I have at the moment suggests that the drive temperatures would vary by several deg C if I left the fans on constant full speed. The current script seems to be keeping them within a 1 deg C range most of the time.

I don't know yet how my hard drive temperatures will be once I have the NAS in service. The only experience I have now is during the ongoing badblocks testing, which has continuous drive activity. The drive temperatures do go above 40 deg C if the fans are on Optimal, but I think that is only during the multi-hour period when the drives are subjected to continuous writes. It looks like the temperatures stay below 40 deg during the read & compare portion of the test.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
I have at the moment suggests that the drive temperatures would vary by several deg C if I left the fans on constant full speed. The current script seems to be keeping them within a 1 deg C range most of the time.

Look at the rate of change, that's what's important here. The temps of the drives may vary by several °C but if it's slow it's not a primary concern. However cycling the temp by 1 °C every 10 min for example is a concern.

The drive temperatures do go above 40 deg C if the fans are on Optimal, but I think that is only during the multi-hour period when the drives are subjected to continuous writes. It looks like the temperatures stay below 40 deg during the read & compare portion of the test.

In a properly designed system the drives shouldn't go over 40 °C under normal conditions (including during scrub, SMART tests, heavy load, ...) so you should start by checking the thermal design of your server ;)
 
Joined
Dec 2, 2015
Messages
730
In a properly designed system the drives shouldn't go over 40 °C under normal conditions (including during scrub, SMART tests, heavy load, ...) so you should start by checking the thermal design of your server ;)
I agree that the cooling in the hard drive bay must be improved. I don't want to interrupt this multi-day long badblocks test to add another fan, but I will be tweaking things until the cooling is acceptable after badblocks has finished.

I'm expecting my server will be idle most of the time, but there will be periods when it is working hard for an hour or more at a time. What is the best way to keep constant hard drive temperatures, if the load on the server may vary wildly?
 
Joined
Dec 2, 2015
Messages
730
I searched through the FAQ section in the SuperMicro support pages, and found a raw command to specify fan duty cycle on X10 boards.
Code:
#100% duty cycle
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x64

#50% duty cycle
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32


The last value is the duty cycle, with max = 64, and minimum = 0. You can set the duty cycle in 64 steps, between minimum and full speed.

I'll play around with that to create a control loop that has more granular control of fan speed.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
I agree that the cooling in the hard drive bay must be improved. I don't want to interrupt this multi-day long badblocks test to add another fan, but I will be tweaking things until the cooling is acceptable after badblocks has finished.

Ok, perfect ;)

What is the best way to keep constant hard drive temperatures, if the load on the server may vary wildly?

A proper P(I(D)) controller, either software or hardware. Since we can't do it in software (<-- edit: lokks like I was wrong, see the edit) I'm currently designing my own hardware PI controller :)

Edit:
I searched through the FAQ section in the SuperMicro support pages, and found a raw command to specify fan duty cycle on X10 boards.

Woaw, so it looks like we can set the duty-cycle, good find ;)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
I searched through the FAQ section in the SuperMicro support pages, and found a raw command to specify fan duty cycle on X10 boards.
Code:
#100% duty cycle
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x64

#50% duty cycle
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32


The last value is the duty cycle, with max = 64, and minimum = 0. You can set the duty cycle in 64 steps, between minimum and full speed.

I'll play around with that to create a control loop that has more granular control of fan speed.
Must. Try.

Lemme check my checklist for the new server:

  • Testing phase 1 - 7 days
    • HDD Burn-in
    • General initial Burn-in
    • Document IPMI fan control
    • Cooling validation
  • Testing phase 2 - 7 days
    • Hot-swap validation
    • xHCI support in FreeNAS 9.3
    • xHCI support in FreeNAS 10
    • xHCI support in FreeBSD 10.2
    • FreeNAS 10 general testing (if possible)
    • Forum-suggested tests
  • Testing phase 3 - 7 days
    • Setup definitive pool (Flexo)
    • Replicate Bender to Flexo
    • Test and validate replication
      • Scheduling
      • Performance
      • Resilience
  • Production testing
    • Switch pools between servers (full backup!)
    • VPN setup
    • VPN testing
Whaddya know, I'm exactly at the right point to mess around with fan control!
 
Joined
Dec 2, 2015
Messages
730
Most of the time the fan goes to the expected speed after using ipmitool raw commands to set the duty cycle, as reported by "ipmitool sdr | grep FAN", but I've had two instances of the fan speed getting stuck on 100% duty cycle (i.e. full speed). In both cases I had to use "ipmitool bmc reset warm" to regain control of the fan speed.

If anyone intends to use these commands in some sort of automatic fan control, it may be wise to have the code check that the fan isn't stuck on full speed, and have it reset the BMC if necessary.
 
Joined
Dec 2, 2015
Messages
730
Can anyone suggest any CLI commands that could be used to obtain a measure of disk activity? I'm ponder possible fan control loop design, and it might be useful to know when there has been a change in the amount of disk activity. Rather than waiting for the disk temperature to increase following an increase in disk activity, it might be useful to have a feed forward gain that increases fan speed as soon as the disk activity increases. I'm envisioning measuring average disk activity over the last minute and over the last X minutes, and using the difference between those values as one of the inputs into the fan speed control loop.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Can anyone suggest any CLI commands that could be used to obtain a measure of disk activity? I'm ponder possible fan control loop design, and it might be useful to know when there has been a change in the amount of disk activity.

It's probably simpler to use a PID controller (easy in software, you don't even need integral and derivative calculus in the language you program in, just the four basic operations: + - * /) and the temp than trying to get a normalized disk activity value. Let me know if you go that route so I can help you if want ;)
 
Joined
Dec 2, 2015
Messages
730
It's probably simpler to use a PID controller (easy in software, you don't even need integral and derivative calculus in the language you program in, just the four basic operations: + - * /) and the temp than trying to get a normalized disk activity value. Let me know if you go that route so I can help you if want ;)
I am planning to try to create a PID control loop, possibly using perl. I was thinking that I might get more stable temperatures with varying disk load if I added another term that was based on disk activity. But, I should probably try a "simple" PID control loop before I assume that it won't be good enough.

My last class on control theory was in 1980 or 81, and I never actually programmed any control loops after I left university, so Google is my friend. I found a few simple equations that should get me started. Mind you, if you have a better example to suggest, or even some working code, I'd be very happy to receive it.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Hi Kevin,

Welcome to the forums. My last class in Controls was around the same time.! I was using the same motherboard as you in a fractal design define S4 case with 8 3TB NAS drives. I too found the disks went slightly above 40C under stress testing.

My solution was to use better fans with a higher minimum speed at the lowest PWM duty cycle. Upgrading to Noctua iPPC 2000s did the trick and using Optimal mode even under 100% CPU and the heaviest disk load the drives stayed under 40C

I've thought about using an arduino and a thermistor bonded to a drive to control the case fans but it seems like a lot of work for little gain.

I've had my system in production for a year now and recently downgraded the motherboard to my X9SCL with an i3-3100 and went from 8 3TB ZFS3 to 6 3TB ZFS2 as I realized that even after ripping all my DVDs and BluRays and loading all my RAW images it was more than enough storage and the e3-1231v3 could be put to better use in my workstation processing images as there simply wasn't enough load on the FreeNAS system to warrant that much CPU.

Still, I'm interested to see what you come up with.

Regards,

Bruce
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
I found a few simple equations that should get me started. Mind you, if you have a better example to suggest, or even some working code, I'd be very happy to receive it.

The one you linked is pretty good I think, it explain how you can do I and D without complex calculus ;)

I've thought about using an arduino and a thermistor bonded to a drive to control the case fans but it seems like a lot of work for little gain.

It can be done in one afternoon/night if you have everything on hand ;)

Personally I don't want a MCU or any other software solution for this (MCU can crash, program can have bugs, ...), I prefer a discrete hardware solution :)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Can anyone suggest any CLI commands that could be used to obtain a measure of disk activity? I'm ponder possible fan control loop design, and it might be useful to know when there has been a change in the amount of disk activity. Rather than waiting for the disk temperature to increase following an increase in disk activity, it might be useful to have a feed forward gain that increases fan speed as soon as the disk activity increases. I'm envisioning measuring average disk activity over the last minute and over the last X minutes, and using the difference between those values as one of the inputs into the fan speed control loop.
Might be easier to just up the D a bit.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710

Bulldog

Dabbler
Joined
Jan 16, 2016
Messages
18
I'm interested in this. Have you gotten anywhere with using the ipmitool raw command? Is there a way to send raw data to control a single fan header's duty cycle instead of all?
 
Joined
Dec 2, 2015
Messages
730
I haven't done any work on a true PID fan control loop, as it turned out that the hard drive temperatures in my FreeNAS box normally run around 35 deg C with the fans running at the slowest speed. I only ever saw warmer temperatures during the multi-day long HD burn-in process, when running badblocks.

Just in case the temperatures ever increase, I have a very simple script that runs every 3 minutes and adjusts the fan speed to either slow, medium or max, depending on the warmest HD temperature (38 deg C or cooler -> slow speed, 39 deg C -> medium speed and 40 deg C or warmer -> max speed).

The Supermicro support site suggests that ipmitool raw commands can be used to control fans in individual regions - I have experimented, but I never succeeded in controlling individual fans.
 
Joined
Dec 2, 2015
Messages
730
My script has evolved to include three different fan speeds as a function of temperature, with the middle speed using a command to set the fans to 50% duty cycle. It also logs the maximum drive temperature.

Code:
#!/usr/bin/perl

# This script works on SuperMicro X9 and X10 motherboards to control case
# fan speed mode in response to hard drive temperatures.
# It should be set as a cron job to run on roughly a five minute interval.

# edit the following values
$number_of_hard_drives = 5;
$hd_designator = "/dev/da";

$LogFile = '/root/HD_TempLog.txt';

# edit nothing below this line

use POSIX qw(strftime);
$datestring = strftime "%F %H:%M:%S", localtime;

open (LOGFILE, ">>$LogFile");

$max_temp = 0;

foreach $item (0..$number_of_hard_drives-1) {
  $command = "/usr/local/sbin/smartctl -A $hd_designator$item | grep Temp";

  # print "$command\n";

  $output = `$command`;
  @vals = split(" ", $output);

  # grab the hard drive temperature, which is the last item from the output
  $temp = "$vals[-1]\n";

  # update maximum drive temperature
  $max_temp = $temp if $temp > $max_temp;
}

if ($max_temp > 39) {
  # at least one hard drive is 40 deg C or higher
  # set fan speed control to Full
  `ipmitool raw 0x30 0x45 0x01 0x01`
}
elsif ($max_temp == 39 ){
  # maximum drive temperature is 39 deg C
  # set fan speed to 50% duty cycle
  `ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32`
}

else {
  # all hard drive temperatures are 38 deg C or cooler
  # set fan speed control to Optimal
  `ipmitool raw 0x30 0x45 0x01 0x02`
}

# log maximum drive temperature
print LOGFILE "$datestring - $max_temp";
close (LOGFILE);
 

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
Lovely @Kevin Horton !
I'm not eager to <test random raw commands> to a fully functioning "production" skylake box >.<
 
Last edited:
Top