PID fan controller Perl script

[Kevin Horton]

I can't post in Useful Scripts, so Hardware will have to do.

I've updated @Stux's hybrid fan controller script to replace his HD fan control logic (which uses three discrete fan duty cycles as a function of warmest HD temperature) with a PID control loop, which controls duty cycle in 1% increments as a function of the average HD temperature. I left the CPU fan control portion alone. The bulk of the script is unchanged from his version, and he deserves full credit for those parts. I'll take the blame for anything I've screwed up. It works on my two systems, but YMMV. Be sure to test it carefully before trusting it.

Thanks for @Stux for his script, and to @Glorious for doing the research and creating his PID control shell script.

You should change the file extension to .pl (the site won't accept .pl uploads), and make it executable, with "chmod +x PID_fan_control.pl".



Update - 2017-09-04 - the latest versions of the script is available on Github.
Update - 2019-03-15 - This script was written and tested on Supermicro X10 motherboards. It may work on X11 boards. It is unlikely to work on X9 boards or earlier.

 

[Stux]

Hi Kevin,

Took a look at your implementation :)

Just a few comments

Code:

sub get_fan_speed2
# get fan speed for specified fan header
{
    my ($fan_name) = @_;
   
    my $command = "$ipmitool sdr | grep $fan_name";

    my $output = `$command`;
    my @vals = split(" ", $output);
    my $fan_speed = "$vals[2]";
   
    return $fan_speed;
}

I noticed the original version of this checks for "no reading" and "disabled" and non-sensical readings. If you haven't already, I'd suggest testing this code while resetting the BMC via IPMI (from a browser for instance). I found that those three readings sometimes happen in this scenario, and they then generate funny readings. Ie, the BMC comes up, but not completely up by the time you read a fan, and depending in what stage of its boot process you get it you get 'no reading', "disabled" or sometimes a very high RPM, like 28800. You should check that you don't end up in a strange place when this happens, and you can check that by reducing the loop delays, and then resetting the BMC a few dozen times ;)

Code:

sub get_fan_mode
{
    my $command = "$ipmitool raw 0x30 0x45 0";
    my $fan_code = `$command`;
   
    if ($fan_code == 1) { $hd_fan_mode = "Full"; }
    elsif ($fan_code == 0) { $hd_fan_mode = " Std"; }
    elsif ($fan_code == 2) { $hd_fan_mode = " Opt"; }
    elsif ($fan_code == 4) { $hd_fan_mode = " Hvy"; }
   
    return $hd_fan_mode;
}

sub get_fan_mode_code
{
    my ( $fan_mode )  = @_;
    my $m;

    if(     $fan_mode eq    'standard' )    { $m = 0; }
    elsif(    $fan_mode eq    'full' )     { $m = 1; }
    elsif(    $fan_mode eq    'optimal' )     { $m = 2; }
    elsif(    $fan_mode eq    'heavyio' )    { $m = 4; }
    else                     { die "illegal fan mode: $fan_mode\n" }

    dprint( 3, "fanmode: $fan_mode = $m\n"); 

    return $m;
}

Concerned that the orthogonality of these functions has been lost... ie fan mode could be mapped to fan mode code, and back again. Of course, it probably doesn't matter, but i'm imagining someone one day making bugs :)

I should setup a git repository. I think we can integrate the SAS, VM, PID etc changes into one all singing/all dancing fan control script :)

 

[Kevin Horton]

Hi Kevin,

Took a look at your implementation :)

Just a few comments

Thanks for the comments. I'll ponder them.

 

[Ceetan]

So, did anything forther happen with this script?

 

[Kevin Horton]

So, did anything forther happen with this script?

I'm still using the script as originally published on two servers, with very good performance, and no detected issues. YMMV.

 

[dj_jazzn]

Until recently, I was using the 2016-10-07 version of the hybrid_fan_control.pl script for rudimentary HDD temperature control.

Just this past week I've started evaluating this latest script. The script is currently running at debug level 1 and has logged about 5 days worth of data. The only change made to the script was the the max_fan_speed variable according to my specific hardware.

Implementing this script certainly provides much narrower temperature control and generally runs much quieter than the previous version. It appears to me that no additional PID tuning is required. I will continue to monitor over the next few days before making this change permanent. Attached is a plot of various factors observed over a 24 hour period.

Many thanks to all who have contributed to the development of this script.

Configuration at the time of this report:
FreeNAS 9.10.2 Stable
Intel Xenon E3-1231v3 | Supermicro X10SL7-F | 4X8GB Crucial 1600Mhz ECC CT2KIT102472BD160B
2X16GB Cruzer Fit Mirrored (Internally mounted 2 Port USB Motherboard Header Adapter USBMBADAPT2)
6X4TB WD40EFRX - ZFS2
Fractal Define R4 | Seasonic G-450 | 2X Noctua NF-A14 Industrial PPC-3000 PWM, 2X Cougar Vortex PWM CF-V12HP

 

[Kevin Horton]

Until recently, I was using the 2016-10-07 version of the hybrid_fan_control.pl script for rudimentary HDD temperature control.

Just this past week I've started evaluating this latest script. The script is currently running at debug level 1 and has logged about 5 days worth of data. The only change made to the script was the the max_fan_speed variable according to my specific hardware.

Implementing this script certainly provides much narrower temperature control and generally runs much quieter than the previous version. It appears to me that no additional PID tuning is required. I will continue to monitor over the next few days before making this change permanent. Attached is a plot of various factors observed over a 24 hour period.

Many thanks to all who have contributed to the development of this script.

Thanks for the performance report. The PID gains in the script are the ones I'm using on my 16 disk system. My 8 disk system has fewer and smaller duty cycle oscillations with:

$Kp = 8/3;
$Ki = 0;
$Kd = 80;

YMMV.

 

[dj_jazzn]

Thanks for the performance report. The PID gains in the script are the ones I'm using on my 16 disk system. My 8 disk system has less duty cycle oscillations with:

$Kp = 8/3;
$Ki = 0;
$Kd = 80;

YMMV.

Good to know, perhaps I will experiment with the tuning factors after all.

 

[dj_jazzn]

Thanks for the performance report. The PID gains in the script are the ones I'm using on my 16 disk system. My 8 disk system has fewer and smaller duty cycle oscillations with:

$Kp = 8/3;
$Ki = 0;
$Kd = 80;

YMMV.

A quick update on the PID tuning factors. Since your 8 disk system is similar in size to my 6 disk system, I opted to use the factors you listed above. This has resulted in a response with much lower Fan Duty oscillations compared to the previous values.

Here is another graph of temperature profiles and fan duty response over a 24 hour period with the new PID parameters.

NOTE: a portion time represented here includes a Long SMART Test.

Temperature perturbations are followed by an under damped response (diminishing oscillations). This can be observed following completion of the SMART test around 12:00 pm and also in the shorter time span represented below.

I may refresh my PID tuning knowledge and continue adjusting the control parameters further.

I do have one question regarding this script:

I have been able to modify various factors and outputs in the script, however I have not been able to toggle the reporting function to off. When I set $debug = 0 in the script, the log file will continue to be recorded. This will persist even after a system reboot. Have I missed something here?

 

[Kevin Horton]

I do have one question regarding this script:

I have been able to modify various factors and outputs in the script, however I have not been able to toggle the reporting function to off. When I set $debug = 0 in the script, the log file will continue to be recorded. This will persist even after a system reboot. Have I missed something here?

The bulk of the script, including the logging architecture, was slavishly copied from the script created by @Stux. I simply replaced his "bang bang" control scheme with a PID controller.

Setting $debug = 0 turns off any specific debug items in the log (i.e. the log is less verbose), but it does not turn off logging completely. I always wanted a log of temps and duty cycles, so I could review periodically to confirm the script was working as expected, so I never looked at modifying the code to allow logging to be selected off.

It would be possible to modify the code to disable logging, either by commenting out all the roughly 20 logging lines scattered throughout the script, or adding conditional statements to not execute them based on $logging = 0.

 

[dj_jazzn]

The bulk of the script, including the logging architecture, was slavishly copied from the script created by @Stux. I simply replaced his "bang bang" control scheme with a PID controller.

Setting $debug = 0 turns off any specific debug items in the log (i.e. the log is less verbose), but it does not turn off logging completely. I always wanted a log of temps and duty cycles, so I could review periodically to confirm the script was working as expected, so I never looked at modifying the code to allow logging to be selected off.

It would be possible to modify the code to disable logging, either by commenting out all the roughly 20 logging lines scattered throughout the script, or adding conditional statements to not execute them based on $logging = 0.

I see. I was assuming that $debug=0 meant no logging at all. Logging temperatures is prudent. Perhaps I will generate a script to periodically archive the log file to a long term location somewhere on a data set .

Thanks for the clarification.

 

[Stux]

I thought the original script was silent at log=0 :-/

It was supposed to be, and if it's not, that's a bug.

I always run it at 1 to get the temps etc

 

[Kevin Horton]

I thought the original script was silent at log=0 :-/

It was supposed to be, and if it's not, that's a bug.

I always run it at 1 to get the temps etc

To be honest, I haven't conducted an actual test using the original script. My response was based on my version of the script, which always spits out a temperature and duty cycle log. I hadn't considered that there may be people who didn't want that.

I'll create a new version that stops all logging if $debug = 0.

 

[Stux]

I hadn't considered that there may be people who didn't want that.

Yeah, I don't think it's a good idea, but I did actually specifically test (and fix) this before releasing the original script

 

[Ceetan]

Can this script pe started on its own as a post-init taks, or does it require a starupscrip simlar to Stuxs' old script?

 

[Kevin Horton]

Can this script pe started on its own as a post-init taks, or does it require a starupscrip simlar to Stuxs' old script?

I start it with a Post-Init task on both my servers. The task points directly at the script. I would have thought that Stux's script could be started the same way.

 

[Ceetan]

I start it with a Post-Init task on both my servers. The task points directly at the script. I would have thought that Stux's script could be started the same way.

I think both works. I did it the other way. So far the script has lead to more cosistant temperatures, and far less noise. :)

 

[JP Powers]

In the event any other Ubuntu Server folks stumble on this script and want to utilize it, I worked with someone who knows Perl far better than I to update a couple portions for Ubuntu Server 16.04. It utilizes the hddtemp & lm-sensors programs instead, and I'm also including the systemd service file so you can autorun it.

First off, the scripter I worked with was gperales on Fiverr. He helped me with the bits of perl that needed changing since this area's not my forte. He changed the sub's for getting HDD temps to utilize hddtemp. To keep things simple there's a global variable (@hd_list) that is a list of HD temps to monitor, which you can remove from or add to as needed. He was quick and the code looks good to me, so a big thanks to him! I then also changed the sub get_cpu_temp_sysctl to utilize lm-sensors (surprisingly enough, that much I can do).

At this point there are likely a number of comments and subs that are slightly "out of sync" with each other due to what's changed. I'm leaving it on purpose, I'd rather let Kevin decide what to do with the changes. As is it's working, which is what matters to me.

I'm currently testing the script and it seems to be working well. The fans are a bit hectic at the moment but I blame the Noctua fans I'm using as they're static pressure fans, not high air flow, and the replacements should be in Monday. Once installed I'll update here to let anyone interested know if it's more stable.

If you download both files, the following commands will install dependencies, move the files to their necessary locations, change permissions and setup the service so it can start automatically. NOTE: I'm assuming you'll know how to download the files and get them to your server on your own, so the second command is simply changing directory to the location you put the files on the server.


sudo apt install perl hddtemp lm-sensors ipmitool

cd <LOCATION YOU DOWNLOADED FILES TO>

sudo mv IPMIFanControl.pl.txt /usr/local/bin/IPMIFanControl.pl

sudo mv IPMIFanControl.service.txt /lib/systemd/system/IPMIFanControl.service

sudo systemctl daemon-reload

sudo systemctl enable IPMIFanControl.service

sudo service IPMIFanControl start


 

[Kevin Horton]

In the event any other Ubuntu Server folks stumble on this script and want to utilize it, I worked with someone who knows Perl far better than I to update a couple portions for Ubuntu Server 16.04. It utilizes the hddtemp & lm-sensors programs instead, and I'm also including the systemd service file so you can autorun it.

First off, the scripter I worked with was gperales on Fiverr. He helped me with the bits of perl that needed changing since this area's not my forte. He changed the sub's for getting HDD temps to utilize hddtemp. To keep things simple there's a global variable (@hd_list) that is a list of HD temps to monitor, which you can remove from or add to as needed. He was quick and the code looks good to me, so a big thanks to him! I then also changed the sub get_cpu_temp_sysctl to utilize lm-sensors (surprisingly enough, that much I can do).

At this point there are likely a number of comments and subs that are slightly "out of sync" with each other due to what's changed. I'm leaving it on purpose, I'd rather let Kevin decide what to do with the changes. As is it's working, which is what matters to me.

I'm currently testing the script and it seems to be working well. The fans are a bit hectic at the moment but I blame the Noctua fans I'm using as they're static pressure fans, not high air flow, and the replacements should be in Monday. Once installed I'll update here to let anyone interested know if it's more stable.

If you download both files, the following commands will install dependencies, move the files to their necessary locations, change permissions and setup the service so it can start automatically. NOTE: I'm assuming you'll know how to download the files and get them to your server on your own, so the second command is simply changing directory to the location you put the files on the server.


sudo apt install perl hddtemp lm-sensors ipmitool

cd <LOCATION YOU DOWNLOADED FILES TO>

sudo mv IPMIFanControl.pl.txt /usr/local/bin/IPMIFanControl.pl

sudo mv IPMIFanControl.service.txt /lib/systemd/system/IPMIFanControl.service

sudo systemctl daemon-reload

sudo systemctl enable IPMIFanControl.service

sudo service IPMIFanControl start


JP - I'm in the Midwest of packing to move. I won't have any free time until after the movers put all our stuff on the moving truck late next week.

Keep testing your version, and post another update in a week. I'll look at your changes then, and determine whether it is practical to add a test to check for FreeNAS or Ubuntu, and use the appropriate bits of the code.

 

[JP Powers]

JP - I'm in the Midwest of packing to move. I won't have any free time until after the movers put all our stuff on the moving truck late next week.

Keep testing your version, and post another update in a week. I'll look at your changes then, and determine whether it is practical to add a test to check for FreeNAS or Ubuntu, and use the appropriate bits of the code.

Sounds good. I know I kept rounding back to this and Stux's script trying to find a solution so I mostly just wanted to provide the option for others that will do the same. Totally up to you if you want to integrate it.