PID fan controller Perl script

JRD

Dabbler
Joined
Apr 21, 2018
Messages
42
@Kevin Horton
With my X11 motherboard I have found I can set any of the 4 fan modes by ipmitool raw commands and change from one to another. The issue with mode 1 (Full) is the fans stay at full speed, even when the mode is changed.

I also tried the commands to set the mode in words, in your script. It works for Optimal and Standard. It works for Full, but that is not suitable, at least for an X11 board. However, Heavy IO gives an error: illegal command. I tried several permutations: HeavyIO, Heavy IO and heavy io.

The script can control both zones, at least I ran it in optimal mode. The cpu fan speed (FAN1) got up to 1400. On my system this is an Intel stock cooler fan with min 800 and max 2500 rpm. So a bit fast but nothing like 3100 or 3200 I got previously running the script starting with fan mode Full.

I then looked to push the system for fan speed and temperature. Within FreeNAS I copied a large number of files into the NAS, total size about 330GB. The cpu only got up to 42c. I couldn't find or understand any other tests to run within the os. Presumably, the scripts can control the fan speeds within FreeNAS but outside this would be ipmi.

Just to be clear, I had tested cpu, memory and hard drives when I first put the system together. All good.

So I went back to Breakin https://www.familybrown.org/dokuwiki/doku.php?id=fester:hvalid_cpu and ran the Standard test.
I got errors for cpu or memory file 1 below
I therefore bought and installed a brand new cpu, same spec and ran the test again. It failed, file 2
So next I ran Memtest 86+ for over 22 hours, 11 passes: no errors, file 3

I took it up with Supermicro. To me it looked like I had eliminated the memory and cpu as the cause of the errors. Surely, the mb was most likely the cause. SM either were unfamiliar with Breakin or unwilling to accept those conclusions. The rep asked my to try Prime95. I have a copy of Ultimate Boot CD which has a Mersenne Prime test. Unfortunately, my keyboard cannot navigate the menus. Legacy USB is enabled in the BIOS. I downloaded and opened the file from the Mersenne website but I don't know what to do with it!

SM also suggested AIDA64. This only supports Windows, not FreeNAS or FreeBSD.

The rep has now said he does not believe there is a hardware fault with the motherboard, file 4

Have I missed something or should I now go for RMA?

Thanks
 

Attachments

  • Breakin file 1.PNG
    Breakin file 1.PNG
    772.5 KB · Views: 658
  • Breakin file 2.PNG
    Breakin file 2.PNG
    842.8 KB · Views: 514
  • Test Results 22 November 2018.jpg
    Test Results 22 November 2018.jpg
    51.8 KB · Views: 463
  • Supermicro Customer Reporting 28th November 2018.PNG
    Supermicro Customer Reporting 28th November 2018.PNG
    6.4 KB · Views: 449
Joined
Dec 2, 2015
Messages
730
If I understand correctly, you have successfully got the script controlling the fans, but now have what looks to you like a motherboard issue. If that is the case, it would be best to start a new thread to discuss that issue. This will avoid off-topic discussion in this thread, and it may lead to some experts reading the thread who wouldn't read a thread about PID fan control scripts.

As far as your apparent motherboard issue, I am not sure where you should go from here. Hopefully some more knowledge folks can provide useful advice.
 
  • Like
Reactions: JRD

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
This is encouraging. This is some hope.

Search in the script for the section titled "## CPU TEMP CONTROL", and set $cpu_temp_control = 0

Search in the script for all the lines "set_fan_mode("full");" and change them to "set_fan_mode("optimal");"

These changes will set the fan mode to Optimal when the script starts up, and whenever the script detects that the BMC is not controlling the fans.

These changes will also prevent the script from controlling the CPU fan, leaving that to the BMC to manage. You'll need to do some testing to confirm that the BMC is setting CPU fan duty cycle appropriatedly.

Sorry to jump into this thread however I have a Supermicro X11SSL-CF board.
According to what is written above, I need to change the script as indicated above to optimal.
If I wanted the script to control the CPU Zone (Zone 1 or Fans 1-4), I would just keep the $cpu_temp_control variable at 1?
 
Joined
Dec 2, 2015
Messages
730
If I wanted the script to control the CPU Zone (Zone 1 or Fans 1-4), I would just keep the $cpu_temp_control variable at 1?
Yes, if you want the script to control a CPU fan as a function of CPU temperature, set the $cpu_temp_control variable to 1, and connect the CPU fan to whichever fan headers are designated for CPU fan on that motherboard.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Hey thanks for you help

I have an X11 board X11SSL-CF

I was using your script and based on the information from the previous posts I tried changing your script using both full and optimal as references. I don't think that however is my problem. The script runs -- but it needs to reset the BMC. After resetting the BMC I get the following:

Code:
2019-02-07 16:02:18: CPU Fan: low
2019-02-07 16:02:18: CPU Fan speed: 2100 RPM
2019-02-07 16:02:18: HD Fan speed: 1200 RPM
2019-02-07 16:02:18: HD fan speed should be low, but 1200 > 1120.
2019-02-07 16:02:18: Fan speeds are not where they should be, will try again.
2019-02-07 16:02:18: Setting fan mode to 2 (optimal)


This is just a snippet.
It then reads hard drive temps and repeats and finally again resets the BMC -- does this over and over again resetting BMC. I'm really stuck here what to do.

In terms of configuration I have a fractal node case
CPU fan hooked to header FAN1
FAN2,3,4 header hooked to fans in CPU compartment (2 x 120 mm Enermax PWM Series TB Silence 120 Ultra Quiet Twister and 1 Notura 140mm NF-P14 PWM)
FANA header connect to splitter to 2x Enermax and 1 Noctura Fan

When adjusting Fan speeds through BMC GUI or Web Interface -- changing between full, standard, optimal -- I don't perceive a sound difference -- only when BMC is reset does system really quiet down.

Here are my limits

Code:
# ipmitool sensor list all
CPU Temp         | 25.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 100.000   | 100.000
PCH Temp         | 29.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000
System Temp      | 32.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Peripheral Temp  | 34.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
VcpuVRM Temp     | 32.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 95.000    | 100.000   | 105.000
DIMMA1 Temp      | 26.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMA2 Temp      | 27.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMB1 Temp      | 27.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMB2 Temp      | 27.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
FAN1             | 2000.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 2300.000  | 2400.000  | 2500.000
FAN2             | 1200.000   | RPM        | nr    | 300.000   | 400.000   | 500.000   | 1000.000  | 1100.000  | 1200.000
FAN3             | 1700.000   | RPM        | nr    | 300.000   | 400.000   | 500.000   | 1000.000  | 1100.000  | 1200.000
FAN4             | 1500.000   | RPM        | nr    | 300.000   | 400.000   | 500.000   | 1000.000  | 1100.000  | 1200.000
FANA             | 1200.000   | RPM        | nr    | 300.000   | 400.000   | 500.000   | 1000.000  | 1100.000  | 1200.000
12V              | 12.192     | Volts      | ok    | 10.144    | 10.272    | 10.784    | 12.960    | 13.280    | 13.408
5VCC             | 4.922      | Volts      | ok    | 4.246     | 4.298     | 4.480     | 5.390     | 5.546     | 5.598
3.3VCC           | 3.350      | Volts      | ok    | 2.789     | 2.823     | 2.959     | 3.554     | 3.656     | 3.690
VBAT             | 3.140      | Volts      | ok    | 2.384     | 2.496     | 2.580     | 3.476     | 3.588     | 3.672
Vcpu             | 0.796      | Volts      | ok    | 0.085     | 0.085     | 0.085     | 1.588     | 1.750     | 1.768
VDIMMAB          | 1.191      | Volts      | ok    | 0.948     | 0.975     | 1.047     | 1.344     | 1.425     | 1.443
5VSB             | 5.026      | Volts      | ok    | 4.246     | 4.376     | 4.480     | 5.390     | 5.546     | 5.598
3.3VSB           | 3.265      | Volts      | ok    | 2.755     | 2.840     | 2.908     | 3.503     | 3.605     | 3.639
VPCH 1.5V        | 1.041      | Volts      | ok    | 0.825     | 0.852     | 0.915     | 1.176     | 1.239     | 1.266
Chassis Intru    | 0x1        | discrete   | 0x0100| na        | na        | na        | na        | na        | na
 
Last edited:
Joined
Dec 2, 2015
Messages
730
This is likely not the whole problem, but it is clear from the ipmitool output that you need to adjust the fan speed thresholds. The actual fan speed for FAN3 and FAN4 is shown as higher than the value for Upper Non-Critical, Upper Critical and Upper Non-Recoverable. You would also need to adjust the upper thresholds for FAN2 and FANA.

After the thresholds are sorted out, you'll probably need to do some testing by setting PWM duty cycles manually to confirm which fans respond to which fan zone.

ipmitool raw 0x30 0x70 0x66 0x01 <ZONE> <DUTY>

Replace <ZONE> with either 0 or 1
Replace <DUTY> with a duty cycle, from 20 or 30, up to 100

I.e., to set zone 1 at 50% duty cycle, run ipmitool raw 0x30 0x70 0x66 0x01 1 50

Check fan speeds with ipmitool sdr | grep FAN
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Hey thanks for the reply.

Am I crazy here however I checked the fan specifications on the manufacture websites. Here is the info

1. Enermax PWM Series TB Silence 120 Ultra Quiet Twister Bearing Cooling Case Fan - .4 Amp Draw - Speed 500-1500 RPM (https://www.enermaxeu.com/products/fans/silent-fans/t-b-silence-pwm/)
2. Noctua SSO Bearing Fan NF-P14s redux - 1200 PWM - 0.08 A Draw - Speed 350 - 1200 RPM (+/- 10%) (https://noctua.at/en/nf-p14s-redux-1200/specification)
3. Intel Stock CPU fan -- I had a hard time finding info about the stock CPU cooler. One website mentioned the stock Intel CPU cooler fan ran from 800-2500 RPM. I unfortunately can't verify this information.

I set the thresholds based on this data. Why would the OS run the fans at a higher RPM than specified? I'm OK adjusting thresholds however what should I set these at? Within the Hard drive compartment driven by FANA A header I have a three way fan splitter powering two Enermax and 1 Noctua fan.

I take it that Zone 0 is the CPU compartment (FAN 1-4 computers) and Zone 1 is the FANA header

Here is fan speed output:
Code:
# ipmitool sdr | grep FAN
FAN1             | 2000 RPM          | ok
FAN2             | 1100 RPM          | cr
FAN3             | 1500 RPM          | nr
FAN4             | 1500 RPM          | nr
FANA             | 1200 RPM          | nr
 
Last edited:
Joined
Dec 2, 2015
Messages
730
Hey thanks for the reply.

Am I crazy here however I checked the fan specifications on the manufacture websites. Here is the info

1. Enermax PWM Series TB Silence 120 Ultra Quiet Twister Bearing Cooling Case Fan - .4 Amp Draw - Speed 500-1500 RPM (https://www.enermaxeu.com/products/fans/silent-fans/t-b-silence-pwm/)
2. Noctua SSO Bearing Fan NF-P14s redux - 1200 PWM - 0.08 A Draw - Speed 350 - 1200 RPM (+/- 10%) (https://noctua.at/en/nf-p14s-redux-1200/specification)
3. Intel Stock CPU fan -- I had a hard time finding info about the stock CPU cooler. One website mentioned the stock Intel CPU cooler fan ran from 800-2500 RPM. I unfortunately can't verify this information.

I set the thresholds based on this data. Why would the OS run the fans at a higher RPM than specified? I'm OK adjusting thresholds however what should I set these at? Within the Hard drive compartment driven by FANA A header I have a three way fan splitter powering two Enermax and 1 Noctua fan.

I take it that Zone 0 is the CPU compartment (FAN 1-4 computers) and Zone 1 is the FANA header

Here is fan speed output:
Code:
# ipmitool sdr | grep FAN
FAN1             | 2000 RPM          | ok
FAN2             | 1100 RPM          | cr
FAN3             | 1500 RPM          | nr
FAN4             | 1500 RPM          | nr
FANA             | 1200 RPM          | nr
All I can say is that in your original post, the speed reported by FAN3 was 1700 rpm, which was quite a bit higher than the upper thresholds you had set. Similar story with FAN4. I have no idea which fans you have connected to those fan headers, and what speed those fans run at 100% duty cycle.

Don't take the manufacturer specs as golden. Sometimes reality differs from the theoretical world of the specs.

The three upper thresholds for each fan should be higher than the speed at 100% duty cycle, to keep the BMC happy.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Ok I've totally got to be doing something wrong, however it would explain the script.

So I changed all my upper thresholds for the fans and then warm reset the BMC

Code:
ipmitool sensor list all | grep FAN
FAN1             | 1000.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 2600.000  | 2800.000  | 3000.000
FAN2             | 1000.000   | RPM        | ok    | 300.000   | 400.000   | 500.000   | 2500.000  | 2200.000  | 2000.000
FAN3             | 600.000    | RPM        | ok    | 300.000   | 400.000   | 500.000   | 2500.000  | 2200.000  | 2000.000
FAN4             | 500.000    | RPM        | nc    | 300.000   | 400.000   | 500.000   | 2500.000  | 2200.000  | 2000.000
FANA             | 600.000    | RPM        | ok    | 300.000   | 400.000   | 500.000   | 2500.000  | 2200.000  | 2000.000


I can change the mode of the fans within the GUI (Standard, Full Optimal), however everytime I try to use the raw commands, nothing seems to occur.

I've tried the following commands:
Code:
ipmitool raw 0x30 0x45 0x01 0x00  (Set to FULL mode)
ipmitool raw 0x30 0x45 0x01 0x02  (Set to Optimal mode)

ipmitool raw 0x30 0x70 0x66 1 100
ipmitool raw 0x30 0x70 0x66 0 100

ipmitool raw 0x30 0x70 0x66 1 20
ipmitool raw 0x30 0x70 0x66 0 20


The only code I get back is when using this command:
Code:
 # ipmitool raw 0x30 0x70 0x66 0 20
 06

I'm not sure what return code 06 specifies. Perhaps I have all the wrong codes. I'm not certain. I've rebooted the BMC a few times with
ipmitool reset BMC warm and ipmitool reset BMC cold. The board resets but the result of typing the raw commands are the same -- nothing seems to occur.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Ok with further testing -- clearly I was typing in the wrong commands -- I can use the ipmitool to adjust the fan speeds.
Reference page: https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/

Commands I tested which all seemed to change the fan speeds accordingly:
Code:
#set fan mode to "full"
ipmitool raw 0x30 0x45 0x01 0x01
#set fans in "system" zone to 37.5%
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x24
#set fans in "peripheral" zone to 25%
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x16
 
Joined
Dec 2, 2015
Messages
730
Which fan mode was the BMC set to when you could control the fans? Optimal, Standard, Full, Heavy IO, etc.

We need to figure out which fan zone your hard drive fans are on.

If you run ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x24 or ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x100, does that change the speed of the hard drive fans?

If not, does ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x24 or ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x100 change the speed of the hard drive fans?

Once you know which fan zone the hard drive fans are on, adjust the $cpu_fan_zone and $hd_fan_zone variables in the script to match.

Also, if you need the BMC fan mode in something other than Full to control the fans, look for the line
Code:
set_fan_mode("full");
and set to one of:

Code:
set_fan_mode("Std");
set_fan_mode("Opt");
set_fan_mode("Hvy");


Then test and report back.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Ok I can verify the CPU compartment is zone 0 and the Hard drives are Zone 1.

Question about your script
Are you sure the syntax Std Opt and Hvy is appropriate since in the script itself they have values of standard, optimal, full and heavyio?

I modified your code in your original script so either full, optimal, standard can be set within a variable at the top of the script and not like the four places where the notation is used.

At least on my X11 board it seems that the fans can be either initially set at full or optimal and then the script will run at set the fan speeds accordingly.

Its clear my server isn't under great load since when the script is running -- the fans are all at a very low speed.
Code:
ipmitool sdr | grep FAN
FAN1             | 1100 RPM          | ok
FAN2             | 400 RPM           | ok
FAN3             | 600 RPM           | ok
FAN4             | 500 RPM           | ok
FANA             | 400 RPM           | ok


What is best way to put system under stress so I can test the script?
 
Joined
Dec 2, 2015
Messages
730
Ok I can verify the CPU compartment is zone 0 and the Hard drives are Zone 1.

Question about your script
Are you sure the syntax Std Opt and Hvy is appropriate since in the script itself they have values of standard, optimal, full and heavyio?
The subroutine "get_fan_mode" sets the BMC fan mode, using the decode from subroutine "get_fan_mode_code". This portion of the script was slavishly copied from @Stux's original script, which was lightly inspired by a previous script of mine, which was triggered by a forum post that lead me to a gold mine of info in SuperMicro's FAQ.


I modified your code in your original script so either full, optimal, standard can be set within a variable at the top of the script and not like the four places where the notation is used.

At least on my X11 board it seems that the fans can be either initially set at full or optimal and then the script will run at set the fan speeds accordingly.
I should probably make a similar change, and post an updated version. I don't have an X11 board for testing, so I never wrote the script to work on those boards.
Its clear my server isn't under great load since when the script is running -- the fans are all at a very low speed.
Code:
ipmitool sdr | grep FAN
FAN1             | 1100 RPM          | ok
FAN2             | 400 RPM           | ok
FAN3             | 600 RPM           | ok
FAN4             | 500 RPM           | ok
FANA             | 400 RPM           | ok


What is best way to put system under stress so I can test the script?
On my system, two things make the hard drives work the hardest: an extended SMART test and a scrub. So, either start an extended SMART test on all drives, or launch a scrub.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
OK so I went about testing your script for a few days. Some observations.
1. With the script off (not running) and setting fan speed to FULL on GUI -- the fan speeds jump all over the place. Sitting next to the server you can hear the fans spin up and down. It's rather annoying
2. The script is able to alter the fan speeds, but when the temperatures reach a threshold (between LOW-MED, MED-HIGH), the fan speeds also tend to jump all over -- the temp seems to be read off the highest core, and where as one core may be 56 degrees, the others are 38 degreesish. Because of the frequency of the reading of the core temps -- (i'm guessing about 1-2 seconds), the highest core temp often changes for example -- 52, 56, 58, 52, 45, etc. The causes the fan speeds to jump all over the place. Without the control script, I find the BMC doing this by itself, so I'm unsure exactly the advantage is.
3. Do others find this to be the same behavior? Ideally it would be great if the script had some hysteresis and then when ramping up ran speeds it would be great if they would ramp up and down gradually based for example on the average of like 5 readings -- particularly in light of the core temps which tend to bounce and change frequently.
 
Joined
Dec 2, 2015
Messages
730
Is it the CPU fan or the HD fans that are not working as desired?

The big advantage of this script is for HD fan control. I doubt there is an advantage for CPU fan control. The PID control constants quite likely would need tweaking for your system - the ones in the script work well on my system, but every system is different.
 

ank0m

Cadet
Joined
May 23, 2019
Messages
8
Hello everybody,

first of all many thanks to everyone who contributed to make this script what it is.

I have a request regarding fan headers/script extension.

The fan zone section (should) look(s) like this (bold passages are my wishes):

# 0 = FAN1, 2, 3, 5
# 1 = FANA
# 2 = FAN4
$cpu_fan_zone = 0;
$hd_fan_zone = 1;
$external_hd_fan_zone = 2;

Fan header section (should) look(s) like follows:

$cpu_fan_header = "FAN5"; # used for printing to standard output for debugging
$hd_fan_header = "FANA"; # used for printing to standard output for debugging
$external_hd_fan_header = "FAN4"; <--- the new fan zone
@hd_fan_list = ("FANA", "FANB", "FANC"); # used for logging to file

Now to my request:
I would like to have the possibility to define another, third fan zone. This uses a header from fan zone 0 (CPU fan zone). So I had only 4 headers for the CPU zone instead of 5, which is enough.

I also would use the "borrowed" header from fan zone 0, which would belong to the new fan zone 2, for also cooling the HDDs, but the fans should be operated with different values (rotation speed) than those from fan zone 1.

Since I have absolutely no clue in fact of programming and before I mess something up, I would be very, veeeeerrry pleased if someone with expert knowledge would take over this task.

The fans (3x 140mm) would be mounted at the outside, in front of my 19" 4U chassis to blow the air to the inside. This should lower the overall dB's as my 19" rack sits right besides my desk.

I pray someone of you is kind enough to help me with this, hopefully doable, task.

Thanks for reading this.


Kind regards,
Frank
 
Joined
Dec 2, 2015
Messages
730
Hello everybody,

first of all many thanks to everyone who contributed to make this script what it is.

I have a request regarding fan headers/script extension.

The fan zone section (should) look(s) like this (bold passages are my wishes):

# 0 = FAN1, 2, 3, 5
# 1 = FANA
# 2 = FAN4
$cpu_fan_zone = 0;
$hd_fan_zone = 1;
$external_hd_fan_zone = 2;

Fan header section (should) look(s) like follows:

$cpu_fan_header = "FAN5"; # used for printing to standard output for debugging
$hd_fan_header = "FANA"; # used for printing to standard output for debugging
$external_hd_fan_header = "FAN4"; <--- the new fan zone
@hd_fan_list = ("FANA", "FANB", "FANC"); # used for logging to file

Now to my request:
I would like to have the possibility to define another, third fan zone. This uses a header from fan zone 0 (CPU fan zone). So I had only 4 headers for the CPU zone instead of 5, which is enough.

I also would use the "borrowed" header from fan zone 0, which would belong to the new fan zone 2, for also cooling the HDDs, but the fans should be operated with different values (rotation speed) than those from fan zone 1.

Since I have absolutely no clue in fact of programming and before I mess something up, I would be very, veeeeerrry pleased if someone with expert knowledge would take over this task.

The fans (3x 140mm) would be mounted at the outside, in front of my 19" 4U chassis to blow the air to the inside. This should lower the overall dB's as my 19" rack sits right besides my desk.

I pray someone of you is kind enough to help me with this, hopefully doable, task.

Thanks for reading this.


Kind regards,
Frank
The script sends commands to the BMC to set a particular fan zone at a specified duty cycle.

Your proposal could only work IF the BMC on the motherboard actually supported a Fan Zone 2 that controlled the FAN4 header. Does your motherboard support a Fan Zone 2 connected to the FAN4 header? If so, please provide a link to the docs for the BMC commands.
 

ank0m

Cadet
Joined
May 23, 2019
Messages
8
As far as my google foo works there are only two fan zone for SM motherboards. So I have to take an other route, maybe a separate fan controler with thermal sensor. Thanks anyway, @Kevin Horton.
 
Top