Register for the iXsystems Community to get an ad-free experience

PID fan controller Perl script

Gaspetaahl

Explorer
Joined
Sep 13, 2018
Messages
59
They are Arctic P12 PWM 120mm (4Pins) I had 3 Pin Fans before but the X10 cant control 3Pin fans

 

Gaspetaahl

Explorer
Joined
Sep 13, 2018
Messages
59
I wondered why the ipmitool commands failed so I edited the script to print each command. This is my output. maybe that helps:
Code:
]# ./PID_fan_control.pl
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
^C
Caught SIGINT: setting fan mode to optimal
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 2
No data available
Get Device ID command failed
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x45 rsp=0xc1): Invalid command


I am confused about the numbers not being in hex. Maybe I did the printing wrong. I never coded in perl.
This is my script:

Edit:
I also saw this when running ipmitool commands after triggering a bmc reset which are the same errors:
Code:
ipmitool bmc reset cold
Sent cold reset command to MC
[root@freenas ~/nas_fan_control]# ipmitool raw 0x30 0x45 0x01 0x01
No data available
Get Device ID command failed
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x45 rsp=0xc1): Invalid command


Edit2:
I updated my BMC and BIOS and I think it might have fixed some errors. This is the new stdout. The "Unable to send RAW command" errors do not appear anymore
Code:
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 37
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
^C
Caught SIGINT: setting fan mode to optimal
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 2


This happens when I execute the ipmitool commands without the script after a reset:
Code:
# ipmitool bmc reset cold
Sent cold reset command to MC
root@freenas:~/nas_fan_control # ipmitool raw 0x30 0x45 0x01 0x01
No data available
Get Device ID command failed
No data available
No data available
No valid response received
 20 01 03 88 02 bf 7c 2a 00 01 08 00 00 00 00


 

Attachments

  • PID_fan_control.txt
    48.4 KB · Views: 56
Last edited:
Joined
Dec 2, 2015
Messages
717
Edit2:
I updated my BMC and BIOS and I think it might have fixed some errors. This is the new stdout. The "Unable to send RAW command" errors do not appear anymore
Code:
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 37
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 20
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 30
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 60
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 1
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 1 100
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x70 0x66 0x01 0 16
/usr/local/bin/ipmitool raw 0x30 0x45 0
^C
Caught SIGINT: setting fan mode to optimal
/usr/local/bin/ipmitool raw 0x30 0x45 0x01 2
It seems strange that ipmitool raw 0x30 0x45 0x01 1 appears to work sometimes, and sometimes it triggers an error. I'm baffled. Maybe there is still some bug in your current version of the BMC. Or maybe the BMC is randomly rebooting for an unknown reason - perhaps a bad BMC, or a flakey power supply, or loose connection somewhere in the system. Is the power supply wattage guaranteed to be adequate for your system?

This happens when I execute the ipmitool commands without the script after a reset:
Code:
# ipmitool bmc reset cold
Sent cold reset command to MC
root@freenas:~/nas_fan_control # ipmitool raw 0x30 0x45 0x01 0x01
No data available
Get Device ID command failed
No data available
No data available
No valid response received
 20 01 03 88 02 bf 7c 2a 00 01 08 00 00 00 00
It takes the BMC many seconds to reboot after a cold reset, so it is expected that there would be no valid response for some period of time.
 

Gaspetaahl

Explorer
Joined
Sep 13, 2018
Messages
59
Its a 630W PSU for a Xeon e3-1231 v3 and 5 HDDs I think it should be adequate.
Also I noticed that the bmc intervenes when I try to manually set the fan speed. Before using the script I used the commands as described here:
which set the fans to a constant speed when using "Full" Fan mode. But now the bmc sometimes ramps up the fans only to set them to low speed afterwards. I sound quite annoying. This also happened before the BMC update.
 
Joined
Dec 2, 2015
Messages
717
Here is my current theory, which I think matches the available evidence since you updated the BMC.

Looking at the Debug Log, I see frequent occurrences where the monitored HD Fan (fan header FAN1) fails to report a speed. If this condition persists, the script assumes that the BMC has locked up (I saw occasional BMC lockups during my testing) and it resets the BMC. The BMC fan mode is set to Full, so the HD fans go to full speed after the BMC resets, then the script will command a lower duty cycle once that part of the control loop runs (the HD temperatures are only checked every 90 seconds, so the fans could be at max speed for up to 90s before the script commands a lower duty cycle).

So, the question is why FAN1 frequently fails to report a speed. Maybe FAN1 or its cable, or the connection to the motherboard is flakey. I'd disconnect and reconnect FAN1 from the motherboard. If the problem persists, try changing $hd_fan_header to "FAN2", or "FAN3", or "FAN4".

Another possibility is that the BIOS is interfering somehow. Comparing the manual for your X10SLL board against the manual for my X10SRH board, I see that the BIOS for your board has an Energy Performance setting, which selects different fan modes. I don't see reference to that on my board. It might be worth checking that setting in the BIOS - looking at the various options, I think "Performance" is likely the one you want.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,248
You do need to set the fan critical thresholds on the BMC to prevent it restting or taking over when fans fall below the thresholds.

Also, I do remember some RAW command flakiness from other people with older BIOSes in the past.

Here's an example of me doing it with another build of mine: Fan Control Thresholds for Node 304/X10-SDV
 
Joined
Dec 2, 2015
Messages
717
You do need to set the fan critical thresholds on the BMC to prevent it resetting or taking over when fans fall below the thresholds.
Excellent point - incorrect fan speed thresholds would explain the current behavior. See this thread for details on how to set the fan speed thresholds.
 
Joined
Dec 2, 2015
Messages
717
What is the make and model of your CPU fan?

What is the output of ipmitool sensor | grep FAN
 

nick23369

Cadet
Joined
Jan 19, 2021
Messages
8
Hi Kevin,
First of all I really thank you for the great script, it works like a charm. I have tested the script on my new board a Supermicro H11SSL-i with a Epyc 7551P with out any major changes and it does what I expect. I started using Truenas in January 2021 and I'm still learning, therefore please excuse any question that might sound silly. My real question is: what is the best way to let the script running in the background. At the moment I can only let it run in a console but as soon as I exit the console the script exits (as it is clearly visible in your script). I also have tried to run it like this:
Code:
 PID_fan_control.pl & 
but as well as soon as I quit the console the script quit. I have as well tried to set up a service, but as well without success: as soon as I launch the service I never get back the prompt and when I leave the console the job quit. I guess I might be missing something in the script. Here there is the rc script that I have added in
Code:
 /etc/rc.d/pid-fan-control
Code:
#!/bin/sh
# PROVIDE: pid-fan-control
# REQUIRE: DAEMON
# pidfancontrol_enable="YES"
# ... to /etc/rc.conf

. /etc/rc.subr
name=pidfancontrol
rcvar=pidfancontrol_enable
command="/usr/sbin/PID_fan_control.pl";
pidfile="/var/run/${name}.pid"
load_rc_config $name
run_rc_command "$1"

I also thought that I could add the script as a post init in TrueNAS, but if I would need to change any parameter I guess I would need to restart TrueNAS (which I don't really want), unless I mange to get the service working properly or any other way to get have the script working in the background.
I tried to look into the discussion but I couldn't find the answer I was looking for.
Any help is really appreciated.
 
Joined
Dec 2, 2015
Messages
717
I run the script automatically as a Init/Shutdown script of type POSTINIT. I have selected the option to use a config file for the parametres (see Github to get the later versions of the script that have the config file option). The script reads the config file each time around the loop, and picks up the new values if there have been changes.

Another option is to use tmux or screen to run the script, but this adds the risk that the script would not be running if the server had an uncommanded reboot.
 

nick23369

Cadet
Joined
Jan 19, 2021
Messages
8
I run the script automatically as a Init/Shutdown script of type POSTINIT. I have selected the option to use a config file for the parametres (see Github to get the later versions of the script that have the config file option). The script reads the config file each time around the loop, and picks up the new values if there have been changes.

Another option is to use tmux or screen to run the script, but this adds the risk that the script would not be running if the server had an uncommanded reboot.
Hi Kevin,
thanks a lot for the super quick answer, I'll follow your suggestion of the POSTINIT, I didn't realize that the script reads the config file more than once. I'll come back if I need further help.
Thanks again!
 

sretalla

Hall of Famer
Joined
Jan 1, 2016
Messages
7,013
I would recommend running your postinit script like this:

tmux new-session -d -s fanscript '/mnt/pool/path/script.pl'

This way, you can stop and restart your script whenever you want to make changes outside of the "live" config by attaching to the tmux session like this:

tmux attach -t fanscript

CTRL + C to stop the script... make your adjustments... then start it again from that same tmux session /mnt/pool/path/script.pl

Then CTRL + B followed by D to detach again
 

nick23369

Cadet
Joined
Jan 19, 2021
Messages
8
I would recommend running your postinit script like this:

tmux new-session -d -s fanscript '/mnt/pool/path/script.pl'

This way, you can stop and restart your script whenever you want to make changes outside of the "live" config by attaching to the tmux session like this:

tmux attach -t fanscript

CTRL + C to stop the script... make your adjustments... then start it again from that same tmux session /mnt/pool/path/script.pl

Then CTRL + B followed by D to detach again
Hi sretalla, thanks for this tip. Currently I'm running the script with tmux as you suggested as a normal process, not yet in the postinit. I didn't want to restart TrueNAS for that, but so far everyting is working as I want. Thanks all for your help!
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,248
I run the script automatically as a Init/Shutdown script of type POSTINIT. I have selected the option to use a config file for the parametres (see Github to get the later versions of the script that have the config file option). The script reads the config file each time around the loop, and picks up the new values if there have been changes.

Another option is to use tmux or screen to run the script, but this adds the risk that the script would not be running if the server had an uncommanded reboot.
I would recommend running your postinit script like this:

tmux new-session -d -s fanscript '/mnt/pool/path/script.pl'

This way, you can stop and restart your script whenever you want to make changes outside of the "live" config by attaching to the tmux session like this:

tmux attach -t fanscript

CTRL + C to stop the script... make your adjustments... then start it again from that same tmux session /mnt/pool/path/script.pl

Then CTRL + B followed by D to detach again
What sretalla said.

Also, when the middleware crashes and reboots it won’t take your init script down with it.
 

gdarends

Explorer
Joined
Jan 20, 2015
Messages
70
Hi,
Can anyone explain the PID control config?
Or maybe somewhere I can read to understand it?
I don't know what values to use.
I am using TrueNAS Scale with 6 HD's
 

gdarends

Explorer
Joined
Jan 20, 2015
Messages
70
I have left the PID config to default for now.

Does this script work with fans with different speeds?
2 x Noctua NF-A12x25 PWM connected to FANA header with max speed of 2000 RPM
1 x Noctua NF-A14 PWM connected to FANB header with max speed of 1500 RPM
I have disabled CPU fan control in the config.
So I'm only doing Zone 1 for HD fan control.
What should I put at $hd_max_fan_speed and $hd_fan_header?


EDIT: This script doesn't work on SCALE it seems. camcontrol isn't available on SCALE / Linux
 
Last edited:
Joined
Dec 2, 2015
Messages
717
I have left the PID config to default for now.

Does this script work with fans with different speeds?
2 x Noctua NF-A12x25 PWM connected to FANA header with max speed of 2000 RPM
1 x Noctua NF-A14 PWM connected to FANB header with max speed of 1500 RPM
I have disabled CPU fan control in the config.
So I'm only doing Zone 1 for HD fan control.
What should I put at $hd_max_fan_speed and $hd_fan_header?


EDIT: This script doesn't work on SCALE it seems. camcontrol isn't available on SCALE / Linux
I'm in the midst of a big road trip, driving all day today and most of tomorrow. Hence the delayed reply.

The required value for $hd_max_fan_speed depends on your fans. Setting a value a bit higher than the max fan speed provided by the manufacturer is a good start, but sometimes the actual fans don't exactly meet the specs, so you may need to increase this setting even a bit more to avoid false detection of runaway fans.

For $hd_fan_header, look at the output of ipmitool sdr and determine which header one of your HD fans is connected to.

I don't have access to SCALE, so I cannot determine what changes could be made to the script to replace camcontrol with whatever equivalent utilities are available on Linux. But there must be something that could do the same functions as camcontrol
 

gdarends

Explorer
Joined
Jan 20, 2015
Messages
70
The required value for $hd_max_fan_speed depends on your fans.
I have 2 different fans that have different max RPM. That's why I wasn't sure if it would work.
For $hd_fan_header, look at the output of ipmitool sdr and determine which header one of your HD fans is connected to.
Same here. I have 2 different fans. Not sure how it would work if they are different.

Yeah, I tried searching for camcontrol equivalent in Linux, didn't get any hits.
 
Joined
Dec 2, 2015
Messages
717
I have 2 different fans that have different max RPM. That's why I wasn't sure if it would work.

Same here. I have 2 different fans. Not sure how it would work if they are different.

Yeah, I tried searching for camcontrol equivalent in Linux, didn't get any hits.
Pick one fan, and use the max rpm and header for that fan.
 
Top