PID fan controller Perl script

[Kevin Horton]

Let's first sort out the fact that the script is trying to also work with your boot drive. Find the first line in "sub get_hd_list", which should be somewhere around line 509 of PID_fan_control.pl. The line should be something like

Code:

my $disk_list = `camcontrol devlist | grep -v "SSD" | grep -v "Verbatim" | grep -v "Kingston" | grep -v "Elements" | sed 's:.*(::;s:).*::;s:,pass[0-9]*::;s:pass[0-9]*,::' | egrep '^[a]*da[0-9]+\$' | tr '\012' ' '`;

Replace "Kingston" with "SanDisk" in that line so it will recognize that it should ignore your SanDisk boot drive.

Kill and restart the script after you have edited it, or restart the NAS.

I honestly don't think this change will stop the script from terminating, but it will stop it from trying to work with the boot drive, so there is a chance that will fix the problem. If the crashes continue, let me know and we'll need to do some more troubleshooting.

I will be on the road for the next four days, so won't have much free time to work this issue. But I should be available again on Tuesday.

 

[Gaspetaahl]

I changed the line as you suggested and let the script run for a few minutes. The fans turned up and down a bit, but I got these errors in stdout:

Code:

No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request

 

[Kevin Horton]

I changed the line as you suggested and let the script run for a few minutes. The fans turned up and down a bit, but I got these errors in stdout:

Code:

No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request

These look like error messages from ipmitool, which is used by the script to read and set data. Do these errors appear continuously, starting shortly after the script starts? Or do they only start after a delay?

 

[Gaspetaahl]

The first errors appear after the script start, with like 10s between each other. The "Unable to send raw command ..." appear after a few minutes and then continue. These are definatly ipmitool errors.

 

[Kevin Horton]

It may help me figure this out if you set the DEBUG LEVEL to 4 and posted a fresh debug log showing the time time period when those ipmitool messages are showing up in stdout.

 

[Gaspetaahl]

This is stdout:

Code:

./PID_fan_control.pl
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
No data available
Get Device ID command failed
No data available
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xc1): Invalid command
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
^C
Caught SIGINT: setting fan mode to optimal

This is the Debug log with loglevel 4:

 

[Kevin Horton]

The big thing that jumps out is that the debug log shows that the HD fan control portion of the script is not getting HD fan speed info.

Please show the output of:

Code:

ipmitool sdr

What is set in the script for "$hd_fan_header" in the "FAN HEADER" portion? "$hd_fan_header" must be set to a header name that produces a valid HD fan speed.

 

[Gaspetaahl]

Code:

CPU Temp         | 37 degrees C      | ok
System Temp      | 28 degrees C      | ok
Peripheral Temp  | 44 degrees C      | ok
PCH Temp         | no reading        | ns
P1-DIMMA1 Temp   | 30 degrees C      | ok
P1-DIMMA2 Temp   | 31 degrees C      | ok
P1-DIMMB1 Temp   | 31 degrees C      | ok
P1-DIMMB2 Temp   | 32 degrees C      | ok
FAN1             | 750 RPM           | ok
FAN2             | 675 RPM           | ok
FAN3             | 750 RPM           | ok
FAN4             | 675 RPM           | ok
FANA             | 825 RPM           | ok
VCORE            | 1.81 Volts        | ok
VDIMM            | 1.51 Volts        | ok
12V              | 12.10 Volts       | ok
5VCC             | 5.05 Volts        | ok
3.3VCC           | 3.24 Volts        | ok
VBAT             | 3.50 Volts        | ok
-12V             | -11.76 Volts      | nr
5V Dual          | 4.97 Volts        | ok
3V AUX           | 3.21 Volts        | ok
1.2V BMC         | 1.22 Volts        | ok
1.05V PCH        | 1.08 Volts        | ok
Chassis Intru    | 0x00              | ok

These are my fan_header settings. I wasnt quite sure what to put there as I have 1 FANA header and headers FAN1, FAN2, FAN3

Code:

## FAN HEADERS
## these are the fan headers which are used to verify the fan zone is high. FAN1+ are all in Zone 0, FANA is Zone 1.
## cpu_fan_header should be in the cpu_fan_zone
## hd_fan_header should be in the hd_fan_zone
$cpu_fan_header = "FANA";                 # used for printing to standard output for debugging
$hd_fan_header  = "FAN1+";                 # used for printing to standard output for debugging
@hd_fan_list = ("FAN1", "FAN2", "FAN3");  # used for logging to file

 

[Kevin Horton]

Code:

CPU Temp         | 37 degrees C      | ok
System Temp      | 28 degrees C      | ok
Peripheral Temp  | 44 degrees C      | ok
PCH Temp         | no reading        | ns
P1-DIMMA1 Temp   | 30 degrees C      | ok
P1-DIMMA2 Temp   | 31 degrees C      | ok
P1-DIMMB1 Temp   | 31 degrees C      | ok
P1-DIMMB2 Temp   | 32 degrees C      | ok
FAN1             | 750 RPM           | ok
FAN2             | 675 RPM           | ok
FAN3             | 750 RPM           | ok
FAN4             | 675 RPM           | ok
FANA             | 825 RPM           | ok
VCORE            | 1.81 Volts        | ok
VDIMM            | 1.51 Volts        | ok
12V              | 12.10 Volts       | ok
5VCC             | 5.05 Volts        | ok
3.3VCC           | 3.24 Volts        | ok
VBAT             | 3.50 Volts        | ok
-12V             | -11.76 Volts      | nr
5V Dual          | 4.97 Volts        | ok
3V AUX           | 3.21 Volts        | ok
1.2V BMC         | 1.22 Volts        | ok
1.05V PCH        | 1.08 Volts        | ok
Chassis Intru    | 0x00              | ok

These are my fan_header settings. I wasnt quite sure what to put there as I have 1 FANA header and headers FAN1, FAN2, FAN3

Code:

## FAN HEADERS
## these are the fan headers which are used to verify the fan zone is high. FAN1+ are all in Zone 0, FANA is Zone 1.
## cpu_fan_header should be in the cpu_fan_zone
## hd_fan_header should be in the hd_fan_zone
$cpu_fan_header = "FANA";                 # used for printing to standard output for debugging
$hd_fan_header  = "FAN1+";                 # used for printing to standard output for debugging
@hd_fan_list = ("FAN1", "FAN2", "FAN3");  # used for logging to file

Try setting $hd_fan_header to "FAN1" (or FAN2, FAN3, or FAN4).

 

[Gaspetaahl]

Thank you, that fixed the errors! I still have to try some things with the config though because I think its quite loud even with temperatures below 40°C.
Thank you very much for all the troubleshooting.

 

[Kevin Horton]

Thank you, that fixed the errors! I still have to try some things with the config though because I think its quite loud even with temperatures below 40°C.
Thank you very much for all the troubleshooting.

Great! Glad to help.

I'll update the comments in the script to provide better guidance on what to put in the $hd_fan_header and $cpu_fan_header settings.

Is the noise coming from the CPU fan, or the HD fans? If it is from the HD fans, you could increase the target average temperature for the HDs.

If it is from the CPU fan, and you can accept an increase in CPU temperature, you could increase the values in the CPU THRESHOLD TEMPS section, and/or decrease the values in the CPU FAN DUTY LEVELS section (I recommend leaving $fan_duty_high at 100, to allow the script to run the CPU fan at max if needed due to high CPU temperature).

 

[Gaspetaahl]

While you're at it, could you also explain to me and add as a comment what exactly these config option do?

Code:

our $config_num_disks = 4; # number of warmest disks to use for average temperature
our $config_Ta = 38;       # target average temperature for X warmest disks
our $config_Kp = 8/3;
our $config_Ki = 0;
our $config_Kd = 30;
our $config_hd_fan_start = 36; # HD fan duty cycle when script starts

I dont understand what the ones without comments do. And I dont know what the number for duty cycle means

 

[Gaspetaahl]

Im sorry I think I didnt let the script run long enough. I got the errors again:

Code:

Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
^C
Caught SIGINT: setting fan mode to optimal

 

[Kevin Horton]

Im sorry I think I didnt let the script run long enough. I got the errors again:

Code:

Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
No data available
Get Device ID command failed
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request
^C
Caught SIGINT: setting fan mode to optimal

How about you attach a copy of your script to this thread, so I can see how all the variables are defined. I'll be busy for the rest of the day, so I likely won't be able to dig into this until tomorrow at the earliest

 

[Gaspetaahl]

Heres my script

 

[Kevin Horton]

Do you have a config file at /root/nas_fan_control/PID_fan_control_config.ini? If so, please post it as well

 

[Gaspetaahl]

had to rename all the files to .ini because the board wont let me upload it

 

[Kevin Horton]

Nothing is jumping out at me yet. How about a copy of the debug log from the time period when the stdout was throwing the ipmitool errors.

 

[Gaspetaahl]

here

 

[Kevin Horton]

The HD fan speeds shown in the debug log are all over the place. It looks like they are perhaps not responding to the commanded duty cycle. And frequently no HD fan speed is reported at all.

What is the make and model of the HD fans? Are these four wire PWM fans? It might be worth taking a close look at the HD fan connections to be sure all connectors are fully seated.