Script to control fan speed in response to hard drive temperatures

[Dice]

..ehh, anyone happen to know where the latest version of the script is at?

 

[Kevin Horton]

..ehh, anyone happen to know where the latest version of the script is at?

I moved on from this script. It worked well enough, but it made fairly gross changes in fan speed, and didn't respond until the warmest HD had changed by 1°C. @Glorious1 came up with the idea to control using a PID controller, based on the average disk temperature, which allows much smaller increments of fan speed, and the script responds to very small changes in average disk temperature.

I'm currently using a perl script, with a PID control loop. The script is basically @Stux's script, but with a PID controller replacing the discrete fan speed contol that he uses. I'm using the script on two systems, one with a Norco RPC-4224, and one with a Fractal Design Node 804. It works extremely well on both systems.

 

[Glorious1]

..ehh, anyone happen to know where the latest version of the script is at?

My scripts are in the Resources section.

 

[Dice]

Thanks.
In conclusion, if I understand correctly the 'maintained and developed' goto-script for variable fan speed is now to be found here:
https://forums.freenas.org/index.php?resources/fan-scripts-for-supermicro-boards-using-pid-logic.24/

 

[Stux]

And here

https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/

And Kevin's version with PID

https://forums.freenas.org/index.php?threads/pid-fan-controller-perl-script.50908/#post-351956

 

[Dice]

Just realized the ipmitool does not work while FreeNAS is running ontop of ESXi. While probably expected, none the less a tad disappointing trade off.

 

[Stux]

IPMItool can work over IP to IPMI. There is a post in my fan thread about it

 

[Incogito]

I second this. I have been running a slightly modified version of the script with ESXi using IPMI over IP for over a month now.
Preferably, connect IPMI on a closed-loop, to mitigate security issues, unless you know what you are doing.

Use as follow:
IPMITOOL="/usr/local/bin/ipmitool -H <IP> -U <User> -P <Password>"

 

[Kevin Horton]

If the VM were to crash, that would lead to loss of fan control if the script was running in the VM.

Wouldn't it be more reliable to run the fan control script directly on the host, rather than in a VM?

 

[Incogito]

If the VM were to crash, that would lead to loss of fan control if the script was running in the VM.

Wouldn't it be more reliable to run the fan control script directly on the host, rather than in a VM?

You are quite correct, that was also my concern.
Using a two-zone Supermicro motherboard, I set the fan control type to "Heavy IO", and let the script control only zone 1 -- which is bound to HDD fans only.
That way CPU is safe, and worst-case scenario, HDD temperature raise beyond 40°C for a little while, until the situation is fixed.

That being said your idea of running the script directly on host is interesting. Thank you

EDIT: Here's a tutorial on compiling IPMITool with static libraries: https://coderwall.com/p/5cqj0g/esxi-ipmitool-works-on-any-linux-system-as-well

I haven't tested it yet, but the process described should be the same regardless of environment.

 

[Kevin Horton]

That being said your idea of running the script directly on host is interesting.

Another approach would be to run the fan control script on FreeNAS in the VM, but to also have a very simple script running on the host. The script on the host would periodically check HDD temps, and if any temp exceeded some high threshold, set HDD fans to 100% and send you an email.

 

[Dice]

Thnx!

I found this documentation to do a quick check that stuff works:
https://docs.oracle.com/cd/E19464-01/820-6850-11/IPMItool.html#50602039_74914

Particularly ipmitool -I lanplus -H <ipaddress> -U <user> -P <password> sdr elist full that displays the typical sensor data:

Code:

CPU Temp		 | 01h | ok  |  3.1 | 29 degrees C
PCH Temp		 | 0Ah | ok  |  7.3 | 35 degrees C
System Temp	  | 0Bh | ok  |  7.1 | 33 degrees C
Peripheral Temp  | 0Ch | ok  |  7.2 | 35 degrees C
VcpuVRM Temp	 | 10h | ok  |  8.1 | 38 degrees C
DIMMA1 Temp	  | B0h | ok  | 32.64 | 31 degrees C
DIMMA2 Temp	  | B1h | ok  | 32.65 | 32 degrees C
DIMMB1 Temp	  | B4h | ok  | 32.68 | 31 degrees C
DIMMB2 Temp	  | B5h | ok  | 32.69 | 31 degrees C
FAN1			 | 41h | ok  | 29.1 | 1500 RPM
FAN2			 | 42h | ns  | 29.2 | No Reading
FAN3			 | 43h | ns  | 29.3 | No Reading
FAN4			 | 44h | ns  | 29.4 | No Reading
FANA			 | 45h | ns  | 29.5 | No Reading
12V			  | 30h | ok  |  7.17 | 12.19 Volts
5VCC			 | 31h | ok  |  7.33 | 4.97 Volts
3.3VCC		   | 32h | ok  |  7.32 | 3.38 Volts
VBAT			 | 33h | ok  |  7.18 | 3.08 Volts
VCPU			 | 34h | ok  |  3.2 | 1.15 Volts
VDIMMAB		  | 35h | ok  | 32.1 | 1.19 Volts
5VSB			 | 38h | ok  |  7.15 | 5 Volts
3.3VSB		   | 39h | ok  |  7.12 | 3.25 Volts
VBMC 1.2V		| 3Bh | ok  |  7.20 | 1.20 Volts
VPCH 1.0V		| 3Ch | ok  |  7.21 | 1.01 Volts
Chassis Intru	| AAh | ok  | 23.1 |

There is hope :p
Now just needing to ...ehh, grasp enough of the scripts to make the edits at appropriate places (still somewhat a jungle with sub-minimal skills for reading code).

@Incogito which script did you modify?

 

[Dice]

Preferably, connect IPMI on a closed-loop, to mitigate security issues, unless you know what you are doing.

What do you mean by closed-loop?

 

[Ericloewe]

What's that noise? Is it a trash bag tied to a pole on a windy day? Is it a moron rubbing two pieces of styrofoam to annoy people? No, it's Pedantry Man!

Preferably, connect IPMI on a closed-loop

You presumably mean a subnet that is dedicated to management tasks, typically achieved with a dedicated VLAN.

A closed-loop system is a system where one or more variables are fed back as inputs to that system, through a net K, which is typically a control algorithm.

 

[Incogito]

What's that noise? Is it a trash bag tied to a pole on a windy day? Is it a moron rubbing two pieces of styrofoam to annoy people? No, it's Pedantry Man!


You presumably mean a subnet that is dedicated to management tasks, typically achieved with a dedicated VLAN.

A closed-loop system is a system where one or more variables are fed back as inputs to that system, through a net K, which is typically a control algorithm.

That, exactly. Accuracy is important, especially when dealing with complex scenarios.
This indeed has nothing to do with a loop. A more accurate definition may have been an isolated network.

Either use a managed switch with an associated VLAN, specifically for Freenas and IPMI -- as suggested by Ericloewe, or add another network card in your server -- assuming you host pfSense or other router appliance directly on ESXi.

If you allow other machines on the subnet, you may expose yourself to eavesdropping. IPMI is a rather insecure protocol as it stands.

@Incogito which script did you modify?

I edited the original spinpid.sh script by Glorious1.
I have uploaded it to pastebin https://pastebin.com/BWhUwRzT

If you want to run it, make sure you understand what it does.
From the top of my mind, I adjusted it to regulate zone 1 only, parse the Seagate S.M.A.R.T status correctly, and perhaps some adjusting to log output.
You can run a diff on it if need be.

 

[Incogito]

Another approach would be to run the fan control script on FreeNAS in the VM, but to also have a very simple script running on the host. The script on the host would periodically check HDD temps, and if any temp exceeded some high threshold, set HDD fans to 100% and send you an email.

If one goes through compiling IPMITool for ESXi, what advantage is there to this solution ?
Another added benefit of your former idea is that it should allow IPMITool to run directly through the BMC controller, instead of using the LAN interface.

 

[Kevin Horton]

If one goes through compiling IPMITool for ESXi, what advantage is there to this solution ?
Another added benefit of your former idea is that it should allow IPMITool to run directly through the BMC controller, instead of using the LAN interface.

If you can get IPMITool running in ESXi, then it would be simpler to just run the script in ESXi.

I haven't had my script ever lockup on me, but it is hard to be certain that this could not occur. If you were worried about the risk of having the script lock up, leaving the hard drives without adequate cooling, then that risk could be mitigated by having a completely separate monitor process running a very simple script with the only task being to command the fans to full speed if the HDD temps were too high. The temperature threshold for this backup fan control would be set sufficiently warmer than the highest temperature that could be achieved by the regular control script so that the backup script would only trigger if the main control script had failed to function.

 

[Stux]

And the scripts I wrote fail high.

For something to go wrong the perl process would have to crash, or FreeNAS hang.

Which is why a 5 minute watchdog in the ESXi is not a bad idea.

Another thing to consider, the scripts leave the IPMI at max, thus is great as it means when you reboot the hardware the fans are at max until the script launches, except in the ESXi case, where it's possible for FreeNAS to reboot etc, yet the IPMI doesn't get a reboot signal....

Ideally if FreeNAS VM is not running then fans should be set to max. IIRC setting max will spin fans up, even if it's already set to max.

 

[Incogito]

If you can get IPMITool running in ESXi, then it would be simpler to just run the script in ESXi.

I haven't had my script ever lockup on me, but it is hard to be certain that this could not occur. If you were worried about the risk of having the script lock up, leaving the hard drives without adequate cooling, then that risk could be mitigated by having a completely separate monitor process running a very simple script with the only task being to command the fans to full speed if the HDD temps were too high. The temperature threshold for this backup fan control would be set sufficiently warmer than the highest temperature that could be achieved by the regular control script so that the backup script would only trigger if the main control script had failed to function.

[...] Irrelevant [...]

Edit: sorry I misread your post yesterday, for some reason I read "if you can't get IPMITool", please disregard

 

[Glorious1]

I edited the original spincheck.sh script by Glorious1.
I have uploaded it to pastebin https://pastebin.com/BWhUwRzT

If you want to run it, make sure you understand what it does.
From the top of my mind, I adjusted it to regulate zone 1 only, parse the Seagate S.M.A.R.T status correctly, and perhaps some adjusting to log output.

FYI, spincheck.sh does not control the fans, it only reads fan and temperature data. You probably mean spinpid.sh.