Script: Hybrid CPU & HD Fan Zone Controller

Bill Cowger

Cadet
Joined
Aug 28, 2015
Messages
8
I do have instructions in the first post. Have you seen those?
Thanks Stux, I got everything running now. Awesome script everything is running great. Thanks for all the hard work. By the way I pretty much modeled or copied your build.
 

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
very nice script, I haven't tested it yet but will give it a try

Just a few questions:
Regarding the fan speeds, how much tolerance has it? I have 3 identical fans all delivering different rpms on the same setting.
1x 2800 1x2500 and 1x2600

As those fans cool the whole chassis I would need to check CPU and HDD temperature and make a change if one of them go over a specific limit, so far I just made a little script that pulls the hdd temps and gets the highest and regulates the fans according to the temp.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Probably +-200RPM at least. But don't forget that the fans may also respond differently to the same PWM output.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
The rpm value is to determine if the fans are at max when they shouldn't be. There is a common failure mode where the BMC will crash and the fans won't be at max when they should be, or they will be at max when they shouldn't be.

That's what the rpm is for.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
The rpm value is to determine if the fans are at max when they shouldn't be. There is a common failure mode where the BMC will crash and the fans won't be at max when they should be, or they will be at max when they shouldn't be.

That's what the rpm is for.

So, I checked a bit further for you, the script only checks the CPU header and the HD header fans (these are configurable), and even if you have 2 or 3 fans connected to that header, only one will be providing speed info.

In answer to your question, the first thing the script does is multiply the max rpm by 0.8. It then assumes if the read RPM Is above 80% of your configured value it's max and if it's less it's not.

80% of 2800 is 2240. And the tolerance is 20% which is quite a lot ;)
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Found something interesting today...

Apparently, some ASRock boards support individual PWM fan control... (including the infamous Avoton boards)

https://blog.chaospixel.com/linux/2016/09/Fan-control-on-Asrock-C2750D4I-C2550D4I-board.html
https://forums.lime-technology.com/topic/37560-fan-speed-control-trouble-on-asrock-e3c226d2i/

This script could be modified to support the Avoton boards etc. You'd basiaclly need to set all fans to their last requested speeds every time you wanted to change either the HD or CPU fan speeds.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Preface: As I have an X9 board, I lack the ability to control the fans with the same level of granularity that X10 and X11 owners enjoy (and the script fails when attempting to issue IPMI commands).

Q1: Is there a way to "neuter" this script to function as described below (ideally) or in a similar manner? I'm far from proficient with editing scripts, so if this is possible, if you wouldn't mind pointing me in the right direction or doing some hand holding it would be appreciated!
Notes: (a) I tried to use Kevin Horton's perl script and it executed via SSH without error, but didn't change the Fan Speed Mode, (b) I've provided my system software and hardware configuration below for reference.

Q2: With an X9, I assume there is no way to set Zone 0 and Zone 1 to different Fan Modes, right? Example = Zone 0 (CPU/System) to "Optimal" and Zone 1 (Peripheral) to "Heavy IO".


Desired Functionality - Summary
  1. If HDD temp < 38°C, use "Optimal" Fan Speed Mode (Peripheral zone, FANA @ 30%)
  2. If HDD temp >= 38°C and < 40°C, use "Heavy IO" Fan Speed Mode (Peripheral zone, FANA @ 75%)
  3. If HDD temp >= 40°C, use "Full" Fan Speed Mode (Peripheral zone, FANA @ 100%)
Desired Functionality - Additional Detail
  • Possibly only items 2 & 3 above are needed, adjusting thresholds accordingly, but I believe all 3 would be ideal as my testing shows Tmax = 36°C / 39°C / 42°C (Full / Heavy I/O / Optimal) which would allow more precise control.
  • If you are scratching your head wondering how I might ever hit sub-38°C with Optimal (making note of the below measurements), I should note that it would be possible with no or little load on the server, or with an ambient lower than 78°F (as is the case now during the summer).
Code:
Fan Mode	Tavg	Tmin	Tmax
Full		34.4°C  33.0°C  36.0°C
Optimal	 39.8°C  38.0°C  42.0°C
Heavy I/O   36.6°C  35.0°C  39.0°C

Software Configuration
  • FreeNAS-9.10.2-U4 on VMware ESXi 6.0 U3
Hardware Configuration (All Details Included in Signature)
  • SuperMicro X9DRi-LN4F+
  • Rear Exhaust Fans: 2 x FAN-0125L4 (Connected to FAN5/6)
  • Chassis Fans: 3 x FAN-0126L4 Chassis Middle Fan (Connected to FANA by splitter)
  • CPU Fans: 2 x Noctua NH-D9L (connected to FAN1/2)
Thank you in advance for your feedback.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Code:
sub get_fan_mode_code
{
	my ( $fan_mode ) = @_;
	my $m;

	if(	 $fan_mode eq	'standard' )	{ $m = 0; }
	elsif(	$fan_mode eq	'full' )	 { $m = 1; }
	elsif(	$fan_mode eq	'optimal' )	 { $m = 2; }
	elsif(	$fan_mode eq	'heavyio' )	{ $m = 4; }
	else					 { die "illegal fan mode: $fan_mode\n" }

	dprint( 3, "fanmode: $fan_mode = $m\n");

	return $m;
}

sub set_fan_mode
{
	my ($fan_mode) = @_;
	my $mode = get_fan_mode_code( $fan_mode );

	dprint( 1, "Setting fan mode to $mode ($fan_mode)\n");
	`$ipmitool raw 0x30 0x45 0x01 $mode`;

	sleep 5;	#need to give the BMC some breathing room

	return;
}  


'standard' = 0
'full' = 1
'optimal' = 2
'heavyio' = 4

can you confirm that

ipmitool raw 0x30 0x45 0x01 <mode>

functions, where <mode> is 0, 1, 2 or 4. Each of those should change the fan mode, which you should be able to verify from the IPMI. Also, I seem to remember that Kevin's earliest scripts worked by changing fan mode, before the PWM stuff was discovered.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
@svtkobra7 take a look at some of the earlier fan controller scripts.

Say, this one, #60

Which purports to be x9 compatible and varies between Optimal, Standard and Full. Features BMC reset and 3 temperature control points.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Thanks for the fast reply - you rock!

can you confirm that

ipmitool raw 0x30 0x45 0x01 <mode>

functions, where <mode> is 0, 1, 2 or 4.

Confirmed. Since I'm using FreeNAS on ESXi I do have to use the remote ipmitool command (as per below). Are you taking me somewhere (if so, I don't quite follow yet)?

ipmitool -I lanplus -H <IP> -U <BMC USERNAME> -P <BMC PASSWORD> raw 0x30 0x45 0x01 <MODE>

Also, I seem to remember that Kevin's earliest scripts worked by changing fan mode, before the PWM stuff was discovered.

Bingo! I believe you are referring to this thread: https://forums.freenas.org/index.ph...in-response-to-hard-drive-temperatures.41294/

Correct me if I'm wrong, but with an X9, this is the best I can do (changing fan modes in response to temp). After a few tries, I got it to "execute" without error, but it didn't actually change the fan mode. If you reference that script, you will note the ipmi raw command is in the same format as the one which you just asked me to test. I of course changed it for remote.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Yeah, you'll need to do that, I figured you should be able to handle that change to the script ;)

As I mentioned, this script almost looks like what you want.

https://forums.freenas.org/index.ph...d-drive-temperatures.41294/page-3#post-283776

Add an $ipmitool variable. Replace the ipmitool hard coded instances, with $ipmitool (as per my script), then change the $ipmitool variable to include your remote ipmitool config stuffs.

Comment out the duty cycle stuff, and leave the mode,

If you want add an additional case to add HeavyIO between Full and Standard.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Note: To avoid any confusion, this post is not regarding Stux's script, rather is a bit off-topic and is specific to a much simpler script which just changes fan modes based upon HDD temperature authored by Kevin Horton. It would be helpful to X9 users only, as X10 users have much better options. Further, it has been customized for an ESXi installation (remote IPMI).

I feel it prudent to make that disclaimer to avoid any confusion.


@Stux I love you man (don't worry not like that) :):):) Read on ... Read on ...

Yeah, you'll need to do that, I figured you should be able to handle that change to the script ;)
I may be a wee bit beyond that now! Ever so slightly ...

Add an $ipmitool variable.
Done - good call.
Code:
Line 20
+	$ipmitool = "/usr/local/bin/ipmitool -I lanplus -H <IP> -U <USER> -f /root/.ipmi ";
All Instances (Replace)
-	ipmitool
+	$ipmitool

So the script wasn't working (as per my earlier mention that it executed but didn't do anything) and while it took me a moment, I figured out why:
Code:
Line 32
$command = "/usr/local/sbin/smartctl -A $hd_designator$item | grep Temp";
Line 40
$temp = "$vals[-1]\n";

That command was printing XX/YY to the log file where XX = min temp and YY = max temp as reported by SMART, therefore nothing would work, so I changed that:
Code:
Line 40
-	$temp = "$vals[-1]\n";
+	$temp = "$vals[9]\n";

Further, I changed the script to align with my specific conditionals: 40°C or higher = Full / 38°C-40°C = Heavy IO / 37°C or lower = Optimal
Code:
Lines 46 - 62
if ($max_temp > 39){
  `$ipmitool raw 0x30 0x45 0x01 0x01`
  # at least one hard drive is 40°C or higher
  # set fan speed control to Full
  }
elsif ($max_temp > 37){
  # at least one hard drive is 38°C or higher
  # set fan speed to Heavy IO
`$ipmitool raw 0x30 0x45 0x01 0x04`
}
else {
  # all hard drive temperatures are 37°C or cooler
  # set fan speed control to Optimal
  `$ipmitool raw 0x30 0x45 0x01 0x02`
}

Then, I run the script via SSH. I hear the fans spin down (Full => Heavy IO). I spent a few moments in awe starring at IPMI View reporting the change. Eventually I snapped out of it and thought, let's drop the A/C a wee bit, to see if it cycles to Optimal. It did, so I pinched myself to ensure I wasn't dreaming.

Finally I add the cron job and it took me a few more moments to determine that the command needed to be entered as "perl /mnt/Tank1/System/Scripts/fancontrol/fancontrol.pl".

So I spent many moments on this, but by all accounts it appears to actually freakin work. Again, thanks for giving me just enough info to get me started on my way with this.

I've uploaded the script here - while I know you didn't write it and don't have much interest in it as it is for lowly X9 users - I would love a second pair of real eyes to see if all looks to be in order. It scares me a tad that it appears to actually be working.

Also, I think I've learned enough for today, but it would be nice to print the fan mode to the log file using "ipmitool raw 0x30 0x45 0x0" with the name, with an example below - is that easily doable?
Code:
2017-06-09 03:00:00 - 40  Full
2017-06-09 04:00:00 - 38  Heavy IO
2017-06-09 04:00:00 - 37  Optimal
 

Attachments

  • fancontrol.zip
    1 KB · Views: 832

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
untested code.

Code:
#this function will return a fan mode string if you pass it a fan mode code.
sub get_fan_mode_string
{
	my ( $fan_mode_code ) = @_;
	my $m;

	if(	 $fan_mode_code == 0 )	{ $m = 'standard'; }
	elsif(	$fan_mode_code == 1 )	 { $m = 'full'; }
	elsif(	$fan_mode_code == 2 )	 { $m = 'optimal'; }
	elsif(	$fan_mode_code == 4 )	{ $m = 'heavyio'; }
	else					 { die "illegal fan mode: $fan_mode_code\n" }

	dprint( 3, "fanmode: $fan_mode_code = $m\n");

	return $m;
}

#2 should be Optimal, ideally you'd actually call a function to get a fan mode from IPMI.
my ($fan_mode_code) = 2; 

# now turn the fan mode into a string...
my $mode_string = get_fan_mode_string( $fan_mode_code );
 

rovan

Dabbler
Joined
Sep 30, 2013
Messages
33
Hey @Stux, do you think it would be possible to add another hdd zone ?
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Supermicro boards only support two zones.
 

Hazimil

Contributor
Joined
May 26, 2014
Messages
172
## FAN SPEEDS ## You need to determine the actual max fan speeds that are achieved by the fans ## Connected to the cpu_fan_header and the hd_fan_header. ## These values are used to verify high/low fan speeds and trigger a BMC reset if necessary. $cpu_max_fan_speed = 1700; $hd_max_fan_speed = 1400;

Hi, I'm about to start testing this script on my box, and wanted to be clear on what you meant by fan speed, are you talking about RPM?

My NAS uses:

3x Middle 8cm (5000 rpm) PWM Fans (for the HDDs)
2x Case 8cm (5000 rpm) PWM Fans
1 x Supermicro 4U Active CPU Heat Sink Socket LGA2011 Square and Narrow ILMs (SNK-P0050AP4) (3800 rpm)

So would I need to change the settings to:

$cpu_max_fan_speed = 3800
$hd_max_fan_speed = 5000

Obviously I want to be sure before I start making changes as I don't to kill me new box!

Thanks.

Yours
Jonathan
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Seems right. It is the actual max rpm achieved by *your* fans. So it's possible you might achieve a bit higher than spec.
 

JP Powers

Cadet
Joined
Jul 6, 2017
Messages
5
Hi folks,

So, I want to apologize up front as I'm going to go a bit off topic here, but it is directly related to this script so I hope it's ok.

I'm looking to figure out a way to implement this script on my Ubuntu Server. I'm having a heck of a time getting my HDD temps in check on my X11SSH-F board and I'm hoping this script can help me. However, I'm pretty sure the method this and Kevin Horton's script pull the HDD temps won't work "out of the box" on linux. I'm no coder so I'm not quite sure where to start going about adjusting the script to work...

I'm running 8 disks and currently run this command to spot check temps:

~$ sudo /usr/bin/hddtemp /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
/dev/sda: ST6000VN0041-2EL11C: 46°C
/dev/sdb: ST6000VN0041-2EL11C: 44°C
/dev/sdc: ST6000VN0041-2EL11C: 49°C
/dev/sdd: ST6000VN0041-2EL11C: 49°C
/dev/sde: ST6000VN0041-2EL11C: 52°C
/dev/sdf: ST6000VN0041-2EL11C: 51°C
/dev/sdg: ST6000VN0041-2EL11C: 45°C
/dev/sdh: ST6000VN0041-2EL11C: 47°C


As you can see those temps are quite high, hence why I'm trying to figure out how to get this working.

I'd definitely appreciate any help from someone who knows what they're doing getting this working, but even some direction may be enough. I feel like the "proper" way to do this is would be merging get_hd_list, get_hd_temp, and get_hd_temps into one sub, utilizing the hddtemp command directly instead of grep'ing smartctl? Is there more to the script that would need to be adjusted than those subs? Does using hddtemp even make sense? I feel like it would simplify getting the temps quite a bit.

Thanks for any help you can provide!
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Basically,you just need to write a sub which returns the max hd temp.

You can skip the get_hd_list of your get had temp function doesn't need it.
 

glauco

Guru
Joined
Jan 30, 2017
Messages
526
Hello everybody.
I'm still very new to the FreeNAS world.
I'm setting up a FreeNAS box in my living room so I want it to be whisper quiet and thought a script such as Stux's or Kevin Horton's mod could help tremendously, but unfortunately neither of the two works as expected on my machine.
Could it be because I'm using a Supermicro X11 board (X11SSM-F)?
This is what I get when I run Stux's script at debug level 1:

Code:
root@FreeNAS:~ # ./hybrid_fan_controller.pl
2017-07-07 12:40:56: CPU Temp: 36.0 <= 36, CPU Fan going low.
2017-07-07 12:40:57: Maximum HD Temperature: 37
2017-07-07 12:40:57: Drives are warm, going to 80%
2017-07-07 12:41:09: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:41:26: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:41:31: Resetting BMC
No data available
Get Device ID command failed
No data available
Get Device ID command failed: 0xc1 Invalid command
Unable to open SDR for reading
2017-07-07 12:42:26: HD Fan speed: No reading
2017-07-07 12:42:38: CPU Fan speed: No reading
2017-07-07 12:42:38: HD Fan speed: No reading
2017-07-07 12:42:50: CPU Fan speed: No reading
2017-07-07 12:42:50: HD Fan speed: No reading
2017-07-07 12:43:02: CPU Fan speed: No reading
2017-07-07 12:43:02: HD Fan speed: No reading
2017-07-07 12:43:14: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:43:31: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:43:36: Resetting BMC
No data available
Get Device ID command failed
No data available
No data available
No valid response received
Error obtaining SDR info: Invalid command
Unable to open SDR for reading
2017-07-07 12:44:33: HD Fan speed: No reading
2017-07-07 12:44:34: Maximum HD Temperature: 37
2017-07-07 12:44:45: CPU Fan speed: No reading
2017-07-07 12:44:46: HD Fan speed: No reading
2017-07-07 12:44:57: CPU Fan speed: No reading
2017-07-07 12:44:58: HD Fan speed: No reading
2017-07-07 12:45:09: CPU Fan speed: 12800 RPM, is nonsensical
2017-07-07 12:45:10: HD Fan speed: 12800 RPM, is nonsensical
2017-07-07 12:45:18: CPU Temp: 46.0 >= 46, CPU Fan going med.
2017-07-07 12:45:44: CPU Temp: 36.0 <= 36, CPU Fan going low.
2017-07-07 12:45:56: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:46:13: CPU fan speed should be low, but 2000 > 1600.
2017-07-07 12:46:18: Resetting BMC

If I understand correctly, the BMC is ignoring the ipmitools commands to lower fan speeds (CPU fan speed should be low, but 2000 > 1600) even after being reset.
Do you have any ideas how I could fix this? Thank you!
 
Top