Fan Scripts for Supermicro Boards Using PID Logic 2020-08-20, previous one was missing a file

[Ericloewe]

Note: PC fans are controlled by PWM, not PCM.

 

[Spearfoot]

Note: PC fans are controlled by PWM, not PCM.

Doooohhh! Thanks.

My point still stands, though -- we can't adjust the PWM on X9-series Supermicro mobos.

 

[Glorious1]

Since modifying the spin scripts to get CPU temperature with sysctl rather than ipmitool, I haven't had occasion to really tax my CPU until recently. When I did, I was surprised to see that reported CPU temps did not go up, so the fans didn't respond. In the past when I've done this there was a clear response.

So I started looking into the CPU temps. Long story short, the temps reported by sysctl on my machine NEVER CHANGE! They are always 47 C for four cores and 49 C for the other four cores. Meanwhile, ipmitool reports about those same temperatures at idle, but maybe 75 C when really working the CPU.

So this script change to sysctl is bad for me. I'm wondering if my machine is anomalous, or other people have this issue as well. I'd be grateful to have some people look into it and report back. Here are two ways to check it out:

One-liners to get temps by both methods:
sysctl dev.cpu | grep temperature
[sudo] ipmitool sdr get "CPU Temp" | grep "Sensor Reading"

A script that gives CPU a workout and reports temps
This script starts by reporting sysctl temps, then starts working the CPU. It reports ipmitool temps every 2 seconds. When it ends, it reports sysctl temps again

To use, it, download, strip off the .txt, put it on your FreeNAS. It needs to be run as superuser, so precede with sudo if needed: loadcpu.sh [#cores] [#seconds], e.g., loadcpu.sh 4 60. If you don't give it cores and seconds, it will default to 1 core and 300 seconds. You should run it on all your cores.

It will kill the workout processes when it stops, either when the time runs out or you hit Ctrl-C. So feel free to kill it early if you get uncomfortable with the temperatures. It is a rough script and may give some unnecessary output, but you'll get the facts, and in my hands it has been safe.

If you want to monitor CPU usage with top or htop while running, you'll need to open a second shell.

 

[Glorious1]

Hello <tap tap> is this thing on? Looking for a little feedback and the sysctl issue.

 

[Stux]

Never seen that. Kernel bug? Which version of FreeNAS are you using?

 

[Glorious1]

Oops, I guess it was a temporary thing. Machine recently rebooted (someone pulled the plug) and now sysctl is telling me things that make sense. Please ignore the above.

Still, though, makes me wonder, if sysctl values can be locked like that, for whatever reason, maybe it's not as reliable a source of information as ipmitool.

 

[Dice]

Hm.
I'm late to the party on fan scripts due to previous server location. That has changed now. I'm thinking - the IPMI solution would be far more preferable to users interested in running FreeNAS as a VM.

 

[Glorious1]

Hm.
I'm late to the party on fan scripts due to previous server location. That has changed now. I'm thinking - the IPMI solution would be far more preferable to users interested in running FreeNAS as a VM.

Just out of curiosity, could you explain why to someone who has no idea what a VM is?

Actually both versions are in the scripts, you just have to comment out the sysctl part and uncomment one line for IPMItool. Assuming you have two zone, in spinpid2.sh you would comment out lines 162-168

Code:

   # Find hottest CPU core
   MAX_CORE_TEMP=0
   for CORE in $(seq 0 $CORES)
   do
	   CORE_TEMP="$(sysctl -n dev.cpu.${CORE}.temperature | awk -F '.' '{print$1}')"
	   if [[ $CORE_TEMP -gt $MAX_CORE_TEMP ]]; then MAX_CORE_TEMP=$CORE_TEMP; fi
   done

and uncomment line 160 (or 159, two similar IPMI commands):

Code:

  #   CPU_TEMP=$($IPMITOOL sensor get "CPU Temp" | awk '/Sensor Reading/ {print $4}')

 

[Stux]

When running FreeNAS inside a Virtual Machine, there are no sysctl core temps.

So, you use IPMI to connect via the network to the BMC to get the temps. You also can’t just use loopback IPMI when inside a VM.

 

[guermantes]

There is are great scripts. Thanks!
Is there a way to make the logging function not overwrite an existing log-file when the scripts start (after reboot or other reason) and rather append to it? Or to rotate the existing log if there is such an existing log when the scripts starts?

 

[Glorious1]

There is are great scripts. Thanks!
Is there a way to make the logging function not overwrite an existing log-file when the scripts start (after reboot or other reason) and rather append to it? Or to rotate the existing log if there is such an existing log when the scripts starts?

I’m on the road and can’t look at it now, but as I recall near the top of the script, where the log file is defined, there is a comment that explains how to do that.

 

[guermantes]

I’m on the road and can’t look at it now, but as I recall near the top of the script, where the log file is defined, there is a comment that explains how to do that.

How utterly daft of me to miss that. Thank you!

EDIT, three weeks later:
@Glorious1, now that I have had time to learn and tweak the script, I just want to say I am so impressed. Thanks again!

Code:

Friday, May 25																 CPU		 New_Fan%  New_RPM_____________________  
		 ada0 ada1 ada2 ada3 ada4 ada5 Tmax Tmean   ERRc	  P	 I	  D TEMP MODE	CPU PER   FANA  FAN1  FAN2  FAN3  FAN4  
17:03:10  *35  *36  *35  *36  *37  *36  ^37  35.83  -0.17  -0.68  0.00  -1.36   33 Full	 50  48	800   600   ---   700   600  
17:09:26  *35  *36  *35  *36  *37  *36  ^37  35.83  -0.17  -0.68  0.00   0.00   40 Full	 40  47	700   500   ---   700   500  
17:15:42  *35  *36  *35  *36  *37  *36  ^37  35.83  -0.17  -0.68  0.00   0.00   41 Full	 40  46	600   500   ---   700   500  
17:21:58  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   1.36   42 Full	 40  47	700   500   ---   700   500  
17:28:14  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  47	600   500   ---   700   500  
17:34:29  *36  *36  *35  *37  *37  *37  ^37  36.33   0.33   1.32  0.00   2.64   41 Full	 40  51	600   600   ---   700   600  
17:40:45  *36  *36  *35  *37  *37  *37  ^37  36.33   0.33   1.32  0.00   0.00   41 Full	 40  52	600   600   ---   800   600  
17:47:02  *36  *36  *35  *37  *37  *37  ^37  36.33   0.33   1.32  0.00   0.00   39 Full	 40  53	600   600   ---   800   700  
17:53:17  *36  *36  *35  *37  *37  *37  ^37  36.33   0.33   1.32  0.00   0.00   41 Full	 40  54	600   700   ---   800   700  
17:59:33  *36  *36  *35  *37  *37  *37  ^37  36.33   0.33   1.32  0.00   0.00   40 Full	 40  55	600   700   ---   800   700  
18:05:48  *36  *36  *35  *37  *37  *36  ^37  36.17   0.17   0.68  0.00  -1.28   40 Full	 40  54	600   700   ---   800   700  
18:12:04  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00  -1.36   41 Full	 40  53	700   600   ---   800   700  
18:18:20  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
18:24:36  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  53	600   600   ---   800   700  
18:30:52  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  53	600   600   ---   800   700  
18:37:07  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
18:43:23  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
18:49:38  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
18:55:54  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
19:02:09  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  53	700   600   ---   800   700  
19:08:25  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   42 Full	 40  53	600   600   ---   800   700  
19:14:41  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  53	600   600   ---   800   700  
19:20:57  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   41 Full	 40  53	600   600   ---   800   700  
19:27:12  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700  
19:33:29  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	700   600   ---   800   700  
19:39:45  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   39 Full	 40  53	600   600   ---   800   700  
19:46:01  *35  *36  *35  *37  *37  *36  ^37  36.00   0.00   0.00  0.00   0.00   40 Full	 40  53	600   600   ---   800   700

 

[scrubolio]

Hello! thank you for the amazing scripts. I finally got it tweaked for my use and set it up as a post-init script but it doesn't seem to run on boot up.

Under CLI I used "ps -auxww" to see if spinpid2.sh was running and there was nothing until I manually ran it.

Do I need to change its permissions or something? I'm running FN 11.1

Thanks!

 

[guermantes]

Hello! thank you for the amazing scripts. I finally got it tweaked for my use and set it up as a post-init script but it doesn't seem to run on boot up.

Under CLI I used "ps -auxww" to see if spinpid2.sh was running and there was nothing until I manually ran it.

Do I need to change its permissions or something? I'm running FN 11.1

Thanks!

FWIW, maybe it is an issue due to the location where you saved the script? I saved my scripts in a "spinpid" folder in the root of my dataset (TANK) and it runs post-init without tweaking permissions.

 

[Glorious1]

Hello! thank you for the amazing scripts. I finally got it tweaked for my use and set it up as a post-init script but it doesn't seem to run on boot up.

Under CLI I used "ps -auxww" to see if spinpid2.sh was running and there was nothing until I manually ran it.

Do I need to change its permissions or something? I'm running FN 11.1

Thanks!

I have my script just in a folder in my user folder, works fine. But I think you might have to give the full path to the script when you enter it in the post init tasks. Also, you have to make it executable, but if it works manually I guess you already did that.

By the way, you can run ps -aux | grep spinpid to see if it's running. But you will have a log where you set it to be to be and can see in there what it's doing.

 

[glauco]

I have two fans to cool down my 6 hard disk drives.
One pushes air in and the other one pulls it out.
They have different rotation speeds: 450-2000 and 300-1500.
I've set thresholds with ipmitool accordingly.
Now I'm wondering what values should I enter as variables RPM_PER_30 and RPM_PER_MAX?
Thanks.

 

[Glorious1]

I have two fans to cool down my 6 hard disk drives.
One pushes air in and the other one pulls it out.
They have different rotation speeds: 450-2000 and 300-1500.
I've set thresholds with ipmitool accordingly.
Now I'm wondering what values should I enter as variables RPM_PER_30 and RPM_PER_MAX?
Thanks.

I'm assuming the two fans are in a single zone? If a single zone, you are (or should be) getting RPM data for only one of them. The other one(s) should only have 3 pins connected in the extender cable from the Y.

Just run spintest.sh and use the values reported for 30% and 100% duty cycle. Those will be for the fan with 4 pins connected.

 

[dak180]

I just found this thread where someone asked ASRock for the raw commands. See the May 5 2015 post for the reply.
They only gave him commands for setting "PWM value" which I think is duty cycle, not for reading. One command works for all fans.

Code:

ipmitool raw 0x3a 0x01 0xAA 0xBB 0xCC 0xDD 0xEE 0xFF 0xGG 0xHH

AA: CPU_FAN1(PWM Value)
BB: Reserved(Set to 0)
CC: REAR_FAN1(PWM Value)
DD: Reserved(Set to 0)
EE: FRNT_FAN1(PWM Value)
FF: Reserved(Set to 0)
GG: Reserved(Set to 0)
HH: Reserved(Set to 0)

PWM Value:	00h -> smart fan mode
01h~64h -> manual fan mode(1%~100%)

So I think if you do
ipmitool raw 0x3a 0x01 0x64 0x00 0x64 0x00 0x64 0x00 0x00 0x00
it should run the fans to full speed. If you then change the 64's to 00's it should reset them to "smart fan mode". Whatever that is.

Here's a nice page showing a bit differently. The system shown here has more fan headers, so the ones "reserved" above are used to control additional fans. Unfortunately again only control, no reading. Actually, you don't absolutely need to read duty cycle, just set it.

Which is your system like?

This page suggests you can read RPMs with the command ipmi-sensors. Can you verify that?

If these commands work for you, it should be possible to adapt one of the scripts. If I'm understanding correctly, each header is basically its own zone and can be controlled independently, which is interesting. It's quite a different setup.

Having an ASRock Rack MB myself, I wrote a fan control script for it inspired by the one you have done here.

It uses a separate config file for settings. Currently it does not control the cpu fan (I find the auto control works well for that), it does recognize three other types of fans: intake, exhaust and HBA. Also, which temp sensors are used can be customized.

 

[glauco]

I'm assuming the two fans are in a single zone? If a single zone, you are (or should be) getting RPM data for only one of them. The other one(s) should only have 3 pins connected in the extender cable from the Y.

Just run spintest.sh and use the values reported for 30% and 100% duty cycle. Those will be for the fan with 4 pins connected.

Thank you for your prompt response!
Yes, they're in a single hard drive zone.
Unfortunately, they're both 4-pin fans and they're currently connected to the FAN2 and FAN4 mobo headers.
Is there something I can do if I don't want to buy a 3-pin replacement fan?

 

[Glorious1]

So with both fans connected with the 4th pin, I'm curious what kind of readout you're getting from the spincheck.sh script. Can you run that for a bit and paste the log?

I'm going to have to review the script and see how it should react if there are different fans reporting speeds in a zone, and how that interacts with the two variables you mentioned - I just don't remember at the moment.