Fan Scripts for Supermicro Boards Using PID Logic 2020-08-20, previous one was missing a file

[Glorious1]

I think -O only means retail packaging vs. bulk.
Here Supermicro indicates only three 4-pin headers. Maybe the 4th pin is there, but apparently it doesn't do anything.
http://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-2C-TLN2F.cfm
I can't locate the manual for that board.

 

[boynep]

It is very strange indeed. Originally I though I might have done something wrong on my part when running the script and changed the configuration of the fan and realised when on full mode fan 3 and fan 4 were running at full speed others at minimal.

It is definitely numbered I looked through each fans and it is PWM headers as well. When running spincheck it works for all the fans.

So, I connected the fan differently.

FAN1 -> 2 Noctua ppc 3000 rpm PWM at the front cooling HDD cages as intake.
FAN2 -> 2 Noctua ppc 3000 rpm PWM one intake on side one exhaust at the back.
FAN3 -> One standard 1000 rpm 3 pin fan as intake and one 600rpm 180mm 3 pin as case exhaust top of the case.
FAN4 -> One Noctua NF-A6-25 running at 3100rpm sitting on top of cpu heat sink.

Now, due to your script my box is very quiet. And when needed there are fans that can go upto 3000rpm to cool the hdd and cpu.

Very happy.

Current Reading:


Name

Status

Reading

CPU Temp Normal 42 degrees C
System Temp Normal 31 degrees C
Peripheral Temp Normal 35 degrees C
MB_10G Temp Normal 44 degrees C
DIMMA1 Temp Normal 36 degrees C
DIMMA2 Temp N/A Not Present!
DIMMB1 Temp N/A Not Present!
DIMMB2 Temp N/A Not Present!
FAN1 Normal 600 R.P.M
FAN2 Normal 700 R.P.M
FAN3 Normal 1000 R.P.M
FAN4 Normal 3100 R.P.M
VCCP Normal 1.84 Volts
VDIMM Normal 1.191 Volts
12V Normal 12.32 Volts
5VCC Normal 4.896 Volts
3.3VCC Normal 3.333 Volts
VBAT Normal 3.197 Volts
5V Dual Normal 5.054 Volts
3.3V AUX Normal 3.282 Volts
Chassis Intru OK

 

[zvans18]

@zvans18 , did switching to FAN1 fix your problems?

No it didn't. Sorry, I changed something, let it run for a while, and then decided to change something else.

Board is Supermicro X10SLL+-F (rest in sig)
Firmware Revision : 01.92
BIOS Version : 3.0

Currently the fans on headers are:
FANA is still the H7
FAN1 is front iPPC 2000
FAN4 is rear iPPC 2000
The other fans are on molex power to be switched for PWM eventually since I thought different RPMs would confuse it based on the RPM_PER_30 value being different

Settings

Code:

LOG=/mnt/Z1/z/spinscripts/spinpid2.log
CPU_LOG=/mnt/Z1/z/spinscripts/cpu.log

ZONE_CPU=1
ZONE_PER=0

DUTY_PER_MIN=30
DUTY_PER_MAX=100
DUTY_CPU_MIN=30
DUTY_CPU_MAX=100

RPM_CPU_30=400
RPM_CPU_MAX=1500
RPM_PER_30=600
RPM_PER_MAX=1800

SP=35
T=5
Kp=4
Ki=0
Kd=40

CPU_T=2

CPU_REF=40
CPU_SCALE=6

Here's 12 hours of output. It's pretty repetitive, you don't need to look through it all, but bottom line there's stretches where it seems to act as intended, but really seems to struggle overall.
https://pastebin.com/6J4UKpFd

 

[Glorious1]

@zvans18, yes, something is seriously wrong there. I hope you'll be willing to work with me to figure it out, as I don't have a 2-zone system to test with. I'll send you a private message later.

 

[Glorious1]

I'm looking for someone to test a new version of the dual zone control script, as I don't have such a system. I'm interested in trying it on both FreeNAS 9.10 and Corral. You'll have to change all the settings for your system and report back with logs. If there is a problem, I may ask for debug logs and for you to try a modified version. If you can do this, please send me a private conversation message.

 

[Glorious1]

Glorious1 updated Fan Scripts for Supermicro Boards Using PID Logic with a new update entry:

Improvements to both control scripts

In spinpid2.sh, the reading of duty cycles from the motherboard is now commented out (disabled) by default. Some boards report incorrect duty, and this causes the script to go bonkers. I have no idea how widespread this fault is, but there is really no harm in not asking the board for duty cycles - the script assumes the duty is what it sets. There is also some minor code cleanup.

spinpid.sh (the single-zone version) has some improvements in logic that make it work better. It holds...

Read the rest of this update entry...

 

[Spearfoot]

This is cool stuff!

Wish I could use it, but FreeNAS doesn't install the IPMI devices when it's virtualized on ESXi.

Also, FWIW... my X9DRi-LN4F+ supermicro board doesn't seem to support the raw duty cycle commands (raw 0x30 0x70 0x66 etc.). Does anyone else have the same problem with X9-series boards?

 

[Ericloewe]

X9 boards use an older BMC, so it's not unexpected that they're incompatible.

 

[Glorious1]

Too bad. Regarding the raw commands, Supermicro gives the X9 raw command for fan mode here, and it's the same as the other boards. This article indicates that the duty cycle commands are also the same. Maybe there is something unique about your model, but that seems unlikely. I guess you tried it outside of ESXi (I don't know what that is)?

 

[Spearfoot]

Too bad. Regarding the raw commands, Supermicro gives the X9 raw command for fan mode here, and it's the same as the other boards. This article indicates that the duty cycle commands are also the same. Maybe there is something unique about your model, but that seems unlikely. I guess you tried it outside of ESXi (I don't know what that is)?

Those commands are for setting the fan mode (standard, full-speed, optimal, heavy i/o) and they work fine on my X9 board. It's the raw commands for setting the fan zone duty cycle that don't seem to be supported. I've sent an inquiry to supermicro tech support; we'll see what they have to say.

 

[theirman]

Hi, first off, thank you for these great scripts.

But I seem to have an issue where one intake fan keeps spinning at max RPM, no matter what the mean temperature is.

My motherboard is a supermicro X10SDV-6C-TLN4F and comes with 4 PMW fan pins.

FAN2 was already taken by the CPU. Never touched that.
I've installed one exhaust fan on FAN1 (NF S12A up to 1200 RPM), and two intake fans respectively on FAN3 & FAN4 (both NF F12 up to 1400 RPM).

Ran spincheck.sh for 2 weeks, and the following speeds were measured in that interval:

So every fan was able to get read by that script.

I've been running spinpid.sh for a while now, and whenever I check my fan speeds, it'll show the FAN4 on max speed, like now:

Code:

Fan Speed
Exhaust: 300 RPM
CPU:	2800 RPM
Intake1: 200 RPM
Intake2: 1400 RPM

And the other ones are that low because the drives are cooled enough by this one fan. Which makes it pretty loud, and the temperature of the drives that are directly by that one FAN lower than the other ones. I'd like both intake fans to be controlled together, so the air is distributed evenly, and it'd produce less noise in general.

The ERRc also goes negative in the night, when it'd be able to spin all fans on a pretty low RPM (because FAN4 keeps being at 1400).

Code:

Wednesday, Jun 28																	   Fan %				  Interim
		  ada2 ada3 ada4 ada5 ada6 ada7 Tmax Tmean  ERRc	  P	 I	  D CPU Driver Prev/New MODE	  RPM Adjustments
...
02:47:30  *36  *33  *35  *37  *36  *37  ^37  35.66 -2.34  -9.36  0.00   0.00  51 Drives 20/20	Full	  300
02:52:44  *36  *33  *35  *37  *36  *37  ^37  35.66 -2.34  -9.36  0.00   0.00  51 Drives 20/20	Full	  300
02:57:58  *36  *33  *35  *37  *36  *37  ^37  35.66 -2.34  -9.36  0.00   0.00  51 Drives 20/20	Full	  300 26 38 44 62 53 44 35 26 20
03:03:12  *37  *33  *36  *37  *36  *37  ^37  36.00 -2.00  -8.00  0.00   2.72  52 Drives 20/20	Full	  300
03:08:26  *37  *33  *36  *38  *36  *37  ^38  36.16 -1.84  -7.36  0.00   1.28  51 Drives 20/20	Full	  300
...

Could anyone tell me why FAN4 isn't accounted for by spinpid?

EDIT: Decided to have a scroll through the manual, and apparently FAN4 should have been called FANA and is part of the second cooling zone...

Does that mean I should be running spinpid2? And somehow move the CPU FAN from FAN2 to FAN4?
Don't fancy moving the CPU fan as it came preinstalled like that, and FAN2 is literally right next to the CPU.
Is it possible to explain to the script that FAN2 is CPU, and FAN3/4 intake?

 

[Stux]

It's possible in my script.

 

[theirman]

I saw you can set the FAN that's connected to the CPU, and the one to your HDD.
What would that make it in my case? FAN2 for the CPU obviously, but what'd be the HDD FAN? FAN3 or FAN4? Do I need to connect both to the same header somehow? What about my exhaust fan at FAN1, will that also be controlled?

I still feel like I should get my hardware configuration checked out first. Does anyone else with an X10 motherboard have the same configuration, where FAN1, FAN2, and FAN3 belong to cooling zone 1, and FAN4 to cooling zone 2? Where FAN2 is controlling the CPU fan, FAN1 the exhaust, and FAN3 & 4 the intakes over the HDD bay?
Or won't that matter, and should I just try the script and check if it runs correctly?

EDIT: Just read some more, and apparently I'd need to connect a splitter to the FAN4 pin, and connect my two intakes to it, so that'd be the "cooling zone 2" aka the peripheral/hdd zone? And just leave my intake on FAN1, and CPU on FAN2. Correct?

Code:

# To use this correctly, you should connect all your PWM HD fans, by splitters if necessary to the FANA header.
# CPU, case and exhaust fans should then be connected to the numbered (ie CPU based) headers. This script will then control the
# HD fans in response to the HD temp, and the other fans in response to CPU temperature. When CPU temperature is high the HD fans.
# will be used to provide additional cooling, if you specify cpu/hd shared cooling.

I guess I'll come over to that thread if I have any more questions. But what's the main difference between your script Stux and the "spinpid2" one here?

 

[Stux]

My script uses a simple thermostat loop rather than a pid controller.

@Kevin Horton has a version with PID too.

If it were me I'd connect all the fans except the CPU fan to one zone and the CPU fan to the other.

If you only have one CPU fan, that can go on the FAN4 header and the rest of our fans on 1,2,3. Thus your peripheral zone becomes your CPU zone and vice versa. The script doesn't really care as long as you configure it right.

Anyway, the logic my script uses is it spins up the 'hd' fans when the HDs need cooling. And it spins up the CPU fan if the CPU needs cooling, and if configured, it will spin up the HD fans (Ie exhaust) if the CPU needs additional cooling.

BTW, I've got an X10SDV + Node 304 on order, so will be playing with this combo in the coming weeks.

 

[theirman]

Well these pidscripts are also basically a thermostat loop, aren't they?

I don't fancy having to unplug the CPU fan from the FAN2 pin. As you can see from this picture, the cord is quite short, and it feels wrong to move it over to FAN4. But I do follow your logic.

I might just go with a Y splitter and attach both intakes to FAN4, and leave the FAN1(exhaust) and FAN2 (CPU) as if. See how that goes with the spindpid2 or your script.

Did you get an actively cooled one (+), or a passive one? Would be interested to know what you do if it's the first.

 

[Stux]

these pidscripts are also basically a thermostat loop, aren't they?

A simple thermostat works by turning a device on/off when the temperature is above/below a setpoint, perhaps with some hysteresis, but no feedback, and in the case of these fan scripts, multiple discrete levels.

A PID control loop, is more complicated, and has feedback and error calculations so that the amount of time on/off or intensity is adjusted to maintain the setpoint without over/undershooting (as much).

https://en.wikipedia.org/wiki/PID_controller

I don't fancy having to unplug the CPU fan from the FAN2 pin. As you can see from this picture, the cord is quite short, and it feels wrong to move it over to FAN4. But I do follow your logic.

I might just go with a Y splitter and attach both intakes to FAN4, and leave the FAN1(exhaust) and FAN2 (CPU) as if. See how that goes with the spindpid2 or your script.

Your choice, I thought you might like to avoid a splitter. In my own systems I use a pair of splitters to run 3 fans off FANA.

Did you get an actively cooled one (+), or a passive one? Would be interested to know what you do if it's the first.

Its supposed to be active. Will see when it arrives ;)

 

[Glorious1]

@theirman, if you have two fan zones, and it sounds like you do, you should use spinpid2.sh, not spinpid.sh. You can use either zone for CPU, you just have to set the setting at the beginning of the script that tells it which zone is being used for CPU vs. peripheral.

Be sure to set the limits for each fan header to match the fans connected to it; see instructions for link.

If you still have one fan at max after you do these things correctly, I would suspect a bad fan or bad connection/wire.

 

[theirman]

I thought you might like to avoid a splitter. In my own systems I use a pair of splitters to run 3 fans off FANA.

I thought some more about it, and yeah, it just seems right to have only the CPU fan on 1 cooling zone, and the other 3 fans on the other. Without using a splitter every single one can also be read out and controlled independently. I've also remembered every Noctua fan came with an PWM extender cable. So unless that CPU fan pin is glued in, I plan to detach it, use one (or more) of these extenders, and plug it into FAN4 (second cooling zone). Both intakes and the exhaust should be part of zone 1 then.
I do find it odd that supermicro themselves didn't make the CPU plug into the second zone. Especially since only the actively cooled board, with that CPU fan, has two zones (and one pin - FAN4 - extra).

Its supposed to be active. Will see when it arrives ;)

Be sure to let us know how you set it up!

@theirman, if you have two fan zones, and it sounds like you do, you should use spinpid2.sh, not spinpid.sh. You can use either zone for CPU, you just have to set the setting at the beginning of the script that tells it which zone is being used for CPU vs. peripheral.

Be sure to set the limits for each fan header to match the fans connected to it; see instructions for link.

If you still have one fan at max after you do these things correctly, I would suspect a bad fan or bad connection/wire.

Yes, I will. I doubt it'll be a bad fan or connection since it's only 6 months old. It'll be because the fan mode is set to full, and the spinpid script being unable to control this second cooling zone.
Once all my fans are properly connected (CPU to FAN4/Z2, and rest to Z1), I'll try out spinpid2.sh first.

 

[Stux]

I thought some more about it, and yeah, it just seems right to have only the CPU fan on 1 cooling zone, and the other 3 fans on the other. Without using a splitter every single one can also be read out and controlled independently. I've also remembered every Noctua fan came with an PWM extender cable. So unless that CPU fan pin is glued in, I plan to detach it, use one (or more) of these extenders, and plug it into FAN4 (second cooling zone). Both intakes and the exhaust should be part of zone 1 then.
I do find it odd that supermicro themselves didn't make the CPU plug into the second zone. Especially since only the actively cooled board, with that CPU fan, has two zones (and one pin - FAN4 - extra).

IPMI's fan modes expect CPU zone to be zone 0 and peripheral zone to be zone 1.

Be sure to let us know how you set it up!

Board arrived. It is active. Still waiting on ram.

Will do a build log eventually.

 

[PnoT]

I'm seeing this when running spincheck and haven't seen anyone post up something similar. Can someone help me with the error it's throwing.

The motherboard is an X10SRL-F-O with the following listed sensors:

Code:

CPU Temp		 | 45.000	 | degrees C  | ok	| 0.000	 | 0.000	 | 0.000	 | 91.000	| 96.000	| 96.000
PCH Temp		 | 44.000	 | degrees C  | ok	| -11.000   | -8.000	| -5.000	| 90.000	| 95.000	| 100.000
System Temp	  | 32.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
Peripheral Temp  | 41.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
VcpuVRM Temp	 | 47.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 95.000	| 100.000   | 105.000
VmemABVRM Temp   | 45.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 95.000	| 100.000   | 105.000
VmemCDVRM Temp   | 40.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 95.000	| 100.000   | 105.000
DIMMA1 Temp	  | 44.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMA2 Temp	  | 41.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMB1 Temp	  | 43.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMB2 Temp	  | 41.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMC1 Temp	  | 44.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMC2 Temp	  | 44.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMD1 Temp	  | 46.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMD2 Temp	  | 43.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
FAN1			 | 3200.000   | RPM		| ok	| 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FAN2			 | 1700.000   | RPM		| ok	| 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FAN3			 | 3200.000   | RPM		| ok	| 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FAN4			 | na		 |			| na	| na		| na		| na		| na		| na		| na
FAN5			 | 2800.000   | RPM		| ok	| 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FANA			 | 3200.000   | RPM		| ok	| 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
12V			  | 11.874	 | Volts	  | ok	| 10.173	| 10.299	| 10.740	| 12.945	| 13.260	| 13.386
5VCC			 | 5.026	  | Volts	  | ok	| 4.246	 | 4.298	 | 4.480	 | 5.390	 | 5.546	 | 5.598
3.3VCC		   | 3.367	  | Volts	  | ok	| 2.789	 | 2.823	 | 2.959	 | 3.554	 | 3.656	 | 3.690
VBAT			 | 3.131	  | Volts	  | ok	| 2.375	 | 2.487	 | 2.599	 | 3.775	 | 3.887	 | 3.999
Vcpu			 | 1.809	  | Volts	  | ok	| 1.242	 | 1.260	 | 1.395	 | 1.899	 | 2.088	 | 2.106
VDIMMAB		  | 1.191	  | Volts	  | ok	| 0.948	 | 0.975	 | 1.047	 | 1.344	 | 1.425	 | 1.443
VDIMMCD		  | 1.191	  | Volts	  | ok	| 0.948	 | 0.975	 | 1.047	 | 1.344	 | 1.425	 | 1.443
5VSB			 | 5.026	  | Volts	  | ok	| 4.246	 | 4.298	 | 4.480	 | 5.390	 | 5.546	 | 5.598
3.3VSB		   | 3.299	  | Volts	  | ok	| 2.789	 | 2.823	 | 2.959	 | 3.554	 | 3.656	 | 3.690
1.5V PCH		 | 1.518	  | Volts	  | ok	| 1.320	 | 1.347	 | 1.401	 | 1.644	 | 1.671	 | 1.698
1.2V BMC		 | 1.209	  | Volts	  | ok	| 1.020	 | 1.047	 | 1.092	 | 1.344	 | 1.371	 | 1.398
1.05V PCH		| 1.050	  | Volts	  | ok	| 0.870	 | 0.897	 | 0.942	 | 1.194	 | 1.221	 | 1.248
Chassis Intru	| 0x0		| discrete   | 0x0000| na		| na		| na		| na		| na		| na
PS1 Status	   | 0x1		| discrete   | 0x0100| na		| na		| na		| na		| na		| na
PS2 Status	   | 0x1		| discrete   | 0x0100| na		| na		| na		| na		| na		| na

The error message:

Code:

.
How many whole minutes do you want between spin checks?
5

NOTE ABOUT DUTY CYCLE (Fan%0 and Fan%1):
Some boards apparently report incorrect duty cycle, and can
report duty cycle for zone 1 when that zone does not exist.

Key to drive status symbols:  * spinning;  _ standby;  ? unknown							  Version 2017-03-03

Monday, Jul 31
		  da0  da1  da2  da3  da4  da5  da6  da7  da8  da9  da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 Tmax Tmean  ERRc CPU  FAN1  FAN2  FAN3  FAN4  FANA Fan%0 Fan%1 MODE
09:45:39  *41  *39  *37  *43  *42  *42  *42  *40  *37  *45  *44  ./spincheck.sh: line 118: let: Tsum += : syntax error: operand expected (error token is "+= ")
*0   ./spincheck.sh: line 118: let: Tsum += : syntax error: operand expected (error token is "+= ")
*0   *44  *40  *41  *41  *40  *39  *36  *36  *35  ^45  36.55  2.98  44  3200  1700  3200   ---  3200	34	34 Standard