Supermicro X9SCM-F, fans ramp up every 20 seconds (sensor ipmi thresholds are lowered to bare minimium)

[naskoboy]

I have lowered the thresholds significantly (see below) and yet fans ramp up every 20 seconds.
Readings-snapshot below was taken right before the fans ramp up - you can see that "FAN A" was the slowest running fan at 300 RPM.
I have no clue why is this happening.
Firmware Revision 03.52 (latest)
Tried "Optimal" as well as "Standard" fan mode in BIOS - no difference.
All fans are Arctic PWM 4pin.

Help appreciated !

 

[Davvo]

We have resources for that, look in my signature.
Issue is that IPMI thinks your fans are stalling.

 

[naskoboy]

We have resources for that, look in my signature.
Issue is that IPMI thinks your fans are stalling.

I have read multiple "IPMI things fans are stalling" posts and followed the advice to lower the "lower" values - see picture above - they are all "75".
Is there anything in there that I have missed ?
thanks

 

[Davvo]

I have read multiple "IPMI things fans are stalling" posts and followed the advice to lower the "lower" values - see picture above - they are all "75".
Is there anything in there that I have missed ?
thanks

They shouldn't be all at the same value iirc.

 

[naskoboy]

They shouldn't be all at the same value iirc.

right, initially I had created lower 200 400 600, nothing worked, so lastly (out of desperation) I set all to 75 75 75.
So, I hear what you are saying but still mystery remains about "why IPMI think fans are stalled given that thresholds are not reached"

 

[Davvo]

Try the following.

Fan script for X9 motherboards?

Hey guys, my build is awfully loud and i tried to get a few of the fan scripts working, seen tons of them on the forum but kept getting errors. I stumbled on that apparently x9scm-f cant take raw commands which kinda sucks. Does anyone of you have a script/workaround for this? Thanks in advance!

www.truenas.com

 

[naskoboy]

getting even more weird:
Yesterday it was doing this "ramp up every 20 seconds" thing,
Let it run for the night (no restarts), the next morning "ramp up" was gone, fan-speeds stabilized around 300-375. Readings are almost the same as yesterday, except that "ramp up" is gone.
Now I am puzzled and worried it might come back for no reason. (It might be the phase of the moon thing....)

Q: Which threshold suggests to IPMI that fan is stalled: Low NR or CT ?

 

[Davvo]

I guess lower critical.

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

www.truenas.com

 

[naskoboy]

"ramp up" came back for no reason (no restart, no changes in configurations, etc).
"Lower" values (75 75 75) are far from reached - (300, 375 RPM, ... )
IMPI does NOT appear to pay much attention to "lower" values
this is so frustrating :(

 

[naskoboy]

I guess lower critical.

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

www.truenas.com

I appreciate your feedback, however IPMI does not appear to respect the "lower" values at all :(

 

[Davvo]

Is the firmware updated?

 

[naskoboy]

Is the firmware updated?

this is the latest firmware AFAIK

 

[Davvo]

Are you using 4-pin fans?

 

[naskoboy]

Are you using 4-pin fans?

yes, all fans are 4-pin (if they were 3pins they would be constantly running, right ?)

 

[naskoboy]

I did try "Full speed" along with ipmitool raw-commands (see here https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/)
However these commands apparently do not work on my mobo :(

root@truenas[~]# ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x64
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc): Invalid data field in request

https://www.truenas.com/community/t...-in-ipmi-supermicro-x9scm-f.70826/post-491174

I am considering to buy this fan-hub and "hide" the slow-running fans behind the fast one (~900RPM).
This way BIOS would not have a chance to freak about "stalled" fans - it will only see 1 fan at 900RPM.
Unfortunately I will lose ability to see speeds reported to the system :(

 

[naskoboy]

It was "ramping up" for about 2 days, ... then it stopped again and stabilized at 300, 375, 450RPM ... I am utterly confused what is going on.
Placed an order for the fan-hub.

ipmitool sensor list all
System Temp | 38.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000
Peripheral Temp | 35.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000
CPU Temp | 0x0 | discrete | 0x0000| na | na | na | na | na | na
FAN 1 | 975.000 | RPM | ok | 150.000 | 225.000 | 300.000 | 3000.000 | 3150.000 | 3300.000
FAN 2 | na | | na | na | na | na | na | na | na
FAN 3 | 450.000 | RPM | ok | 150.000 | 225.000 | 300.000 | 2625.000 | 2775.000 | 3000.000
FAN 4 | 375.000 | RPM | ok | 150.000 | 225.000 | 300.000 | 2625.000 | 2775.000 | 3000.000
FAN A | 300.000 | RPM | nc | 150.000 | 225.000 | 300.000 | 2625.000 | 2775.000 | 3000.000
Vcore | 0.840 | Volts | ok | 0.480 | 0.512 | 0.544 | 1.488 | 1.520 | 1.552
3.3VCC | 3.328 | Volts | ok | 2.816 | 2.880 | 2.944 | 3.584 | 3.648 | 3.712
12V | 12.084 | Volts | ok | 10.494 | 10.600 | 10.706 | 13.091 | 13.197 | 13.303
VDIMM | 1.528 | Volts | ok | 1.152 | 1.216 | 1.280 | 1.760 | 1.776 | 1.792
5VCC | 5.120 | Volts | ok | 4.096 | 4.320 | 4.576 | 5.344 | 5.600 | 5.632
-12V | -12.098 | Volts | ok | -13.844 | -13.650 | -13.456 | -10.934 | -10.740 | -10.546
VBAT | 3.088 | Volts | ok | 2.816 | 2.880 | 2.944 | 3.584 | 3.648 | 3.712
VSB | 3.344 | Volts | ok | 2.816 | 2.880 | 2.944 | 3.584 | 3.648 | 3.712
AVCC | 3.344 | Volts | ok | 2.816 | 2.880 | 2.944 | 3.584 | 3.648 | 3.712
Chassis Intru | 0x0 | discrete | 0x0000| na | na | na | na | na | na

 

[Davvo]

I am utterly confused what is going on.

IPMI... Always finds a way.

Placed an order for the fan-hub.

Probably a good decision that will help you retain both your sanity and your hair (if you are an engineer however you are doomed to become at least partially bald).

 

[naskoboy]

Installed the fan-hub, the 3 slow-running fans were all hooked into the fan-hub, only RPM of the fastest one (~375) is being reported back to motherboard/BIOS/IPMI.
So, now motherboard is "seeing" only 2 fans running at 975 and 375 RPM - hope this would outsmart the IPMI

Code:

ipmitool sensor list all
System Temp      | 36.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Peripheral Temp  | 34.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
CPU Temp         | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
FAN 1            | 975.000    | RPM        | ok    | 150.000   | 225.000   | 300.000   | 3000.000  | 3150.000  | 3300.000
FAN 2            | na         |            | na    | na        | na        | na        | na        | na        | na
FAN 3            | na         |            | na    | na        | na        | na        | na        | na        | na
FAN 4            | 375.000    | RPM        | ok    | 150.000   | 225.000   | 300.000   | 2625.000  | 2775.000  | 3000.000
FAN A            | na         |            | na    | na        | na        | na        | na        | na        | na