Fans ramping up and down and "Fan Mode" option not visible in IPMI (Supermicro X9SCM-F)

Status
Not open for further replies.

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
First specs of my system:
  • Motherboard: Supermicro X9SCM-F
  • Chassis: Fractal Design Node 804
  • CPU: Intel Xeon E3-1260L
  • CPU cooler: Noctua NH-L9i (this has a 4-pin PWM fan)
  • Memory: 32GB ECC (4x 8GB Kingstong ValueRAM KVR1333D3E9SK2/16G)
  • HBA: Dell PERC H200 (flashed in IT mode)
  • Storage SDD: 2x Samsung PM863 240GB connected through the two onboard SATA3 (6Gb/s) ports
  • Storage HDD: 4x Western Digital Red 4TB (connected through the HBA)
  • Storage USB: 2x SanDisk Ultra Fit 32GB
I was using the three default fans of the Fractal Design Node 804 case but they were quite loud and I read in the manual you need 4-pin PWM fans for them to be automatically controlled. I figured this might also been the reason the "Fan Mode" wouldn't show up in the IPMI web interface.

v2b0dHa.png


So I went and purchased three new fans. I got three Noctua NF-F12 PWM fans which have 4-pin connectors and can be PWM controlled. I installed them and selected "Optimal" Fan Mode in the BIOS because I want my server to be as quiet as possible.

Unfortunately things haven't turned out well, the fans keep ramping up and down with 10-15 second intervals. Is there any way I can solve this problem? I found a thread to control fan speeds through IPMI but that only works on the X11 range of motherboards.

Second problem: the "Fan Mode" configuration option doesn't show up for me in the IPMI web interface as shown below. Is there a way to enable it?

ykv9Euf.png


Thanks in advance for your help.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You need to adjust the fan speed threshold in the IPMI but that must be done from the command line because there is no GUI for it. The fans ramp up because the current setting is not aware of how slow the Noctua fans run and the firmware thinks that the fans have stalled. It's just a few settings.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
Have you installed all available ipmi updates?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Here's a good resource for setting the fan thresholds via IPMI:

https://forums.freenas.org/index.ph...nge-ipmi-sensor-thresholds-using-ipmitool.35/

Also, this post from @charlie89 lists IPMI commands for fan control of X9 series boards, which should work for yours:

https://forums.freenas.org/index.php?threads/fan-script-for-x9-motherboards.69550/#post-481280
Exactly.
Thanks in advance for your help.
The "Lower Non-Recoverable" is probably what is getting you. Those Noctua fans spin so slow that the default settings will have the system thinking they are stalled. It ramps the speed and every thing looks "normal" to the system, so it releases and when the speed idles back down it hits the low speed threshold again and the cycle repeats. I had this happen in one of my servers when I switched to Noctua fans and after I adjusted the thresholds, it worked fine.
 

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
Have you installed all available ipmi updates?

Thanks for replying. I did update the normal BIOS and verified if IPMI was at the latest version and it was. According to the Supermicro website the latest version is "Firmware Revision: R 3.52". Should I do an update anyway since I cannot see the "Fan Mode" option in the web interface or are there more updates I need to do? The fact that I cannot change the fan speed mode in the IPMI web UI is annoying. I have to reboot the PC and do it from the BIOS.

o1MXoxL.png

You need to adjust the fan speed threshold in the IPMI but that must be done from the command line because there is no GUI for it. The fans ramp up because the current setting is not aware of how slow the Noctua fans run and the firmware thinks that the fans have stalled. It's just a few settings.

Here's a good resource for setting the fan thresholds via IPMI:

https://forums.freenas.org/index.ph...nge-ipmi-sensor-thresholds-using-ipmitool.35/

Also, this post from @charlie89 lists IPMI commands for fan control of X9 series boards, which should work for yours:

https://forums.freenas.org/index.php?threads/fan-script-for-x9-motherboards.69550/#post-481280

Thank you for the information. I have configured the new thresholds and I would love to hear some feedback. The command to list all the sensors does work:
Code:
root@freenas-new[~]# ipmitool sensor list all
System Temp	  | 31.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
Peripheral Temp  | 30.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
CPU Temp		 | 0x0		| discrete   | 0x0000| na		| na		| na		| na		| na		| na
FAN 1			| 1050.000   | RPM		| ok	| 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN 2			| 525.000	| RPM		| nc	| 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN 3			| na		 |			| na	| na		| na		| na		| na		| na		| na
FAN 4			| 525.000	| RPM		| nc	| 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN A			| 525.000	| RPM		| nc	| 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
Vcore			| 0.768	  | Volts	  | ok	| 0.480	 | 0.512	 | 0.544	 | 1.488	 | 1.520	 | 1.552
3.3VCC		   | 3.312	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
12V			  | 12.137	 | Volts	  | ok	| 10.494	| 10.600	| 10.706	| 13.091	| 13.197	| 13.303
VDIMM			| 1.536	  | Volts	  | ok	| 1.152	 | 1.216	 | 1.280	 | 1.760	 | 1.776	 | 1.792
5VCC			 | 5.024	  | Volts	  | ok	| 4.096	 | 4.320	 | 4.576	 | 5.344	 | 5.600	 | 5.632
-12V			 | -12.486	| Volts	  | ok	| -13.844   | -13.650   | -13.456   | -10.934   | -10.740   | -10.546
VBAT			 | 3.056	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
VSB			  | 3.328	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
AVCC			 | 3.312	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
Chassis Intru	| 0x1		| discrete   | 0x0100| na		| na		| na		| na		| na		| na

So I looked up the specs of my fans and I saw that the lower limit of my fans are 300 APM. Does that mean I should set the Lower Non-Critical to 300? Because everything above 300 should be within specs I think. I've tested changing the thresholds and it seems to work.
Code:
ipmitool sensor thresh "FAN 2" lower 100 200 300

I did the same for the other two fans and this is the output of my sensors now.
Code:
root@freenas-new[~]# ipmitool sensor list all
System Temp	  | 31.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
Peripheral Temp  | 30.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
CPU Temp		 | 0x0		| discrete   | 0x0000| na		| na		| na		| na		| na		| na
FAN 1			| 1125.000   | RPM		| ok	| 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN 2			| 525.000	| RPM		| ok	| 75.000	| 225.000   | 300.000   | 18975.000 | 19050.000 | 19125.000
FAN 3			| na		 |			| na	| na		| na		| na		| na		| na		| na
FAN 4			| 9600.000   | RPM		| nc	| 75.000	| 225.000   | 300.000   | 18975.000 | 19050.000 | 19125.000
FAN A			| 525.000	| RPM		| ok	| 75.000	| 225.000   | 300.000   | 18975.000 | 19050.000 | 19125.000
Vcore			| 0.784	  | Volts	  | ok	| 0.480	 | 0.512	 | 0.544	 | 1.488	 | 1.520	 | 1.552
3.3VCC		   | 3.312	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
12V			  | 12.137	 | Volts	  | ok	| 10.494	| 10.600	| 10.706	| 13.091	| 13.197	| 13.303
VDIMM			| 1.536	  | Volts	  | ok	| 1.152	 | 1.216	 | 1.280	 | 1.760	 | 1.776	 | 1.792
5VCC			 | 5.024	  | Volts	  | ok	| 4.096	 | 4.320	 | 4.576	 | 5.344	 | 5.600	 | 5.632
-12V			 | -12.292	| Volts	  | ok	| -13.844   | -13.650   | -13.456   | -10.934   | -10.740   | -10.546
VBAT			 | 3.056	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
VSB			  | 3.328	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
AVCC			 | 3.312	  | Volts	  | ok	| 2.816	 | 2.880	 | 2.944	 | 3.584	 | 3.648	 | 3.712
Chassis Intru	| 0x1		| discrete   | 0x0100| na		| na		| na		| na		| na		| na

After changing the thresholds I restarted my FreeNAS server and changed the fan speed mode in the BIOS to "Optimal" again and the fan are no longer ramping up and down! This is very nice. But I do see that Fan A is spinning very low I don't know if I should be worried by this. This the second exhaust fan that cools the hard drives. I could always switch it to fan header 3 to get the same RPM as the other two, any thoughts?

U0A72gw.png


Should I do something with the upper limits also or are these not really relevant? If so, I should probably Upper Non-Critical to 1500 and increase with 100 for the other two?

Finally, the other commands don't seem to work and I also cannot get into my ipmi shell. These commands should work on the X9-series of motherboard according to that other thread, the Supermicro FAQ entry and the Reddit thread. This command for "read" should be.
Code:
root@freenas-new[~]# ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0xff
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)

This doesn't work either. I'm using the same information to connect that I use to login to the IPMI web UI.
Code:
root@freenas-new[~]# ipmitool -I lanplus -H 192.168.***.*** -U *** -P '***' shell
IPMI LAN send command failed
Error: Unable to establish IPMI v2 / RMCP+ session

Any thoughts?
 

droeders

Contributor
Joined
Mar 21, 2016
Messages
179

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
Hi, I've tried your command and this is the output I'm getting

Code:
root@freenas[~]# ipmitool -v raw 0x0 0x0
Running Get PICMG Properties my_addr 0x20, transit 0, target 0
Error response 0xc1 from Get PICMG Properities
Running Get VSO Capabilities my_addr 0x20, transit 0, target 0
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
RAW REQ (channel=0x0 netfn=0x0 lun=0x0 cmd=0x0 data_len=0)
RAW RSP (6 bytes)
 00 20 20 20 20 20


I've used the ipmicfg tool before to reset the admin password of the BMC. I will look into it this weekend. So far everything is at satisfactory levels with regard to noise and cooling. But I would like to tune the fans a bit more. I would like to have the exhaust fan for my hard drives spin at 100 or 200 RPM higher compared to the two other fans in the motherboard compartment. I could also consider moving my single intake fan to the hard drive compartment.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
This craziness in the interface not showing things---this happens all the time with Supermicro if you install new IPMI updates and TRY TO RETAIN THE OLD SETTINGS. You must not click "retain the old settings". :)
 

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
This craziness in the interface not showing things---this happens all the time with Supermicro if you install new IPMI updates and TRY TO RETAIN THE OLD SETTINGS. You must not click "retain the old settings". :)

Unfortunately your suggestion did not work. I re-flashed the IPMI firmware and made sure to unchecked the boxes for retaining any settings. After it was done I still couldn't see the "Fan Mode" option under "Configuration".
 
Joined
May 10, 2017
Messages
838
All my X9SCL and X9SCM boards are the same, no fan control with the web IPMI but it does show up with IPMI view:

upload_2018-11-4_11-14-38.png
 

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
Ah, I was not aware this utility existed, thank you very much.

Do you know if it's possible to set the fan speed of one particular fan on my motherboard? I'd like to have the exhaust fan of my hard drives run 100-200 RPM faster compared to the other two.
 
Joined
May 10, 2017
Messages
838
Do you know if it's possible to set the fan speed of one particular fan on my motherboard?

These boards have 2 fan zones, fans 1 to 4 are a zone and fan A is another, you can set each zone to a different speed using IPMI raw commands.
 

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
Yeah I found some other threads with those command and tried some of them but it didn't work for me. Then I would have to put the exhaust fan of my hard drives on the Fan A connector and adjust the speed for that zone.

Hopefully I can find something that works for me.
 
Joined
May 10, 2017
Messages
838
I believe these boards don't have the same fine adjustment as most X10/X11 boards, you can only set each zone to Full, Standard or Optimal, though I've heard otherwise but still didn't have the chance to test.
 
Joined
May 10, 2017
Messages
838
you can only set each zone to Full, Standard or Optimal

On second though this is likely wrong, the only commands I remember getting my X9SCL or X9SCM boards to accept was changing the fan mode, and IIRC it always changed both zones, i.e., these boards don't accept individual zone control, unlike some other X9 boards, I believe only dual socket models (X9D..)

Code:
ipmitool raw 0x30 0x45 0x00
Gets the mode 0/1/2/4  Standard/Full/Optimal/Heavy IO

ipmitool raw 0x30 0x45 0x01 0x00
Sets the mode to Standard

ipmitool raw 0x30 0x45 0x01 0x01
Sets the mode to Full

ipmitool raw 0x30 0x45 0x01 0x02
Sets the mode to Optimal

ipmitool raw 0x30 0x45 0x01 0x04
Sets the mode to Heavy IO (not available on these boards)
 

Mannekino

Patron
Joined
Nov 14, 2012
Messages
332
Hmm makes sense, I guess I'm out of luck then. I set the mode to Standard and seems to be OK. Although 2 of my drives are around 38-39 degrees. When the CPU gets a bit hotter the fans spin faster and the disk temperature drop to around 35 degrees.
 
Status
Not open for further replies.
Top