How To: change sensor thresholds with IPMI using ipmitool

Status
Not open for further replies.

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
This thread is now closed. This document has been ported to the Resources section and is available here.

The discussion has been copied over the Resource's automagic discussion thread, available through the Resource's page.

The original document follows below, inside the spoiler tag.


Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to 100% speed). This behavior is determined by the sensor thresholds.

Supermicro X10 motherboard fan profiles are controlled via IPMI. Unfortunately, only very basic functions are exposed via its web interface and Supermicro's own IPMI utility is superbly well hidden on Supermicro's website and uses Java. The pragmatists among us would complain about Java, use the application, and move on - however, for some reason (*cough* Java *cough*) Supermicro application and its nice GUI do not always work properly. Thus, third-party universal IPMI tools are needed.

This guide should also be helpful for owners of other motherboards with similar problems or for those trying to learn how to interact with the motherboard's sensors using IPMI.

This is Part 1 of this guide and covers how to perform these operations “locally” (in other words, from FreeNAS’ console). Part 2 covers how to perform these operations remotely using ipmiutil, running on a client machine.

In this guide, I will be using ipmitool, running on the server – all interactions are via a remote terminal session.
Ipmitool can also be used remotely over the network, in the same way that ipmiutil is used in Part 2. If you're interested in this usage scenario, please read the relevant man pages.

Ipmitool forces the user to jump through more hoops in order to accomplish some tasks, but is included in FreeNAS and allows setting thresholds for sensors that are not attached.
Ipmiutil is somewhat more straightforward, but requires a remote client and refuses to change thresholds on sensors that are not attached.

While I will provide the necessary steps and explain some options, I recommend you read through the appropriate man page sections to learn about the various options.

Please note that all input may be case sensitive. For details, check the relevant man pages. Particularly, sensor names are case-sensitive.

Start by loading ipmitool:

Code:
# kldload ipmi.ko


Now, run the following command to list all sensors:

Code:
# ipmitool sensor list all


This will list the system’s sensors, current readings and current thresholds, as exemplified below:

Code:
[root@freenas] ~# ipmitool sensor list all
CPU Temp		 | 37.000	 | degrees C  | ok	| 0.000	 | 0.000	 | 0.000	 | 95.000	| 98.000	| 100.000
System Temp	  | 30.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
Peripheral Temp  | 40.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 80.000	| 85.000	| 90.000
PCH Temp		 | 48.000	 | degrees C  | ok	| -11.000   | -8.000	| -5.000	| 90.000	| 95.000	| 100.000
VRM Temp		 | 39.000	 | degrees C  | ok	| -9.000	| -7.000	| -5.000	| 87.000	| 92.000	| 97.000
DIMMA1 Temp	  | na		 |			| na	| na		| na		| na		| na		| na		| na
DIMMA2 Temp	  | 31.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
DIMMB1 Temp	  | na		 |			| na	| na		| na		| na		| na		| na		| na
DIMMB2 Temp	  | 32.000	 | degrees C  | ok	| 1.000	 | 2.000	 | 4.000	 | 80.000	| 85.000	| 90.000
FAN1			 | na		 |			| na	| na		| na		| na		| na		| na		| na
FAN2			 | na		 |			| na	| na		| na		| na		| na		| na		| na
FAN3			 | 900.000	| RPM		| ok	| 300.000   | 400.000   | 500.000   | 3700.000  | 3800.000  | 3900.000
FAN4			 | 900.000	| RPM		| ok	| 300.000   | 400.000   | 500.000   | 3600.000  | 3700.000  | 3800.000
FANA			 | 1100.000   | RPM		| ok	| 400.000   | 600.000   | 800.000   | 25300.000 | 25400.000 | 25500.000
Vcpu			 | 1.755	  | Volts	  | ok	| 1.242	 | 1.260	 | 1.395	 | 1.899	 | 2.088	 | 2.106
VDIMM			| 1.453	  | Volts	  | ok	| 1.096	 | 1.124	 | 1.201	 | 1.642	 | 1.719	 | 1.747
12V			  | 12.000	 | Volts	  | ok	| 10.164	| 10.521	| 10.776	| 12.918	| 13.224	| 13.224
5VCC			 | 4.969	  | Volts	  | ok	| 4.225	 | 4.380	 | 4.473	 | 5.372	 | 5.527	 | 5.589
3.3VCC		   | 3.344	  | Volts	  | ok	| 2.804	 | 2.894	 | 2.969	 | 3.554	 | 3.659	 | 3.689
VBAT			 | 3.015	  | Volts	  | ok	| 2.400	 | 2.490	 | 2.595	 | 3.495	 | 3.600	 | 3.690
AVCC			 | 3.329	  | Volts	  | ok	| 2.399	 | 2.489	 | 2.594	 | 3.494	 | 3.599	 | 3.689
VSB			  | 3.284	  | Volts	  | ok	| 2.399	 | 2.489	 | 2.594	 | 3.494	 | 3.599	 | 3.689
Chassis Intru	| 0x0		| discrete   | 0x0000| na		| na		| na		| na		| na		| na


The thresholds listed are, in order: lnr, lcr, lnc, unc, ucr, unr

These are acronyms for:

Lower Non-Recoverable
Lower Critical
Lower Non-Critical

Upper Non-Critical
Upper Critical
Upper Non-Recoverable

Take note of the sensor names for the sensors you want to change (if you don't know which fans you want to edit, I recommend you check the IPMI log in the web interface to see which fans have sensor events being asserted).

Now, calculate the lowest and highest angular velocities your fan is rated to run at (check the manufacturer's specs). For instance, my Noctua NF-F12 IndustrialPPC 3000 PWM are rated at 750RPM +-20% at the low end, so 600RPM or less is an appropriate lower non-critical threshold value. To get the other values, subtract 100 for the lower critical and 200 for the lower non-recoverable.

Take note of your results.


To set the lower thresholds, use:

Code:
~# ipmitool sensor thresh "*sensor name*" lower *lnr* *lcr* *lnc*


Replacing *sensor name*, *lnr*, *lcr* and *lnc* with the appropriate values.


To set the upper thresholds, use:

Code:
~# ipmitool sensor thresh "*sensor name*" upper *unc* *ucr* *unr*


Replacing *sensor name*, *unc*, *ucr* and *unr* with the appropriate values.

It has come to my attention that ipmitool will gladly set the thresholds in absurd orders (lower non-critical lower than lower non-recoverable, for instance). The effects of this are unknown. In any case, I recommend that attention be paid to the proper order to avoid potential future issues.

Please note that invalid inputs will be rounded to the nearest valid value. On my system, for instance, setting a fan threshold to 540 will actually set it to 500. 550 is set as 600.

Repeat this for all fans whose thresholds you wish to change.

Edit 2015-04-02: Added warning regarded out-of-order thresholds

 
Last edited:

RobertT

Explorer
Joined
Sep 28, 2014
Messages
54
Thanks so much for writing this.
I just got my new ASRock C2750D4I motherboard.
Apparently there is a (well I dont want to call it bug.. Oh hell a BUG!) where if you goto firmware update and then cancel when it asks you to upload the firmware the passwords for all the accounts are messed up.
So bottom line is that you are then unable to login to the IPMI anymore with any password. So this is how I fixed it with ipmitool (in case someone else has the issue)..

From freenas root shell:
  1. # kldload ipmi.ko
  2. # ipmitool user list 1
  3. # ipmitool user set password 3 foobar
Explanation:
  1. This loads the ipmi module
  2. List all the users - The "1" is a channel and this was the only one that seemed to work for me.
  3. This will set the password with the "ID" of 3 to foobar. Replace the 3 with whatever ID is appropriate from the output of command #2.
 

troun

Dabbler
Joined
Jul 13, 2013
Messages
33
Using ULV Xeon, -> CPU voltage alert -> motherboard bipping -> me getting crazy

So simply, thank you, thank you and thank you!!!
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211
Thanks very much for this how-to. I just changed all four fans on my case to Noctua NF-S12A PWM fans, and I used the low-noise adapter that brings the RPMs down a bit from standard. I got 105 emails about nonrecoverable fan speeds before I found and implemented the ipmitool method. I thought it was quite straightforward. Now I'll keep an eye on temperatures for a while and see if the low-noise adapters are going too far.

It was interesting to see that my motherboard automatically responded to the alert by changing fan speed mode to either "Full Speed" or "High I/O Speed". When I set it back to "Standard Speed", the RPMs dropped and the lnr alert tripped again. Now alles ist in Ordnung.

Thanks again for the very well written guide.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
I wouldn't bother with the low-noise adapters, since the motherboard controls the fans decently and they're quiet anyways. Even my Noctua NF-F12 industrialPPC 3000s are bearable with the motherboard controlling them.
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211
I think you're right, so I took off the adapter cables. I had fan speed mode settings in the IPMI web gui at "Full Speed" before I shut down to remove them. When I powered back up it was stuck in that mode and wouldn't change (though the UI said it did). I had to cycle power completely to the motherboard (shut down, turn off power supply, then back up) to get it to really change.
 

kyletak

Cadet
Joined
Jan 13, 2015
Messages
5
Thank you!! This solved the issue for me where my fans kept spinning up/ slowing down ~5 seconds
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Thanks for all you research on this Eric.

I've been trolling SuperMicros FAQ section and the documentation on the I/O chip that they use on the X10SL7-F and am going do do some research on the fans before I go into production with my first build.

There is a lot of conflicting information but from what I can tell it appears that on the UP boards they typically have headers designated FAN-1 through FAN-4 that all output the same PWM signal based on the programmed profile from the CPU sensor. Furthermore the expectation is that the "tac" signal I putted on FAN-1 is expected to be from the CPU fan.

What I find intriguing is that some FAQ posts and other motherboard manuals suggest that the FAN-A header is for the Chassis fans and may be controlled by the system sensor, usually located at the rear of the motherboard on the lower half. The peripheral sensor appears to be located on the front lower portion of the board to measure the temperature neat the expansion slots.

What I am hoping is that the FAN-A header is controlled separately and can be used to control the ambient case temperature independent of the CPU. If this is the case the this could be tweaked with the idea of keeping the HDD below the magik 40 degree C mark.

To test this I have installed 5-way 4-pin fan splitter connected to the FAN-A header and connected 4 120mm cougar fans to it and modded the splitter to feed the tac output from the three non monitored fans back to FAN-2 through FAN-4.

Once my CPU arrives I will do a bit of testing with a heat gun ( carefully with low wattage ) once I get my magnifying glass out and find the thermistors.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
My board (X9SCM) seemed to have all of the fans on the board output the same signal. They were also based on the CPU temp, so there is no thermistor, thermocouple, or equivalent to see. Yes, this means that your fans will speed up (or slow down) based on CPU temp and not the disks themselves. There's two options:

1. Check your disks and make sure they stay cool despite the fact that they don't track with the hard drive temps. I'd expect that a system with proper cooling will have no problem with this as there should be at least some correlation between CPU temp and hard drive temp.
2. Force the fans to some speed.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
After some testing, my X10SL7-F (who has also FAN-1 to 4 and FAN-A headers) split the headers in two groups: FAN-1 to 4, and FAN-A.

After some reading what I've understood is that the CPU fan should be connected to FAN-1 and that the PWM for FAN-1 to 4 is controlled by the CPU temp.

FAN-A PWM is controlled by the system or peripheral (I can't recall which one) temp but only in the fan mode "Optimal Speed", in the others fan modes it's the same as the FAN-1 to 4 PWM.
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
I believe FAN A is controlled with the PCH temperature.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
One or two posts in their FAQs for cooling state that the FAN-A is controlled by the "System" thermistor which makes sense as this would be the most accurate measurement of the temperature inside the enclosure. The "Peripheral" thermistor would be skewed by HBAs and Graphics cards and the PCH by, well, the PCH.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
And on one specific board here is where they say that sensor is. So hopefully we can infer that the design might be similar in recent X10s.

I have a question into SuperMicro Support asking:
  1. Where is the "System" Temperature Sensor Located on a X10SC7?
  2. Does the "System" sensor control the PWM duty-cycle on the "FAN-A" header?
  3. What are the lower and upper temperature thresholds for FAN-A and what is the low RPM fan limit for the tachometer input?
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
After some testing, my X10SL7-F (who has also FAN-1 to 4 and FAN-A headers) split the headers in two groups: FAN-1 to 4, and FAN-A.

After some reading what I've understood is that the CPU fan should be connected to FAN-1 and that the PWM for FAN-1 to 4 is controlled by the CPU temp.

FAN-A PWM is controlled by the system or peripheral (I can't recall which one) temp but only in the fan mode "Optimal Speed", in the others fan modes it's the same as the FAN-1 to 4 PWM.
You do seem to be correct, and my most recent post (in the other thread) was wrong: according to

http://www.supermicro.com/support/faqs/faq.cfm?faq=13433

the CPU fan should be connected not to FanA but to Fan1. I wish their manual had been clearer.
 
Last edited:

J-NAS

Dabbler
Joined
Oct 27, 2014
Messages
42
@Ericloewe, thank you so much for this thread! You are a life-saver!

I think you're right, so I took off the adapter cables. I had fan speed mode settings in the IPMI web gui at "Full Speed" before I shut down to remove them. When I powered back up it was stuck in that mode and wouldn't change (though the UI said it did). I had to cycle power completely to the motherboard (shut down, turn off power supply, then back up) to get it to really change.

I was quite concerned until I read your post. Thanks for this--now it works as expected.

What formula have people been using for the UPPER limits? For instance, what should I set for a fan whose maximum is 1800? I set them at 1900 2000 2100, thinking perhaps that's why they were still running full speed after a reboot. I mean, they were initially set at 23,000, so does it matter?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
@Ericloewe, thank you so much for this thread! You are a life-saver!

I was quite concerned until I read your post. Thanks for this--now it works as expected.

What formula have people been using for the UPPER limits? For instance, what should I set for a fan whose maximum is 1800? I set them at 1900 2000 2100, thinking perhaps that's why they were still running full speed after a reboot. I mean, they were initially set at 23,000, so does it matter?

I imagine the upper limits are irrelevant for the vast majority of PC fans out there. I can imagine a fan controlled by circuitry that fails in a way that spins it up to dangerous speeds, but I cannot imagine why anyone would build such a fan.

I just used the same procedure for both lower and upper limits, but I don't think it'll matter (unless you underestimate it, perhaps).
 

TurtleTech

Cadet
Joined
Apr 18, 2015
Messages
2
Thanks so much for writing this.
I just got my new ASRock C2750D4I motherboard.
Apparently there is a (well I dont want to call it bug.. Oh hell a BUG!) where if you goto firmware update and then cancel when it asks you to upload the firmware the passwords for all the accounts are messed up.
So bottom line is that you are then unable to login to the IPMI anymore with any password. So this is how I fixed it with ipmitool (in case someone else has the issue)..

From freenas root shell:
  1. # kldload ipmi.ko
  2. # ipmitool user list 1
  3. # ipmitool user set password 3 foobar
Explanation:
  1. This loads the ipmi module
  2. List all the users - The "1" is a channel and this was the only one that seemed to work for me.
  3. This will set the password with the "ID" of 3 to foobar. Replace the 3 with whatever ID is appropriate from the output of command #2.


Password reset completed. Thank you !
 

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
Thanks so much for writing this.
I just got my new ASRock C2750D4I motherboard.
Apparently there is a (well I dont want to call it bug.. Oh hell a BUG!) where if you goto firmware update and then cancel when it asks you to upload the firmware the passwords for all the accounts are messed up.
So bottom line is that you are then unable to login to the IPMI anymore with any password. So this is how I fixed it with ipmitool (in case someone else has the issue)..

From freenas root shell:
  1. # kldload ipmi.ko
  2. # ipmitool user list 1
  3. # ipmitool user set password 3 foobar
Explanation:
  1. This loads the ipmi module
  2. List all the users - The "1" is a channel and this was the only one that seemed to work for me.
  3. This will set the password with the "ID" of 3 to foobar. Replace the 3 with whatever ID is appropriate from the output of command #2.


Thank You!!! This just happened to me and I was about to do it manually with a USB drive, this was far easier!
 

STREBLO

Patron
Joined
Oct 23, 2015
Messages
245
When I try to make changes it all works and then updates the sensor values but my fans don't actually seem to change the new values. My fans just keeps spinning up and down . it even says that I've changed them when I query the sensor values . Do you have to do something else after updating them to make the change work?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Did you tried to power cycle?
 
Status
Not open for further replies.
Top