How To: Change IPMI Sensor Thresholds using ipmitool

[Octopuss]

Can anyone explain the thresholds to me? I understand it's for some sort of warnings or to ramp up the fans if the speed drops below whatever, BUT:
1) what is the difference between the three low thresholds
2) what's the purpose of all the upper thresholds?

 

[Ericloewe]

Non-critical is a warning; critical means it's time to do something; and non-recoverable means the fan gets marked as failed, for whatever that's worth.
The upper thresholds are indeed slightly odd, no point in giving them too much thought.

 

[gyrene2083]

Hello All,

Full disclosure, I am not a Truenas user, I am an UNRaid user. I have searched everywhere and even posted on UNRaid and haven't received an answer yet. I am hoping, I could get some help and understanding.

I have a Supermicro X10DRU-i+ , with stock fans (nidec ultraflo v80e12bga5-57) they are rated for 9500rpm (see pic below)

This particular board has Fans 1-8, there are no FanA or FanB. I have my CPU fans on Fan 4, and Fan 6. UNRaid has a IMPI plugin that I tried and it seemed to work for Fans 1-4, but Fans 7, 8 in particular are ramped up to Full speed, oh which is another thing, I don't have standard mode in my IPMI webpage. I just have Optimal, full, and I forget the other.

Ideally I would like to get the fans down to around 3000-3500rpms. I may be able to live with that, but even at optimal it's over 4000rpms and unfortunately, my den where I work is my server room. lol

So here is where I am lost I've seen this

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32 (which would be 50%) that would be 4750rpms

If I entered this;

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x23 (would this be 37%) that would give me around 3515rpms

If the above is true, how would I write that into a script so that when UNRaid boots it sees the changes, and I do apologize for asking here and no one knows, but at least do I have the right concept?

Thank you all in advance.

 

[Kevin Horton]

Hello All,

Full disclosure, I am not a Truenas user, I am an UNRaid user. I have searched everywhere and even posted on UNRaid and haven't received an answer yet. I am hoping, I could get some help and understanding.

I have a Supermicro X10DRU-i+ , with stock fans (nidec ultraflo v80e12bga5-57) they are rated for 9500rpm (see pic below)
View attachment 56204

This particular board has Fans 1-8, there are no FanA or FanB. I have my CPU fans on Fan 4, and Fan 6. UNRaid has a IMPI plugin that I tried and it seemed to work for Fans 1-4, but Fans 7, 8 in particular are ramped up to Full speed, oh which is another thing, I don't have standard mode in my IPMI webpage. I just have Optimal, full, and I forget the other.

Ideally I would like to get the fans down to around 3000-3500rpms. I may be able to live with that, but even at optimal it's over 4000rpms and unfortunately, my den where I work is my server room. lol

So here is where I am lost I've seen this

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x32 (which would be 50%) that would be 4750rpms

If I entered this;

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x23 (would this be 37%) that would give me around 3515rpms

If the above is true, how would I write that into a script so that when UNRaid boots it sees the changes, and I do apologize for asking here and no one knows, but at least do I have the right concept?

Thank you all in advance.

Step one is to ensure that the IPMI fan speed thresholds are correct. Go back to the first post in this thread and read the Resource linked from the first line. Adjust the fan speed thresholds as required.

Step two is to set the fan mode to Full using the web page that you mentioned. This ensures that you won't be fighting for the board for control of the fan speeds as you work through the next steps.

Step three is to experiment a bit to determine if you can in fact control the fan speeds using ipmitool, and to confirm which fans are in each IPMI zone.

Check the fan speeds with ipmitool sdr | grep FAN.

Set the duty cycle in Zone 0 to 50% with ipmitool raw 0x30 0x70 0x66 0x01 0x00 50. Check the fan speeds with ipmitool sdr | grep FAN, and note which fans are in Zone 0. Check the fan speeds a few times over a period of a couple of minutes to ensure that they remain stable. If Zone 0 has the CPU fan, pay attention to the CPU temperature and set the duty cycle back to 100% if required with ipmitool raw 0x30 0x70 0x66 0x01 0x00 100

Set Zone 0 back to 100% duty cycle with ipmitool raw 0x30 0x70 0x66 0x01 0x00 100

Set the duty cycle in Zone 1 to 50% with ipmitool raw 0x30 0x70 0x66 0x01 0x01 50. Check the fan speeds with ipmitool sdr | grep FAN, and note which fans are in Zone 1. Check the fan speeds a few times over a period of a couple of minutes to ensure that they remain stable. If Zone 1 has the CPU fan, pay attention to the CPU temperature and set the duty cycle back to 100% if required with ipmitool raw 0x30 0x70 0x66 0x01 0x01 100

Set Zone 1 back to 100% duty cycle with ipmitool raw 0x30 0x70 0x66 0x01 0x01 100

Step 4 is to determine the minimum allowable duty cycle for your HD fans. The fans will stall if the duty cycle is set too low. The fans in one of my NASs stall if the duty cycle is less than 30%. The other NAS is good down to 20%.

If the HD fans are in Zone 0, try different duty cycles with ipmitool raw 0x30 0x70 0x66 0x01 0x00 XX where you replace "XX" with the desired duty cycle. Watch the fan speeds for a few minutes at the lowest setting to be sure they don't stall.

If the HD fans are in Zone 1, the command to set the duty cycle is ipmitool raw 0x30 0x70 0x66 0x01 0x01 XX where you replace "XX" with the desired duty cycle.

If the results of the above testing show that you can in fact control the HD fan duty cycle, then you need a script to do the job. You may be able to use one of the following scripts as a starting point.

PID control loop shell script
Hybrid CPU & HD Fan Zone Controller
PID control loop in Perl

If you want to continue this discussion it would be better to start a new thread.

 

[gyrene2083]

Thank you so much for that write-up. I will dive into this and let you know how I fare. Thanks again.

 

[MatthewC]

Anyone have advice and/or guidance on how to set sensor thresholds that are disabled for modification by the OEM? I have been unsuccessful with this guide, any of the IPMI tools (free, open, utils, tool, etc), via the IPMI web interface, or changing the sensor.conf file It seems that I may be be able to do it through raw commands but I haven't been able to figure it out. The BMC keeps preventing me from doing it.

If it is helpful at all, I am using a TYAN S8253-2T motherboard.

 

[jgreco]

Anyone have advice and/or guidance on how to set sensor thresholds that are disabled for modification by the OEM? I have been unsuccessful with this guide, any of the IPMI tools (free, open, utils, tool, etc), via the IPMI web interface, or changing the sensor.conf file It seems that I may be be able to do it through raw commands but I haven't been able to figure it out. The BMC keeps preventing me from doing it.

If it is helpful at all, I am using a TYAN S8253-2T motherboard.

You'd probably have to hack the BMC firmware and install a modified version. What sensor thresholds do you want to modify that are disabled by the BMC?

 

[MatthewC]

I want to add an UNC value for the memory channels. Right now the only value I can modify is the UCR value, the rest are non-modifiable. The reason I want to add an UNC value is because the entire fan control system is based on mainly the cpu temps, so if I run a memory intensive task at load, although the BMC can see the memory channel temperatures rising, it doesn't take any action to respond to them until they hit the UCR value at which point it's too late to cool the channels down fast enough before the system shuts down. The CPU temps never get above 55, even at full load, if the system is running at 100% RPMs. The current UCR value for the memory channels is 85. The ram temperatures are getting high because I am not running the board in a chassis that it was designed for, very tight airflow channels with very high rpm fans. The modules are also 256 GB each.

TYAN support did provide some guidance on how to use raw in IPMItool to enable/disable the smart fan control algorithm to set new fan thresholds (using the 0x2e command), so I figured there might be a way to instruct the BMC to enable non-modifiable fields into modifiable ones. However, I couldn't get most of their raw command examples to work for me. I've also tried reading through the IPMI user manual to get a better understanding of how to use the raw command interface, but without success so far. I've tried reaching out to support again but have't heard back in awhile.

 

[MatthewC]

I want to add an unc value for the memory channels. Right now the only value I can modify is the ucr value, the rest are non-modifiable. The reason I want to add an unc value is because the entire fan control system is based on the cpu temps, so if I run a memory intensive task at load, although the BMC can see the memory channel temps rising, it doesn't respond to them until they hit the ucr value, at which points it's too late to cool them down fast enough before the system shuts down. The CPU temps never get above 55 even at full load. The current ucr value is 85. The ram temperatures are getting high because I am not running the board in a chassis that it was designed for, very tight airflow channels with very high rpm fans. The modules are also 256 GB each.

TYAN support did provide some guidance on how to use raw in ipmitool to enable/disable the smart fan control algorithm and to set new fan thresholds, so I figured there might be a way to instruct the bmc to enable non-modifiable fields into modifiable ones. I couldn't get most of their raw command examples to work for me and I tried reading through the ipmi user manual to get a better understanding of how to use the raw command interface but without success so far. I tried reaching out to them again but have't heard back in awhile.

The system can keep everything perfectly cool if running at 85-100% rpms as a baseline, but it wouldn't be anywhere near necessary to run them that high if I could get the fans to ramp up sooner in response to the memory channel temperatures rising.

 

[MatthewC]

I should also say the ucr value of 85 is the only value set. The rest are null.

 

[Ericloewe]

Yeah, I suspect you’re unlikely to find a solution short of the shotgun that is hacking around the BMC firmware.

 

[MatthewC]

Bummer, that's quite annoying, such a simple change made so complicated. How does one even go about doing that?

 

[Ericloewe]

It's something of an open-ended problem. Could take anything from "carefully massaging the configuration" to "full-on replacement of the firmware".

The good news is that they're probably doing minimal rebadging and adaptation of AMI's MegaRAC platform (absolutely ubiquitous on the ASpeed 2400/2500/2600 family), so there's bound to be a bunch of material out there on how to go about such tasks.

 

[MatthewC]

I am not sure what you meant by "carefully massaging the configuration" but after spending some more time reading about this it looks like it might be possible to get what I want by make additions/adjustments to the SDR? Is this what you meant? Seems like it's easy to make a mistake doing so and I am wary to mess around with it.

 

[akkk44]

Anyone have advice and/or guidance on how to set sensor thresholds that are disabled for modification by the OEM? I have been unsuccessful with this guide, any of the IPMI tools (free, open, utils, tool, etc), via the IPMI web interface, or changing the sensor.conf file It seems that I may be be able to do it through raw commands but I haven't been able to figure it out. The BMC keeps preventing me from doing it.

If it is helpful at all, I am using a TYAN S8253-2T motherboard.

Very similar questions here: I have a TYAN Barebone B8252, and BMC keep giving a faulty temp reading of the NVME U.2 SSD. (I know it is faulty cause I tired putting the drive through a U.2-PCIE adaptor and it works perfectly fine)

I tried using the IPMI raw command to disable the sensor but BMC reports failure, also tired delsdr command form supermicro, no luck either. I have not tried freeIPMI (mainly ipmi-sensors-config). After reading your post I think that will also be useless.

Do you have any idea about a possible way to disable a sensor on Tyan board, after all the researches?

Thanks in advance

 

[Ericloewe]

Can't you just edit the thresholds to silence the error?

 

[akkk44]

Can't you just edit the thresholds to silence the error?

Tried editing threshold in the BMC GUI interface and through sensor thresh command, none works

 

[Alkaid_c]

It worked, but the guide is not complete. I am on X10SDV-4C-TLN2F with the latest BIOS (version 2.3). After I typed the command to change the lower threshold of FAN3, it immediately went to full speed. Soft restart cannot fix it. I unplugged and plugged in the power cable, then it finally turned to normal.

 

[samarium]

I chased down a whole bunch of forum threads earlier in the year, here, serverthehome, anywhere I could find references, and constructed my own script for setup for the IPMI, and for enabling/tuning on startup, hooked thru as /etc/rc.local. Let me know if you are interested. This is for an X10SLM, and also works on an H11SSL. Doesn't as yet cater for CPU getting hotter than normal, phase2 : - )

 

[Ericloewe]

It worked, but the guide is not complete. I am on X10SDV-4C-TLN2F with the latest BIOS (version 2.3). After I typed the command to change the lower threshold of FAN3, it immediately went to full speed. Soft restart cannot fix it. I unplugged and plugged in the power cable, then it finally turned to normal.

Not always necessary, it's more of an intermittent bug.