Supermicro fan control, IPMIUTIL or FEEIPMI?

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
It's not a classic rack-mount chassis, it's more like a big cubic desktop case with all the system in a rack who takes half the space of the case (the other half is used to trap the noise while letting the air flows).
...
The air is taken on the top rear through an air filter, sucked by the fans on the top separation (not represented on the 3D view), blown at the front of the drives and above the MB, then it passes through the rack floor (by holes under the MB) and is exited at the bottom front.
Interesting. I see those fans are only 3-wire so you are limited to voltage control so with the stock control only 64 discreet speeds (6-bit) and only on the System header (which is translated to the FAN-A I suspect).
Though the case is pressurized by the fans, exhausting the heat out the bottom appears to be counter-intuitive (heat rises).
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Nop, they are 4 wires PWM fans (Arctic F12 PWM) ;)

Yeah, I had to compromise on that, but I think it's no big deal with forced air flow (it matters primarily when you use natural convection or when you have big installations like a datacenter for example).

I'll add 3 fans beside the 3 existing ones (they will be right above the drives, the existing ones are above and in front of the drives) because when I added the 4 drives to the 4 existing ones I didn't think it would change much the temps because of the rather large space in-between the drives, but it's the case surprisingly.
 
Last edited:

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
For your reading pleasure here is the data sheet for the Nuvoton NCT6776D I/O controller used on the X10SL7-F and others.

Of note is that the unit supports 3 temperature inputs:
  • CPU (can be a discreet input or from the CPU itself)
  • System (analogous to the System on the motherboard)
  • AUX (which I suspect is mapped to the "peripheral" or "I/O" on the motherboard)
These inputs can be Thermistors or Thermal Diodes. I suspect the former is implemented as on one of the FAQ posts at SMs website they refer to the Sys and Aux devices as "RTx" and "RTy".

and, in fact, a quick look with a macro lens enabled me to find two discrete components:
RT-1.jpg

"RT1" this is located at the rear of the motherboard just between the SAS ports and the LSI2308 and is likely the "System" sensor
RT-2.jpg

RT2 is located at the front of the motherboard between the ASPEED BMC and the I/O slots and so is the "Peripheral" or "Aux" sensor.

The Controller Supports 5 outputs, one controlled by CPU, one by System and three by the AUX sensor.

The Super I/O requires FAN drivers to shape & pull-up the output signals and translate the 0-2.048V Voltage signal to X-12V.

I have not yet located those but I suspect they likely use something like a Nuvoton NCT3941s.
 
Last edited:

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Wow, there is some serious reverse engineering going here, I like that :D

I've seen that it may use some kind of closed loop control for the fans ? it's not good news for me if it is active (because I plan to emulate a fixed RPM). But I'm thinking, what if I emulate nothing? do you think the MB won't shutdown the server if there is no fans connected? after all, somebody might want to not use the MB fan headers and power the fans directly from the PSU so it shouldn't be a problem, no?
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Wow, there is some serious reverse engineering going here, I like that :D

I've seen that it may use some kind of closed loop control for the fans ? it's not good news for me if it is active (because I plan to emulate a fixed RPM). But I'm thinking, what if I emulate nothing? do you think the MB won't shutdown the server if there is no fans connected? after all, somebody might want to not use the MB fan headers and power the fans directly from the PSU so it shouldn't be a problem, no?
The fan headers may freak out if they don't see any pulses from the tachometers on fans but the Motherboard should not shutdown until the fail-safe trip occurs which will only happen when the junction temperature of the CPU starts getting critical. I also expect that the FAN FAIL LED will be asserted and there will be alarms in IPMI and traps in SNMP generated that you may want to suppress.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Hmm... not that great. I think I'll do some tests to see what happens. Thanks ;)

As an aside, I can't find any info on the minimum duration the power button must be closed to be detected, do you know this by chance?
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Hmm... not that great. I think I'll do some tests to see what happens. Thanks ;)

As an aside, I can't find any info on the minimum duration the power button must be closed to be detected, do you know this by chance?
Default is instantaneous but can (and should :eek: )be changed to 4s delay in BIOS
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Yep, already changed ;)

But I wanted to talk about a short press (that initiate a soft shutdown), I guesstimated the minimum duration needed to about 100 ms and I've used a pulse of about 300 ms in my controller to be safe, but I've still a doubt if it's enough or not.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
I find that if I can get a look at the data sheets for the modules and read them first when I start digging into the IPMI, BIOS and SNMP I have a better idea of which hardware limitations lead to some of the interface options and what the various "knobs" you can set actually do.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Don't bother to search, it's not critical (at worst I need to change a resistor, that's all), thank you anyway ;)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Yep, already changed ;)

But I wanted to talk about a short press (that initiate a soft shutdown), I guesstimated the minimum duration needed to about 100 ms and I've used a pulse of about 300 ms in my controller to be safe, but I've still a doubt if it's enough or not.

You probably want to leave at default. Default allows the OS to register an ACPI event (power button pressed) and act accordingly. FreeNAS performs a clean shutdown on an ACPI power button press. Immediate is at the motherboard level and just halts the system.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
The default was "immediate" for me and obviously I wanted the normal operation (< 4 sec = soft shutdown by the OS, > 4 sec = hard shutdown by the MB), that's why I changed it ;)
 
Last edited:

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
Surely Supermicro is not thinking like a desktop builder (because it isn't one) when it suggests that the CPU fan should be connected to Header A: there must be some reason for the designer of a server motherboard to recommend connecting the CPU fan to Header A.

It' s true that some of the Supermicro fans (the small ones) run at very high speeds (even up to 22,500rpm), but their larger fans run at slower speeds; e.g., the max. for their 120mm fans is 4,000rpm. And some of those smaller high-speed fans are still not moving as much air as the larger, slower ones.

My FreeNAS system is built in a tower case with 3 x 120mm and 1 x 200mm case fans connected to Fan Headers 1-4 and currently running at 400-600rpm. The CPU fan connected to Header A is running at 1100rpm. The PCH temp. is 39C, the CPU temp. 31C, the DIMM temps. 21-22C. Fan mode is set to Optimal.

The drive cages have their own fans, speed unknown; the drive temps. range from 25-30C.
Based on a Supermicro FAQ http://www.supermicro.com/support/faqs/faq.cfm?faq=13433 (admittedly for a different motherboard), I am now satisfied that the CPU fan should in fact be connected to FAN1. After making that change and connecting the fan in the side of the case to FANA, the CPU fan is running at 1500rpm, the 200mm case fan at 600rpm, and the 120mm fans at 800-900rpm. The temperatures are about the same as before. I did have to change some fan threshold settings using IPMITOOL.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Well I finally got my CPU so I'm able to start playing umm working on getting things ready for the build.

  1. With respect to setting the power button for "instantaneous" versus "4 second delay" I was in the 4 second delay camp but noticed that shortly after changing this in the BIOS that the power off function in the IPMI web interface no longer shut the system down. On changing back to instantaneous I was able to use the power down function again via the IPMI web interface. This is reproducible in my case. I tried it several times and got the same behaviour. So at the risk of incurring the wrath of the resident grinch for violating one of the prime directives as set out in his hardware config guide I humbly suggest that instantaneous be used at least during testing.
  2. I have 4 Cougar CF-v12HP 120mm fans installed in my Fractal Design Define R4 case in a push-pull configuration. The two in the front pushing air into the case through the drive bays and one at the top rear and the one at the back of the case pulling (exhausting) air out. Though the fans are rated for 800-1500 rpm (downhill, with a tail-wind, and a moon in the third phase of Venus and with 64-byte UDP packets) I observed in initial testing a measured speeds of 500-600rpm at the minimum speed and 1500-1600rpm maximum with no apparent rhyme or reason. Both front fans appear to run fast at the maximum so I though that it might be due to back pressure due to the front grill and little tiny slots between the HDDs but on the minimum speed side one front fan and one rear run at 500rpm and the others at 600. Something about the best laid plans of mice and men ... So my intention in buying these 120mm fans as they run at 800rpm minimum and thus were not subject to the cycling phenomenon was proven incorrect. So I ran the tools as outlined here and played with the thresholds. Observation 1. The threshold settings are in increments of 100rpm. Observation 2. In playing around with the settings and rebooting whilst flashing BIOS updates I eventually got to the point where the fans were running at full speed and dicking around with the 4 obtuse mode settings had no effect nor were any alerts present in the logs. However ... I eventually noticed the annoyingly bright power LED on the case blinking at me and remembered that I had connected it to the OH/ps fail/fan fail led rather than the power LED as I know when the damn thing is powered on. The 1 Hz blink rate indicated that the BMC had decided I had a fan failure. So I rest the mode back to "optimal" and had to reboot the IPMI ( as opposed to the system) and the speeds reverted to nice and quiet.
  3. I'm still playing with the thresholds and temperatures but initial findings are that the system sensor shows about 42 deg C idle whereas the periphial reads about 5 deg C cooler. I was concerned about this as this sensor is adjacent to the HDD cages whereas the periphial sensor is by the expansion slots (which are not populated). Thinking about this I realized that the sys sensor on this board is less than 2cm under the LSI2308 which, as others have noted, runs at high temperatures. So I suspect there is some localized heating. Once I finish the memtest86 tests and the CPU torture tests and finally get to the HDD tests I should be able to get some SMART readings off the lower HDDs which should provide further insight.
Currently I'm 8 hours into the 1st pass of memtest86 and the temperatures are nice and low (CPU 47-50, RAM 35-37, PCH 37-40, aux 33-35 and, sys 39-42) with the fans running at the lowest speeds (500-600rpm) and I would estimate under 25dBm noise in a ambient of 19.5 - 22.5 deg. C. I'm noting temps as I go along as well as power consumption at the wall and will post these later but that may not be for a while as I'm off to Florida for a week tomorrow evening so I'll have to shut the system down while I'm away.
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Power on/off works fine for me, with the four second delay option.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,449
Well I finally got my CPU so I'm able to start playing umm working on getting things ready for the build.

I'm noting temps as I go along as well as power consumption at the wall and will post these later but that may not be for a while as I'm off to Florida for a week tomorrow evening so I'll have to shut the system down while I'm away.
You could take your system with you to Florida and run memtest there. If it passes the torture test there with the nice temperature and high humidity content, you should definitely have a reliable system.
Is it Spring break season already?
 
Last edited:

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Power on/off works fine for me, with the four second delay option.
Tested it again and with BIOS ver 2.00 (as shipped) with 4-sec delay enabled the Web IPMI interface Power Off function does not work. Changing to "Instantaneous" fixes that. Once I get back from Florida I will see if there is a BIOS update but I doubt it as the "current" BIOS for the X10SL7 is "x10sl74_424" indicating a date of 24 April 2014 (7_424 Y_MDD) which agrees with the datestamp of the image in the zip file. Failing an update fixing this I will open a support case with SuperMicro.
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
Tested it again and with BIOS ver 2.00 (as shipped) with 4-sec delay enabled the Web IPMI interface Power Off function does not work. Changing to "Instantaneous" fixes that. Once I get back from Florida I will see if there is a BIOS update but I doubt it as the "current" BIOS for the X10SL7 is "x10sl74_424" indicating a date of 24 April 2014 (7_424 Y_MDD) which agrees with the datestamp of the image in the zip file. Failing an update fixing this I will open a support case with SuperMicro.
One of the items in the Change List for BIOS rev. 2.0 is:

'Correct BIOS setup item "Power Button Function" behavior'

but I don't know what the problem was that they were fixing (trying to fix?).
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
One of the items in the Change List for BIOS rev. 2.0 is:

'Correct BIOS setup item "Power Button Function" behavior'

but I don't know what the problem was that they were fixing (trying to fix?).

I believe the four second/instant options had their names/descriptions switched.
 

Z300M

Guru
Joined
Sep 9, 2011
Messages
882
I believe the four second/instant options had their names/descriptions switched.
I still have BIOS 1.1, but I did update the IPMI firmware to 1.42, but I have discovered that "shutdown" in FreeNAS, whether from the GUI or from the console menu, does shut the machine down (confirmed by the "No Signal" message), but it almost immediately powers up again -- not much good when the UPS initiates a shutdown.

Supermicro Tech Support says that it may be necessary to update the BIOS when the IPMI firmware is updated, so I guess that's what I need to try.
 
Top