CPU Cooling fan (quiet)

[jyavenard]

So far, I've tried with 3 PWM fans or controller: they all behaved the same... as soon as the speed goes below the threshold (450rpm) then the other fans start going full speed (my guess that's to compensate) and the fan in question goes full speed too.

It does the same thing when you remove one fan while the machine is running, the other fans go full speed, until you plug back the fan (the fans in the supermicro chassis are hot swappable)

the noctua fans only works properly when the fan mode is set to full speed.
Or, like I mentioned earlier if the CPU temperature is high enough so the speed of the noctua is over 450rpm.

When I had the CPU at 68 degres, the MB put the fans at 900rpm and they work just fine at that speed.

The issue is whenever the speed fall below the threshold.
Googling shows that this is a common behaviour with the supermicro boards.

So long as the fan run at quite a high nominal speed: things are fine. As when running at a lower setting it will still be above the minimum threshold.

the noctua I have, have a maximum speed of 1800rpm, when the supermicro board makes them run at low speed: if it goes below their set minimal threshold: this is when problem starts.
Problem, it seems the supermicro ipmi doesn't seem to let you change the threshold value: they are hard set.

I seriously doubt noctua is the kind of fan manufacturers that wouldn't follow the specs....

 

[cyberjock]

I seriously doubt noctua is the kind of fan manufacturers that wouldn't follow the specs....

I totally agree. I don't own any personally, but they are well respected in the air cooling community.

Normally I'd say the same thing about Supermicro, but there are some that aren't a fan of Supermicro.(pun not intended).

I'm going to order http://www.newegg.com/Product/Product.aspx?Item=N82E16835608028 (300RPM minimum). That should definitely cause problems, which is exactly what I want to check out.

 

[jyavenard]

I'm going to order http://www.newegg.com/Product/Product.aspx?Item=N82E16835608028 (300RPM minimum). That should definitely cause problems, which is exactly what I want to check out.

that's the one I put in my SM chassis: 3 of them...

at full speed, the only thing I hear now is the intel fan...
(I also put two of the non PWM version in the EATON UPS too :) ): the eaton was a pain, they use non standard connector for their fans, so I had to take my soldering iron. But now I can't hear the UPS either.

Unfortunately, I had to put back the intel fan and remove the SM fanless CPU cooler: with the 3 noctua + fanless CPU cooler: at idle speed I would get temps of 52 degres, and when compiling freenas it would jump to 69 degres. With the intel active cooler, idle is 35 degres and below 60 degres when compiling freenas.

In my second chassis, I have kept the original 7900rpm original 80mm fan which are noisy as, but with the same fanless CPU cooler: that one idle at 32 degres in the same room.

that chassis will go into a place with a basement: so noise isn't a problem.

As soon as I've resolved this fan threshold issue: I'm going to have an almost silent setup, despite using server grade gears.
Waiting for supermicro to answer my technical query , will see what they say (but don't have much hope)

 

[jgreco]

The BMC interprets a missing or out-of-spec fan as failed. The system is designed for a rack mount chassis and this causes fan fail to illuminate (which you probably don't have wired to an LED, tsk tsk) and the remaining fans to rev full. In the typical SM chassis design, usually several of the front bulkhead fans are ducted towards the CPU heatsink, so a single fan failure does not result in an immediate disaster.

From our build notes:

Code:

For large fans in 4U cases, low fan speeds may alarm IPMI.
 
Fan speed thresholds can be set via IPMI, use ipmiutil on linux
 
/usr/sbin/ipmiutil sensor -N <ip> -U <user> -R <pass> -i 0x8a -n 41 -l 0x04 -h 0x20
 
0x8a is fan1, 41 is sensor, 0x04 is a slower fan low limit
0xcd is fan2

This was a specific change made for a specific board and chassis here and may need to be different for your deployment. The setting is persistent so RECORD WHAT WAS THERE BEFORE if you might need to change it back. I do not recall how we came upon the 0x8a and 0xcd so some Google-fu might be helpful.

 

[jyavenard]

my query just got answered by the freeipmi folks (they wrote the utility to read and write to the ipmi device which in particular would let you change the threshold).

Their answer:

There is nothing in the IPMI spec that says changes (SDR records) need to be committed(written) to the device.

Infact many implementations have the SDRs in eprom and can not be reprogrammed.

in further email exchange, he confirmed that those values are read-only on the SM board :(

 

[jyavenard]

This was a specific change made for a specific board and chassis here and may need to be different for your deployment. The setting is persistent so RECORD WHAT WAS THERE BEFORE if you might need to change it back. I do not recall how we came upon the 0x8a and 0xcd so some Google-fu might be helpful.

you usually get those values using the pwmcontrol utility. It scans the sensors and find the fans ; from that point it runs a little wizard that goes through all the values and return the rpm speed.

It's very handy, and then using the fancontrol daemon, you can fully override the bios/ipmi and set what the minimal fan speed is going to be, when to run the feel at lowest setting and from what temperature point to set the maximum speed.

It works brilliantly; unfortunately, that series of kernel modules do not exist in FreeBSD, and my attempt to port them lead me to a can of worms and I stopped after a couple of days. Instead I bought quieter fans :)

for more info about pwmcontrol and fancontrol: https://wiki.archlinux.org/index.php/Fan_Speed_Control

 

[jyavenard]

The BMC interprets a missing or out-of-spec fan as failed. The system is designed for a rack mount chassis and this causes fan fail to illuminate (which you probably don't have wired to an LED, tsk tsk) and the remaining fans to rev full. In the typical SM chassis design, usually several of the front bulkhead fans are ducted towards the CPU heatsink, so a single fan failure does not result in an immediate disaster.

From our build notes:

Code:

For large fans in 4U cases, low fan speeds may alarm IPMI.
 
Fan speed thresholds can be set via IPMI, use ipmiutil on linux
 
/usr/sbin/ipmiutil sensor -N <ip> -U <user> -R <pass> -i 0x8a -n 41 -l 0x04 -h 0x20
 
0x8a is fan1, 41 is sensor, 0x04 is a slower fan low limit
0xcd is fan2

This was a specific change made for a specific board and chassis here and may need to be different for your deployment. The setting is persistent so RECORD WHAT WAS THERE BEFORE if you might need to change it back. I do not recall how we came upon the 0x8a and 0xcd so some Google-fu might be helpful.

AWESOME... you're the man !

Thanks for that... I managed to get something like I wanted...

getting ipmiutil to compile turned out to be a pain, but I got there...

on my system I had:

02a2 SDR Full 01 01 20 a 04 snum 41 FAN1 = 07 OK 525.00 RPM
02e5 SDR Full 01 01 20 a 04 snum 42 FAN2 = 00 Absent 0.00 na
0328 SDR Full 01 01 20 a 04 snum 43 FAN3 = 11 OK 1275.00 RPM
036b SDR Full 01 01 20 a 04 snum 44 FAN4 = 0e OK 1050.00 RPM
03ae SDR Full 01 01 20 a 04 snum 45 FANA = 09 OK 675.00 RPM

so for me, fan1 is 0x2a2, sensor 41
fan2: 0x2a5, sensor 42
fan3: 0x328, sensor 43
fan4: 0x36b, sensor 44
fan5: 0x3ae, sensor 45

I didn't touch fan4 as that's the intel CPU fan, and it works just fine

./util/ipmiutil sensor -N 192.168.10.251 -U ADMIN -R "password" -n 41 -i 0x02a2 -l 0x04 -h 0x20
./util/ipmiutil sensor -N 192.168.10.251 -U ADMIN -R "password" -n 43 -i 0x0328 -l 0x04 -h 0x20

./util/ipmiutil sensor -N 192.168.10.251 -U ADMIN -R "password" -n 45 -i 0x03ae -l 0x04 -h 0x20

a threshold of 0x4 is 150rpm failure low and 220rpm critical low, 0x20 is 2475rpm critical high and 2550rpm failure high: which suits me just fine...

the CPU temperature has now increased from 35 degres to 44 degres which is still within spec for idle temp..
Will compare how it goes for the disk temperature..

it's good that I have the two chassis side by side, can compare what it's supposed to be with the original design...

 

[jgreco]

my query just got answered by the freeipmi folks (they wrote the utility to read and write to the ipmi device which in particular would let you change the threshold).

Their answer:



in further email exchange, he confirmed that those values are read-only on the SM board :(

Okay, so I'm going to point a few things out here.

What this seems to be implying is that I'm an idiot and a liar, and on top of that that our shop is crap. There isn't another realistic way to reconcile your "correcting" me by stating that someone else told you the values are read-only on the SM board. But the reality is that they told you something relative to some past experience that may involve a different board series, BMC, firmware revision, or other thing, or they might even be in simple error. We had a very real problem and beat our way through the problem. For us, in an ESXi environment, ESXi will red flag when the BMC reports problems. So misflagging a fan as failed is really annoying. You can see the fix:

On a typical Supermicro board, fans one and two, I believe unaltered (since the build records do not mention modification), relevant output from ipmiutil just pointing at the BMC over IP:

Code:

00cd SDR Full 01 01 20 a 04 snum 41 FAN 1            = 6b OK  8025.00 RPM
        hi-unrec 19125.00 hi-crit 19050.00 hi-noncr 18975.00 lo-noncr 600.00 lo-crit 450.00 lo-unrec 300.00
0110 SDR Full 01 01 20 a 04 snum 42 FAN 2            = 66 OK  7650.00 RPM
        hi-unrec 19125.00 hi-crit 19050.00 hi-noncr 18975.00 lo-noncr 600.00 lo-crit 450.00 lo-unrec 300.00

Notice that the low noncritical threshold is 600; I believe that's the one that ESXi will flag a warning off of (and flags red below low critical). This could actually be tuned upwards but I don't think we're quite that paranoid.

So on the modified system,

Code:

008a SDR Full 01 01 20 a 04 snum 41 FAN 1            = 05 OK  675.00 RPM
        hi-unrec 4590.00 hi-crit 4455.00 hi-noncr 4320.00 lo-noncr 405.00 lo-crit 270.00 lo-unrec 135.00
00cd SDR Full 01 01 20 a 04 snum 42 FAN 2            = 05 OK  675.00 RPM
        hi-unrec 4590.00 hi-crit 4455.00 hi-noncr 4320.00 lo-noncr 405.00 lo-crit 270.00 lo-unrec 135.00

Notice that the lo- thresholds are substantially lower. These fans usually sit at 675 but I think they normally run one step above minimum, and the minimum PWM actually ended up running them just below 600. This happens when all VM's are migrated off.

There is no reason for you to correct me and imply that I'm wrong when I've given you such an intricate command that generally goes right to the heart of your problem.

Fundamentally, you are dealing with a system that drives a fan using PWM to control the speed, and monitors the speed with the tach sensor, and since there is no magic coupling between those things, the BMC has some guidance as to what reasonable values might look like ... but this is only true for the Supermicro fans. The moment you start replacing parts, you have to start doing bench testing to establish new "acceptable" parameters. We've done all the hard work, tried a bunch of stuff, figured out what works. It's ipmiutil. ipmiutil is the most awesome tool for twiddling Supermicro BMC settings via IPMI.

 

[jyavenard]

Okay, so I'm going to point a few things out here.

What this seems to be implying is that I'm an idiot and a liar, and on top of that that our shop is crap. There isn't another realistic way to reconcile your "correcting" me by stating that someone else told you the values are read-only on the SM board. But the reality is that they told you something relative to some past experience that

I wasn't correcting anyone... I actually only saw your answer *after* I posted mine... It just happened you posted while I was typing my answer.

The freeipmi utility, certainly didn't let me modify the value (and be certain that I pointed out to them that they were wrong)

You'll also see, that I tried what you suggested right after, and that yield to a very happy person still up at 1:45AM playing with this stuff.

I've finally achieved what I've been trying for weeks: a system that is almost silent, yet that can handle the occasional time when it needs to deal with extra load

So far from me, from correcting you or suggesting anything like you wrote!

Your analysis of the problem was spot on too: it's not a PWM issue but a threshold's incorrect value.

may involve a different board series, BMC, firmware revision, or other thing, or they might even be in simple error. We had a very real problem and beat our way through the problem. For us, in an ESXi environment, ESXi will red flag when the BMC reports problems. So misflagging a fan as failed is really annoying. You can see the fix:

On a typical Supermicro board, fans one and two, I believe unaltered (since the build records do not mention modification), relevant output from ipmiutil just pointing at the BMC over IP:

Code:

00cd SDR Full 01 01 20 a 04 snum 41 FAN 1            = 6b OK  8025.00 RPM
        hi-unrec 19125.00 hi-crit 19050.00 hi-noncr 18975.00 lo-noncr 600.00 lo-crit 450.00 lo-unrec 300.00
0110 SDR Full 01 01 20 a 04 snum 42 FAN 2            = 66 OK  7650.00 RPM
        hi-unrec 19125.00 hi-crit 19050.00 hi-noncr 18975.00 lo-noncr 600.00 lo-crit 450.00 lo-unrec 300.00

Notice that the low noncritical threshold is 600; I believe that's the one that ESXi will flag a warning off of (and flags red below low critical). This could actually be tuned upwards but I don't think we're quite that paranoid.

So on the modified system,

Code:

008a SDR Full 01 01 20 a 04 snum 41 FAN 1            = 05 OK  675.00 RPM
        hi-unrec 4590.00 hi-crit 4455.00 hi-noncr 4320.00 lo-noncr 405.00 lo-crit 270.00 lo-unrec 135.00
00cd SDR Full 01 01 20 a 04 snum 42 FAN 2            = 05 OK  675.00 RPM
        hi-unrec 4590.00 hi-crit 4455.00 hi-noncr 4320.00 lo-noncr 405.00 lo-crit 270.00 lo-unrec 135.00

I wonder how ipmiutil make the link between the value I enter, and the value it actually writes:
e.g.:

$ ./util/ipmiutil sensor -N 192.168.10.251 -U ADMIN -R "password" -g fan -n 41 -i 0x02a2 -l 0x04
ipmiutil ver 2.92
isensor: version 2.92
sensor_num = 0x41
idx = 0x2a2
Opening lan connection to node 192.168.10.251 ...
Connecting to node 192.168.10.251
-- BMC version 1.10, IPMI version 2.0
_ID_ SDR_Type_xx ET Own Typ S_Num Sens_Description Hex & Interp Reading
02a2 SDR Full 01 01 20 a 04 snum 41 FAN1 = 18 OK 1800.00 RPM
Setting SDR 02a2 sensor 41 to lo=04 hi=ff
GetThreshold[41]: 41 3f 08 06 04 fd fe ff
SetThreshold[41]: 41 07 04 03 02 00 00 00

I asked for 0x04, yet it wrote 04 03 02 (and modified the high threshold, though it still appears as correct).
I haven't managed to go back to the original value (300 / 450),

 

[vitek]

a quick question how would i get the ipmiutil installed on an esxi machine?

 

[jgreco]

Go to "create VM", create your favorite ipmiutil compatible OS (think we've been using a Ubuntu 64bit vm because ipmiutil is available precompiled), attach the VM ethernet to your management network, and go.

 

[jyavenard]

after letting it run overnight, the CPU is now at a constant 10 degres above the other machine.
But more annoyingly the disks themselves run at 11-12 degres above the standard fan: 31 degres vs 42-43

What I need really is having the noctua run at full speed, but the intel fan at a lower rate...

I think the noise with fans full on is bearable enough at this stage.. so going to use the "full speed" mode on SM board

 

[jyavenard]

a quick question how would i get the ipmiutil installed on an esxi machine?

it will compile and run fine in a "ports" jail on freenas itself

I've put the static binaries there:
http://www.avenard.org/files/freenas/ipmiutil-freenas.tar.gz

they will run in freenas directly

 

[jyavenard]

Got the answer to my question from supermicro tech support:

"Fan threshold rpm is hard-coded in the BIOS. There is no way to change it at user discretion."

thank you (not) Supermicro for a useful and 'knowledgable' answer :(

 

[jgreco]

two theoretically knowledgeable parties claiming not possible. interesting.

new company motto ...?

"where the impossible becomes merely mundane..."

 

[cyberjock]

Its probably more like "not possible" as in "we don't want you doing that".

 

[jyavenard]

two theoretically knowledgeable parties claiming not possible. interesting.

the freeipmi folks actually wants to fix this, and I've sent them debug info about it..

 

[jyavenard]

a quick note: freeipmi has been fixed and now works with the SM IPMI.

to compile it, in a ports jail do:

svn co svn://svn.savannah.gnu.org/freeipmi/branches/Release-1_3_0_branch freeipmi

then apply the patches for the freeipmi freebsd patches:

for i in `ls /usr/ports/sysutils/freeipmi/files/patch-*`; do echo $i ; patch -p0 < $i; done

then run configure with:

LDFLAGS=-L/usr/local/lib CFLAGS=-I/usr/local/include LIBS=-lpthread ./configure

and then run make

It's a bit easier to understand the value you set with freeipmi than with ipmiutil, though it's more work to perform the same action.

it's just a matter of time before it's in the ports and as such, automatically added to freenas

 

[benze]

Based on all the wonderful information on this forum I have ordered the following:

ASRock E3C226D2I LGA1150/
Intel Xeon Quad-Core Processor E3-1220 v3
Seasonic 360W 80PLUS Gold ATX12V Power Supply

FRACTAL DESIGN | NODE 304
I am still missing memory and hard drives

Can somebody recommend a good CPU fan that is not noisy?

I'm looking at a similar setup; specifically the ASRock E3C226D2I seems to be the only mobo I can find that is a mini ATX that supports 6 SATA drives & ECC memory. I like the design of the Node 304 but I can't find any other mini ATX mobo that supports ECC.

Are you able to post your complete configuration somewhere? It would definitely help me out figure out the other parts and pieces to make this whole setup work.

Thanks,

Eric

 

[Sir.Robin]

You mean mini-ITX ;)


Sent from my iPhone using Tapatalk