Fan spin up every few seconds, despite changing fan threshold.

[sebkar]

Hi.

My motherboard thinks that one of my fans stops working every few seconds (fan speed=0) which makes it ramp up to full speed for a second or two. This is extremely annoying since I have the server in my office during the winter time. I have removed the chassi panels and made sure that the fan does indeed not spin down.

The thing is that I've had this problem before but I made it stop by changing the thresholds for when the motherboard should ramp up to max speed. Now the problem is back. I tried changing the thresholds again but without any success.

I even set them all to 0, which the MB confirms, and yet it doesn't work.

I am however pretty dumb when it comes to truenas, or pretty much anything that isn't basic windows, so maybe I'm missing something obvious?

Motherboard: Supermicro X11SSM-F
Fan: Unknown Noctua aimed for regular PC´s. If it for some reason is important to know the exact model I can look it up.
Truenas version: 13.0-U3.1

Thanks!

 

[emk2203]

Motherboard: Supermicro X11SSM-F? Nimm die fan scripts.

Wenn das nicht hilft: Falscher Lüfter? Kaputte Geschwindigkeitserkennung? Vielleicht mal zur Probe mit einem anderen Lüfter ersetzen.

 

[sebkar]

Update:
I read that I could have to shut the machine off, unplug the chord and then wait a while for the settings to really stick.

I did that and things changed a bit. Instead of the fan ramping up as soon as it reached its normal speed again (which happened every five seconds or so) it is now less repetative in the way that it can take up to approximately 30 seconds before it ramps up again.

So the main problem still exists (the motherboard thinks that it needs to run the fan on max speed since it thinks it stops working every now and then, despite me having changed the speed thresholds), but it has changed character.

Motherboard: Supermicro X11SSM-F? Nimm die fan scripts.

Wenn das nicht hilft: Falscher Lüfter? Kaputte Geschwindigkeitserkennung? Vielleicht mal zur Probe mit einem anderen Lüfter ersetzen.

Thanks for the tip but I'm reluctant to install scripts for this, unless I really have to. Mainly because I'm sick and tired of being forced to spend an hour, or five, every single time I update truenas and realises that something broke or changed. Every single time.

Adding a script is, I assume, yet another source of failure during an update.

 

[Etorix]

Adding a script is, I assume, yet another source of failure during an update.

No it is a very stable setting if the script lives in a safe place (e.g. your home dataset) and the "task" to call it upon startup is set from the GUI.

Thresholds should be set from the BMC, based on actual values for your fan. Instructions for this are in the discussion thread for the script, which I highly recommand.

 

[sebkar]

No it is a very stable setting if the script lives in a safe place (e.g. your home dataset) and the "task" to call it upon startup is set from the GUI.

Thresholds should be set from the BMC, based on actual values for your fan. Instructions for this are in the discussion thread for the script, which I highly recommand.

Maybe you're right, but so is truenas and yet I'm always ending up frustrated every time I'm trying to do something.

But would the script even help me if the script leans back on the thresholds? I mean, if I change the thresholds with ipmitool and the fan still enters panic mode, would the script change override that?

I have tried to set sensible thresholds and when that tried I tried to set the lower ones to unreasonable low and when that didn't work I set them to 0. Still doesn't work.

 

[sebkar]

Update, maybe:

I found this https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/

A guide on how to set a fixed speed on the fans. Not optimal but for my user case it is more than adequate, so I tried it.

What happens is that the fan speeds are changed for half a second or so and then they are reset again.

Is something broken? What am I doing wrong?

 

[Glorious1]

Fan: Unknown Noctua aimed for regular PC´s. If it for some reason is important to know the exact model I can look it up.

How can you set the fan thresholds if you don't know the specs of your fans? Get your fan model numbers, look up the specifications, and go to EricLoewe's Guide and follow it closely to set your thresholds. If you still have problems after that, post the output of ipmitool sensor list all (in code tags).

You also don't mention what Fan Mode you have set in IPMI > Configuration, which seems like important information. Have you tried changing that?

Although I understand being reluctant about using scripts, I suggest you use spincheck.sh for a few hours or so (after correctly setting thresholds and mode). It only reads and logs fan data, it doesn't alter anything. This may help in diagnosing what is going on. Copy and paste some output here that seems to show any problem occurring.

 

[sebkar]

How can you set the fan thresholds if you don't know the specs of your fans? Get your fan model numbers, look up the specifications, and go to EricLoewe's Guide and follow it closely to set your thresholds. If you still have problems after that, post the output of ipmitool sensor list all (in code tags).

Correct me if I'm wrong but as long as my purpose is to stop my fans revving up every time they falsely report a rpm that supermicro think is to low then the exact specs of the fan isn't important.

At this point I'm so frustrated that I don't want it to ever rev up (until I find a proper solution or change the fan) and that should be possible if I put all the lower thresholds at "0". Or have I misunderstood how it works?

The current fan mode is set to "optimal speed" since that is what most, but not all, people are telling others with similar problem to use. I have however tried all other modes as well.

The output is as follows, the fan in question is FAN3:
CPU Temp | 20.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 95.000 | 100.000 | 100.000 PCH Temp | 18.000 | degrees C | ok | 0.000 | 5.000 | 16.000 | 90.000 | 95.000 | 100.000 System Temp | 14.000 | degrees C | ok | -10.000 | -5.000 | 0.000 | 80.000 | 85.000 | 90.000 Peripheral Temp | 15.000 | degrees C | ok | -14.000 | -9.000 | -4.000 | 76.000 | 81.000 | 86.000 VcpuVRM Temp | 21.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 95.000 | 100.000 | 105.000 DIMMA1 Temp | na | | na | na | na | na | na | na | na DIMMA2 Temp | 14.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000 DIMMB1 Temp | na | | na | na | na | na | na | na | na DIMMB2 Temp | 13.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000 FAN1 | 1000.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000 FAN2 | 1100.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000 FAN3 | 200.000 | RPM | ok | 0.000 | 0.000 | 0.000 | 25300.000 | 25400.000 | 25500.000 FAN4 | 1100.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000 FANA | 1200.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000 12V | 12.256 | Volts | ok | 10.144 | 10.272 | 10.784 | 12.960 | 13.280 | 13.408 5VCC | 4.896 | Volts | ok | 4.246 | 4.298 | 4.480 | 5.390 | 5.546 | 5.598 3.3VCC | 3.299 | Volts | ok | 2.789 | 2.823 | 2.959 | 3.554 | 3.656 | 3.690 VBAT | 3.000 | Volts | ok | 2.384 | 2.496 | 2.580 | 3.476 | 3.588 | 3.672 VCPU | 0.103 | Volts | ok | 0.076 | 0.076 | 0.076 | 1.516 | 1.516 | 1.516 VDIMMAB | 1.182 | Volts | ok | 0.948 | 0.975 | 1.047 | 1.344 | 1.425 | 1.443 5VSB | 4.974 | Volts | ok | 4.246 | 4.376 | 4.480 | 5.390 | 5.546 | 5.598 3.3VSB | 3.231 | Volts | ok | 2.789 | 2.891 | 2.959 | 3.554 | 3.656 | 3.690 VBMC 1.2V | 1.209 | Volts | ok | 1.020 | 1.047 | 1.092 | 1.344 | 1.371 | 1.398 VPCH 1.0V | 1.007 | Volts | ok | 0.872 | 0.890 | 0.917 | 1.043 | 1.052 | 1.070 Chassis Intru | 0x0 | discrete | 0x0000| na | na | na | na | na | na

But please note that the RPM (200) does change pretty much all the time, which of course is the topic for this thread. Also note that I have tried with more sensible thresholds, such as 100, 100, 200 and the default 300, 500, 700 and much else. But without success.

I guess I could use that script when I find the time in a day or three but I already now what it will say. The temperature is good and the fan speed goes from 0 to max (which is around 1.400 it seems like).

(Yes, the temps shown above is most likely correct. During the wintertime we put on an extra shirt instead of having summer heat indoors and when I'm away from my office/gaming room then I have the radiator turned of so it gets even colder.)

However
In a desperate attempt I read someone saying that a reset of the ipmi could fix the problem. I don't know enough to understand if there was any truth to that but I figured there could be no harm trying. After doing that I immediately got two revups in about ten seconds time but after that it behaved nicely for several hours. But when I got back home after being away for two days the problem was back. It now revs up every five seconds or so.

 

[Etorix]

Correct me if I'm wrong but as long as my purpose is to stop my fans revving up every time they falsely report a rpm that supermicro think is to low then the exact specs of the fan isn't important.

How do you expect to set proper fan behaviour without knowing the fan specifications?
That's equivalent to "please help me fix my pool" without knowing the layout, number and model(s) of drives…

The current fan mode is set to "optimal speed" since that is what most, but not all, people are telling others with similar problem to use. I have however tried all other modes as well.

"Optimal" relies on valid lower thresholds. The defaults are reasonable values for server-style fans, not for low-noise Noctua.

But please note that the RPM (200) does change pretty much all the time, which of course is the topic for this thread. Also note that I have tried with more sensible thresholds, such as 100, 100, 200

This is not a valid set of lower thresholds: They must be different values, multiples of 100, and in increasing order.
This set of values tells IPMI that if fan speed drops to the lower "critical" value the situation is already "unrecoverable". Panic is bound to follow.

 

[sebkar]

How do you expect to set proper fan behaviour without knowing the fan specifications?
That's equivalent to "please help me fix my pool" without knowing the layout, number and model(s) of drives…

I'm guessing that it's 50/50 on whether I'm thicker than usual or if you're misunderstanding the problem. So if I'm in the wrong I would appreciate if you explained why as if I was a five year old.

Let's pretend that the optimal thresholds are 0, 100, 200.

How would that stop the fan revving up when the motherboard thinks that it reaches 0 RPM?

This is not a valid set of lower thresholds: They must be different values, multiples of 100, and in increasing order.
This set of values tells IPMI that if fan speed drops to the lower "critical" value the situation is already "unrecoverable". Panic is bound to follow.

I can buy that it should be different values but I see no reason why it must in my case. And beside, when I've searched around for an solution for this problem it is very common to find people who have the same value for two or all of the thresholds and it works for them.

Anyway, I have also tried with different values, including 0, 100, 200.

And btw, between every change i turn everything off, unplug the cord and wait a few minutes before turning it back on. Which is a pita but since it's a hit or miss with supermicro actually changing the settings without doing this then I see no other options. It usually registers the first change or two but any more than that it just gives up.

 

[Glorious1]

The output is as follows, the fan in question is FAN3:

FAN3 refers to a fan header, not to a fan. There may be 0 or more fans connected to the header. If it's just one header acting up, try connecting the fan(s) connected to it to another header, using a Y connector if needed. That may help distinguish between header issue vs. fan issue.

 

[sebkar]

Update:

I updated the bios and during the last 20 hours it has only revved up once according to the log. So still not 100 % perfect but close. Let's hope it continues like that.

I know supermicro explicitly says not to update the bios unless the problem is described in the changelog, which it wasn't, but oh well. I did get a scare after the update when the server didn't want to boot, nothing at all happened when I tried to power it on. But I left it unplugged during dinner time and when I got back it worked fine :)

FAN3 refers to a fan header, not to a fan. There may be 0 or more fans connected to the header. If it's just one header acting up, try connecting the fan(s) connected to it to another header, using a Y connector if needed. That may help distinguish between header issue vs. fan issue.

Thanks. That thought didn't even occur to me, that it could be a connector issue. But hopefully it's solved.

 

[grigory]

I have this same issue with a iXsystems Mini XL+, the unit functioned quietly for about 3 weeks and then started revving 'FAN2' and assertions start appearing in the ipmi log for FAN2.
Before I came across this thread I had found this one https://www.truenas.com/community/t...-few-seconds-about-every-20-25-seconds.96980/
which suggests setting the threshold:

Code:

ipmitool sensor thresh FAN2 lower 300 400 700

Although this did not seem to work. Ultimately that poster RMA'ed the MiniXL+, I'm not sure that is neccessary though.

@sebkar is your machine still quiet after a bios update?

Thanks!

 

[sebkar]

@sebkar is your machine still quiet after a bios update?

Sadly not.

Everything seemed to work fine for almost a week. With the fan revving up only a handful of times due to it falsely reporting 0 RPM according to the log. So not perfect but acceptable.

However, one day during work I got an alert mail from my server that the log was almost full. And to no surprise the log had quickly been filled with entries about that darn fan which again revs up all the time.

I remembered that I had two spare fans somewhere so despite really wanting the quiet Noctua to work I decided to try them out. One was silent fan from BeQuiet and that had the exact same problem.

The other one was another Noctua with PWM but this time one from the "industrial" series. It has for the last hour behaved nicely and is reliably spinning at 300 RPM with no false stops. Not yet at least.

The problem is that its a 140 mm fan and my case only has room for 120 mm. But I rammed it into the case so it's now pulling air from about 20 cm from the drives it should cool. Probably not enough on a hot day but it's 15 degrees Celsius in the office at the moment so it should be fine until next time I order something from the web.

 

[sebkar]

I know Supermicro is aimed towards enterprise so it should come as no surprise but as a regular consumer I'm honestly a bit upset that they care so little that they can't fix something that the cheapest boards on the market don't have a problem with, if they even ever had; slow spinning fans.

Not sure if I have the patience to go with them again when I upgrade my server in a year or two.

 

[WI_Hedgehog]

If a server fan is replaced with an desktop fan there's no guarantee the signaling back to the MB is going to be what the MB expects. Also, you can't expect Supermicro to tune a rackmount board for desktop use any more than you could expect IBM to invest 3 employees into making a PC into a rackmount for a one-off setup.

Anyway, put a server fan back in and set the fan speed to something reasonable for a server room and see if that solves the fan speed issue. (It's a starting point.) If that works change one variable, I'd try lowing the fan speed to what you wanted and seeing if that causes the failure. If not, restore the fan speed to the previous reasonable value and put the Noctua in and see if it signals the board fine at high speed. If not you have a pretty good idea the problem lies in the fan--that fan; until you try another known-to-be-working fan of the same type you don't know if this fan is bad or just doesn't work with that mainboard.

 

[sebkar]

If a server fan is replaced with an desktop fan there's no guarantee the signaling back to the MB is going to be what the MB expects. Also, you can't expect Supermicro to tune a rackmount board for desktop use any more than you could expect IBM to invest 3 employees into making a PC into a rackmount for a one-off setup.

Just to be clear; I haven't replaced anything since it's a home build. And yes, I do realise that most commercial servers have their fans on very high speeds. But that's no justification for developing a motherboard that thinks that any fan that goes below circa 250 RPM is the same as 0 RPM. Because that's what seems to be happening from my limited experience and googling.

I'm not expecting supermicro to spend thousands of man hours to tune their boards for home use. But is it really to much to ask that they put one guy to occasionally work on something as extremely basic as fan control? They have after all taking their time to do a shitty implementation of some kind of fan control so why make something that's actually usable?

I'd try lowing the fan speed to what you wanted and seeing if that causes the failure.

It does, hence this thread.

If not, restore the fan speed to the previous reasonable value

Ok. Let's say that I want it to constantly spin at 400 RPM which would most likely fix the problem. How would I do that?

and put the Noctua in and see if it signals the board fine at high speed.

If I change the fan mode so that it spins at full capacity then yes, everything is reported correctly.

If not you have a pretty good idea the problem lies in the fan--that fan; until you try another known-to-be-working fan of the same type you don't know if this fan is bad or just doesn't work with that mainboard.

It's not the fan, it's the motherboard. I have two "faulty" slow spinning fans and a functional faster spinning fan that supports that. As well as a ton of google searches.

I put faulty in quotation marks because there are nothing wrong with them (tried in a pc).

 

[grigory]

Thanks for the continued responses on this issue. Last week after my post here I opened a support ticket with iXsystems. At first they asked to RMA my unit, suggesting that there might be a hardware issue. While I was awaiting a response from them I tried a few other fan settings. One of them being changing all the fans to the same lower threshold.

Previously I had seen FAN2 in the log and thought it needed to have the threshold changed and so I had done that, but it didnt seem to control the fan.
I ran the following commands (setting the threshold on ALL 4 fans/headers):
ipmitool sensor thresh FANA lower 100 200 300
ipmitool sensor thresh FAN1 lower 100 200 300
ipmitool sensor thresh FAN2 lower 100 200 300
ipmitool sensor thresh FAN3 lower 100 200 300

and then my sensor were reading:
FAN1 | 500.000 | RPM | ok | 100.000 | 200.000 | 300.000 | 25300.000 | 25400.000 | 25500.000
FAN2 | 400.000 | RPM | ok | 100.000 | 200.000 | 300.000 | 25300.000 | 25400.000 | 25500.000
FAN3 | 3300.000 | RPM | ok | 100.000 | 200.000 | 300.000 | 25300.000 | 25400.000 | 25500.000
FANA | 500.000 | RPM | ok | 100.000 | 200.000 | 300.000 | 25300.000 | 25400.000 | 25500.000

After this I have not had any issues and will not send the unit for RMA (ixSystems said if things looked good now then I could wait), its been almost 2 weeks without any fans spinning up.

 

[WI_Hedgehog]

Keeping in mind you asked for an explanation in simplified terms:

Server systems tend to maintain a minimum fan speed because hardware longevity (such as hard drive life) is the primary concern. Noise isn't much of a consideration as long as it doesn't impact efficiency "too much."

Below a certain RPM, industrial fans with heavy industrial windings draw a large amount of current, and of course that's bad. The result is the minimum fan speed is "medium" instead of "low." Remember, industrial fans often work at higher static pressures and therefore are designed to "really push air hard," which is in opposition to a quiet fan design.

Let me address that: In desktop PCs there isn't a concern for how much rack space costs. Rack space is really expensive, so servers tend to be built "really tight" with things in close proximity. Desktops generally have a lot of open space for airflow--look at how tall the CPU cooler is! You won't find that in a rack server that has 250x the computing power! When things are packed tight air has a harder time getting through the tight spaces and "static pressure" increases, therefore server fans have blade designs meant for forcing a lot of air through a rack system 24 hours a day, 7 days a week, 52 weeks a year, for 10 years straight. Desktop PCs, not so much. A gaming PC is run for 4 hours a day Monday through Thursday, maybe 16 hours Friday-Saturday-Sunday, for two years before replacement--that's a huge difference! PCs have lots of space for intake and exhaust fans and they move a ton of volume at low pressure. PCs may idle down to nothing between games and therefore fan speed can be dead silent, whereas servers are always working (otherwise they're losing money), so somehow servers are going to be processing something.

A home NAS will sit idle most of the time, so should be quiet and idle at very, very low power. A small office NAS will have 3 people accessing at any one time on average, and at times eight people will be actively in need of data stored on the server which isn't in the server cache, otherwise it sits idle until the nightly backup hits. A data-center NAS will have 16 to what, maybe 1,000 queries a second? (it really depends on the type of workload) Regardless, if you stuck a gaming rig in a data center as one of the workhorses you can expect the data-center to beat a PC to death in a matter of days--that Western Digital Black drive...fuhgeddaboudit.

So, yeah, if your concern is you can't get the server-room mainboard to reliably run the fan below "medium" speed (whatever the mainboard happens to consider "medium"), par for the course. It just is. Accept it or put a script in that does what you want and pray for the best. Or get a [mini-]server mainboard from Asus that will run down to the "low zone."