Supermicro Xeon D 1521 Overheating on Stress Tests

[chrisolson09]

Hey all, I finally got my build put together and have been running some tests with Ultimate Boot CD to make sure the hardware is in good shape before installing FreeNAS.

Everything checks out except when I run the CPU stress tests my cpu is getting HOT! Like 90C +...and that is even with the case completely open.

I’m using fractal design node 304 case, 6 WD red drives, and based on people’s recommendation i added a fan (Noctua Fan) on top of the heat sink for extra cooling but I’m still running into issues.

I’ve tried flipping the cpu fan around so it’s pulling hot air away from CPU but that didn’t seem to help much. I’ve also had the case fans going on high

You have any thoughts? Any help you guys could offer would be really appreciated!!

Chris

 

[nightshade00013]

More airflow and a better cooler. The OEM coolers sent with cpu's suck. Even the AMD wraith coolers can be bested by an aftermarket one. I do my best to pick up and use hyper 212 evo's and to me they are some of the best yet cheap coolers you can get. The caveat is they are very tall so check to make sure it will fit in your case if you go that route. Otherwise the Noctua coolers are pretty good at a higher cost.

I will say that most of the cpu stress tests are well beyond what you will ever see in a workload unless you are using the box to transcode movies all the time.

 

[MrToddsFriends]

Everything checks out except when I run the CPU stress tests my cpu is getting HOT! Like 90C +...and that is even with the case completely open.

I’m using fractal design node 304 case, 6 WD red drives, and based on people’s recommendation i added a fan (Noctua Fan) on top of the heat sink for extra cooling but I’m still running into issues.

Open case is not the use case you are interested in. The interesting use case is utilizing the airflow created by front and exhaust fans, together with the airflow created by the CPU fan. Of course 90°C is way too hot and I would have performed a very first test with the case open, too.

May I ask which Noctua fan you are using exactly and how you mounted it to the heat sink? For my poor little Avoton (C2750) a Noctua NF-A6x25 FLX used with the U.L.N.A. definitely provides more than enough cooling.

 

[chrisolson09]

thanks guys for your input. This is really baffling me. I've read multiple people who are using this setup and seem to not have a problem at all.

Open case is not the use case you are interested in. The interesting use case is utilizing the airflow created by front and exhaust fans, together with the airflow created by the CPU fan. Of course 90°C is way too hot and I would have performed a very first test with the case open, too.

yeah I definitely know the open case is not what I should be interested in but yeah like you I thought the first tests I would do with an open case. I could try running with a closed case and see what happens but I have a hard time believing it's going to make that big of difference.

May I ask which Noctua fan you are using exactly and how you mounted it to the heat sink? For my poor little Avoton (C2750) a Noctua NF-A6x25 FLX used with the U.L.N.A. definitely provides more than enough cooling.

The fan I'm using is the same as yours except it's the PWM 4pin version. I thought I'd go that route that way the motherboard could control the speed as necessary. Here is the link to the one I got

https://www.amazon.com/gp/product/B00VXTANZ4/ref=oh_aui_detailpage_o07_s00?ie=UTF8&psc=1

Also here is a picture of how I have the fan mounted. I've read about multiple people going this route. I'm just thinking out loud, I could maybe try and get the fan closer to the heat sink

 

[Redcoat]

Also here is a picture of how I have the fan mounted. I've read about multiple people going this route. I'm just thinking out loud, I could maybe try and get the fan closer to the heat sink

Yes, you should get the fan down on the heatsink - no gap - forcing the air flow through through the fins. I assume that you are pulling air upwards (which I believe would be best, but you will be able to test).

I hope that you have an alarm and/or shutdown limit in the case this fan fails - when it would limit the effectiveness of the passive heatsink and aid the "cooking" of your CPU. Whatever, you need to be comfortable with your provisions for this failure scenario.

Closed case testing is a must - it will help identify anything you need to attend to in the air supply to the area you are concentrating on cooling.

 

[chrisolson09]

Yes, you should get the fan down on the heatsink - no gap - forcing the air flow through through the fins. I assume that you are pulling air upwards (which I believe would be best, but you will be able to test).

I hope that you have an alarm and/or shutdown limit in the case this fan fails - when it would limit the effectiveness of the passive heatsink and aid the "cooking" of your CPU. Whatever, you need to be comfortable with your provisions for this failure scenario.

Closed case testing is a must - it will help identify anything you need to attend to in the air supply to the area you are concentrating on cooling.

Redcoat, thanks for your suggestions. I will see about getting the fan down as close to the heatsink as possible and see if that helps. Also, I know at some point I need to do close case testing but do you think a closed case will help my problems at all? If it's getting that hot with a open case woudn't it get even hotter with a closed case?

Also, I'm new to all of this so I haven't specifically setup up a shutdown limit in case the fan fails. Do you have any suggestions on how I could go about doing that? doesn't the board have built failsafes for if it gets too hot?

 

[Redcoat]

Redcoat, thanks for your suggestions. I will see about getting the fan down as close to the heatsink as possible and see if that helps. Also, I know at some point I need to do close case testing but do you think a closed case will help my problems at all? If it's getting that hot with a open case woudn't it get even hotter with a closed case?

No pun intended, but this is a "case by case" thing - as you won't be operating the server open (will you?) then you need to carry out this testing in the intended operating configuration (and location) if there's anything that could alter the local environment in a negative way. I can't speculate on what you are currently experiencing with the case open but I just looked at Stux's build in that case and it looks very constricted with cables, etc., so I reckon it needs the assistance of the case feed and exhaust fans to get a decent supply of air to it. I also see that his fan mount is down on the sink and has a shroud mounted to direct the air ingress (at least I assume it is exhausting upwards).

Do you have any suggestions on how I could go about doing that? doesn't the board have built failsafes for if it gets too hot?

I don't know what your board has - there are plenty scripts in the Resource section that can help you towards setting up what you need. Look at @Spearfoot 's posting - also search for @Bidule0hm 's posting with scripts which has some information easy to assimilate as a starter kit (it's referenced in @Spearfoot 's posting).

 

[chrisolson09]

No pun intended, but this is a "case by case" thing - as you won't be operating the server open (will you?) then you need to carry out this testing in the intended operating configuration (and location) if there's anything that could alter the local environment in a negative way. I can't speculate on what you are currently experiencing with the case open but I just looked at Stux's build in that case and it looks very constricted with cables, etc., so I reckon it needs the assistance of the case feed and exhaust fans to get a decent supply of air to it. I also see that his fan mount is down on the sink and has a shroud mounted to direct the air ingress (at least I assume it is exhausting upwards).

cool, thanks again for all your suggestions! i really appreciate it! That does makes some sense how having the case there could help give a better supply of air. it definitely is tight in that case!... I'll give it a shot! I'll also work on getting the fan down right on the heat sink. I do have it currently exhausting upwards which seems to make the most sense.

 

[Bidule0hm]

Actually blowing on a heatsink is better because the flow is turbulent instead of laminar and the PC are less sensitive to the restriction being on the exhaust side than on the intake side of the fan.

BTW what I can see on the picture is that it's a very small heatsink for a CPU.

 

[Redcoat]

Actually blowing on a heatsink is better because the flow is turbulent instead of laminar and the PC are less sensitive to the restriction being on the exhaust side than on the intake side of the fan.

Yes, in general I agree with you better to blow on heatsink than pull (not to get into "turbulent" and "laminar" here ... hopefully it's always the former in this context inside the heatsink) but with these "simple" fans having too much resistance immediately on their outlet side can significantly reduce their volumetric capability (giving rise to recirculation on the upstream and downstream sides) so that "pull" may work better in some restricted circumstances, I have found from experience (and in my most recent case with my overheating C2750). Hence my suggestion to the OP to test both.
BTW, I looked at Stux's pic again, and his fan is blowing down directly coupled as you suggest it should be.

 

[chrisolson09]

Yes, in general I agree with you better to blow on heatsink than pull (not to get into "turbulent" and "laminar" here ... hopefully it's always the former in this context inside the heatsink) but with these "simple" fans having too much resistance immediately on their outlet side can significantly reduce their volumetric capability (giving rise to recirculation on the upstream and downstream sides) so that "pull" may work better in some restricted circumstances, I have found from experience (and in my most recent case with my overheating C2750). Hence my suggestion to the OP to test both.
BTW, I looked at Stux's pic again, and his fan is blowing down directly coupled as you suggest it should be.

yeah either way I don't think it's probably going to make that big of difference. I'll try closing the case and putting the fan right on top of the heat sink and hopefully I can get the temps that are in the normal range and then I can play with whether it's pushing or pulling to optimize things

 

[Redcoat]

I'll try closing the case and putting the fan right on top of the heat sink and hopefully I can get the temps that are in the normal range and then I can play with whether it's pushing or pulling to optimize things

Very good. Maybe get a bit of that good ole WI 31F, 55% RH, fresh air to feed it to keep things in check ...

 

[MrToddsFriends]

Two hints from other Supermicro mITX / Noctua builds (both in pushing configuration):

1) Noctua NF-A6x25 FLX or PWM on Supermicro X10SDV (Xeon-D):
https://www.servethehome.com/near-silent-powerhouse-making-a-quieter-microlab-platform/

2) Noctua NF-A6x25 FLX on Supermicro A1SAi-2750F (Avoton, my own build):

In my build I used zip ties to mount the fan to the heat sink, which has the indisputable disadvantage that the motherboard has to be dismounted for a fan replacement.

And two questions for the TO:

1) Did you monitor the CPU fan speed when applying CPU load? Did you play with fan speed envelope settings in the motherboard BIOS and/or in IPMI?

2) Possibly hard to answer: Are you able to rule out the possibility of a faulty thermal pad and heat sink assembly on your motherboard? If this is the case a warranty replacement might be appropriate.

 

[chrisolson09]

Very good. Maybe get a bit of that good ole WI 31F, 55% RH, fresh air to feed it to keep things in check ...

yeah if this stupid thing doesn't start cooling i'm going to throw it in the snow bank in my front yard! :P

 

[chrisolson09]

Two hints from other Supermicro mITX / Noctua builds (both in pushing configuration):

1) Noctua NF-A6x25 FLX or PWM on Supermicro X10SDV (Xeon-D):
https://www.servethehome.com/near-silent-powerhouse-making-a-quieter-microlab-platform/

2) Noctua NF-A6x25 FLX on Supermicro A1SAi-2750F (Avoton, my own build):
View attachment 23219

In my build I used zip ties to mount the fan to the heat sink, which has the indisputable disadvantage that the motherboard has to be dismounted for a fan replacement.

And two questions for the TO:

1) Did you monitor the CPU fan speed when applying CPU load? Did you play with fan speed envelope settings in the motherboard BIOS and/or in IPMI?

2) Possibly hard to answer: Are you able to rule out the possibility of a faulty thermal pad and heat sink assembly on your motherboard? If this is the case a warranty replacement might be appropriate.

Thanks for the suggestions. I like the idea of zip tying the fan to the heat sink. I may give that a go.

As far as your questions... I have not monitored the fan speed. I would like to just to make sure that the motherboard is able to adjust the fan speed as necessary. I don't have FreeNAS installed yet though. I've just been using Ultimate Boot CD to run some of the hardware tests and I don't know how (or if I can) monitor the fan speeds through that.

I haven't looked at the BIOS or IPMI. I admit I'm a total newbie when it comes to this. I'm trying to learn as much as I can. So again I really appreciate all the help that you've given.

Finally, I thought about if it could possibly be a faulty hardware but I don't know how to verify if this is the case.

 

[chrisolson09]

So I currently am testing things and things seem to be little better...I am now running a stress test with the case closed and I changed the fan around so it is right on top of the heat sink instead of there being a gap. The fan is pushing the air through the heat sink.

I saw an interesting phenomenon happen...I started the test and it again had temps that climbed to close to 90C (Not quite as hot as previously)...I was about the shut things down when all of a sudden the temps started dropping. They have now leveled out at 79C under full CPU load.

What is everyone's thoughts on this...do you feel this is within a normal range for being under full load or is this still extreme?

 

[chrisolson09]

Alright, I give...I started a different stress test (Mersenne Prime Blend Test) and now it's again hovering right around 90-91C

 

[nightshade00013]

You probably need more airflow through the case with that cooler. With the air blowing down towards the board the air has no place to really go unless you have enough airflow to push the heat out of the case. What ends up happening is the air hits the board and a portion of it is sucked back into the fan. The only time I would use a cooler like that is if it was mounted in a 1U server chassis so a ton of air can move across it or if I had a ton of airflow through a case. You will be much better suited to just pick something up to replace it that can direct the air towards the back of the case. If you CAN'T do that at all you will need to put the maximum number of fans in the case and push air in from the front/bottom and pull it out from the back/top and make sure that you keep as much cabling out of the way as possible so that they air flows through rather than being blocked by stuff.

 

[MrToddsFriends]

I haven't looked at the BIOS or IPMI. I admit I'm a total newbie when it comes to this.

Definitely monitor the fan speed using IMPI, should be accessible via
Server Health -> Sensor Readings -> Fan Sensors
in the IPMI web interface. If you are faced with a Java Version warning after logging in to IMPI, ignore that.

And while you are at it, change the default IPMI password.
https://forums.servethehome.com/index.php?resources/supermicro-ipmi-default-login-and-password.1/

The Fan Sensors view in the IPMI web interface should look similar to this:

In my system FAN2 is the (non-PWM) Noctua NF-A6x25 FLX, FAN1 and FAN3 are two Noctua NF-A9 PWM fans (Node 304 intake fan replacements, quiet and subject to monitoring). Start with the medium setting for the standard Node 304 intake fans (which are not monitored in IMPI).

Available FAN mode settings in the IPMI web interface are accessible via
Configuration -> Fan Mode

 

[chrisolson09]

Definitely monitor the fan speed using IMPI, should be accessible via
Server Health -> Sensor Readings -> Fan Sensors
in the IPMI web interface. If you are faced with a Java Version warning after logging in to IMPI, ignore that.

And while you are at it, change the default IPMI password.
https://forums.servethehome.com/index.php?resources/supermicro-ipmi-default-login-and-password.1/

The Fan Sensors view in the IPMI web interface should look similar to this:
View attachment 23226

In my system FAN2 is the (non-PWM) Noctua NF-A6x25 FLX, FAN1 and FAN3 are two Noctua NF-A9 PWM fans (Node 304 intake fan replacements, quiet and subject to monitoring). Start with the medium setting for the standard Node 304 intake fans (which are not monitored in IMPI).

Available FAN mode settings in the IPMI web interface are accessible via
Configuration -> Fan Mode

Thanks for all that info, it really helps! I will look into that and see what I can come up with. Would you recommend swapping out the stock Node 304 fans? I thought about doing that based on @Stux build but after reading about the need to have a script for the fan with the different fan zones it seemed quite a bit over my head and figured I'd stick with the stock ones and see how it went.