E3-1241

[NineFingers]

I've been doing the burnin testing of my new box, and I've hit a point of concern regarding CPU temperature. The motherboard monitor shows that the 1241 is hitting up to 95C during the stress test. I've been looking round the web to find operating temps for this CPU, but can't find anything. Should I be concerned by the temps I'm seeing?

Config:
Define R4
Supermicro X10SL7-F
Seasonic 55oW G-series
Crucial 16G kit
E3-1241 V3 (stock cooler)
6x2T ED Red (2 of which have been RMAd in the first week!)

 

[philiplu]

I'd say yes, you should be concerned. My rig is very similar to yours (same board, CPU, PSU, cooler, XL R2 instead of R4). When I put it together, I didn't get the stock cooler attached properly, and my temps hit 100C and the CPU went into thermal shutdown. I reseated the cooler and made sure the locking pins were properly engaged (didn't bother removing and reapplying thermal material), and temps were much better. During burn-in, I saw Prime95 hitting 75C or so, and the CPU idles at 41C. Now that the NAS is in service, I see max CPU temps < 60C when I'm getting CPU usage around 75%.

When I was still seeing burn-in temps up to 75C, I considered replacing the cooler and picked up a Cooler Master Hyper 212 Evo (I've used that in multiple other builds). But I decided not to bother changing out the stock cooler, since the max temps under 60C in normal use seemed good enough.

So my guess is your cooler isn't latched down tightly enough. You shouldn't be seeing temps that high.

 

[NineFingers]

Thanks for the feedback

Since our installs are similar, did you have issues with power supply lengths? Could you post a pic of the inside of your machine? I'd like to see the wiring layout you did.

 

[philiplu]

I had to use an extender on the motherboard power cables (at least the 24-pin, can't remember if the 8-pin reached OK). Just picked up something at the local Fry's. Also ended up with some Molex extenders to reach the 5-in-3 hot-swap bay. But everything tucks in pretty well behind the motherboard.

I'll try to post some pics here, but it'll be a day or two. I'm running the initial backups of a few TBs from some Win clients, and that'll have to finish before I can power down and pop open the box.

 

[Ericloewe]

Yeah, that system should absolutely not be in production yet. 95ºC is absolutely unacceptable.
However, it may be a bug in the IPMI.

Please post the output of

Code:

sysctl -a |grep temper

 

[NineFingers]

As philiplu suggested, I re-seated the heatsink, and it does appear that it made some progress.

While running Prime95 as philiplu mentions, I normally have CPU temps reading mid-70s, but spikes up to 88 occur.

While at idle in FreeNAS, I get the following output from sysctl -a:
hw.acpi.thermal.tz0.temperature: 27.8C
hw.acpi.thermal.tz1.temperature: 29.8C
dev.cpu.0.temperature: 43.0C
dev.cpu.1.temperature: 43.0C
dev.cpu.2.temperature: 47.0C
dev.cpu.3.temperature: 47.0C
dev.cpu.4.temperature: 42.0C
dev.cpu.5.temperature: 42.0C
dev.cpu.6.temperature: 40.0C
dev.cpu.7.temperature: 40.0C

IPMIView reports CPU temp of 40.0C... so I'm guessing the "bug" doesn't exist?

Should I continue my quest for lower temps? Any more ideas?

Thanks for the input.

 

[philiplu]

That still seems a bit high. Either I misremembered my CPU idle temps, or the thermal material has had time to settle in (IIRC, temps will get better over a period of weeks as the material spreads a bit under heat). Here are my current temps on an idle system:

Code:

[phil@Marvin ~]$ sysctl -a | grep temper ; sudo ipmitool sdr type temp
hw.acpi.thermal.tz0.temperature: 27.8C
hw.acpi.thermal.tz1.temperature: 29.8C
dev.cpu.0.temperature: 32.0C
dev.cpu.1.temperature: 32.0C
dev.cpu.2.temperature: 31.0C
dev.cpu.3.temperature: 31.0C
dev.cpu.4.temperature: 34.0C
dev.cpu.5.temperature: 34.0C
dev.cpu.6.temperature: 37.0C
dev.cpu.7.temperature: 37.0C
CPU Temp         | 01h | ok  |  3.1 | 32 degrees C
System Temp      | 0Bh | ok  |  7.1 | 35 degrees C
Peripheral Temp  | 0Ch | ok  |  7.2 | 37 degrees C
PCH Temp         | 0Ah | ok  |  7.3 | 39 degrees C
VRM Temp         | 48h | ok  |  8.1 | 33 degrees C
DIMMA1 Temp      | B0h | ok  | 32.64 | 26 degrees C
DIMMA2 Temp      | B1h | ok  | 32.65 | 27 degrees C
DIMMB1 Temp      | B4h | ok  | 32.68 | 27 degrees C
DIMMB2 Temp      | B5h | ok  | 32.69 | 26 degrees C

So I'm seeing temps about 10C lower than yours. What is your fan setting? Mine is at Full Speed, with the CPU fan (on FAN1) running at 2000 rpm. I'd suggest pushing your fan to full if it's not already, then watching temps for a few days/weeks to see if temps drop a bit. If they stay above 40C idle, you probably need to either try reseating the cooler again, or swapping it out for a better cooler.

 

[NineFingers]

Now I have to figure out what ipmitool is and how to install it...

 

[philiplu]

Looks like ipmitool is installed as part of FreeNAS 9.2.1.7, in /usr/local/bin. I can see it in the installation .img.xz, with the same timestamp as on my system, so I don't think I did anything special to get it installed. The man page is here. It's a nice tool for directly querying IPMI from within FreeNAS, without having to run IPMIView elsewhere.

 

[NineFingers]

I've taken the heatsink off, and replaced the thermal compound. It seems to have been a good thing. Temps under load read up to 71C, which is a huge improvement. Idle temps are:
hw.acpi.thermal.tz0.temperature: 27.8C
hw.acpi.thermal.tz1.temperature: 29.8C
dev.cpu.0.temperature: 41.0C
dev.cpu.1.temperature: 41.0C
dev.cpu.2.temperature: 37.0C
dev.cpu.3.temperature: 37.0C
dev.cpu.4.temperature: 35.0C
dev.cpu.5.temperature: 35.0C
dev.cpu.6.temperature: 36.0C
dev.cpu.7.temperature: 36.0C

Trying to run 'ipmitool sdr type temp' gets:
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Device ID command failed
Unable to open SDR for reading

so I need to figure that out.

I've not yet tested changing the fan speeds.

Bottom line... temps are mych improved, but still not quite where I'd like them to be.

 

[philiplu]

To enable ipmitool (by loading the ipmi kernel module), you need to follow the instructions in section 5.3 of the Users Guide. Once you've got ipmitool running, you can use it to check the fan speeds. Just run "ipmitool sdr" to retrieve all the sensor readings, not just temperatures.

 

[philiplu]

Here are the pics of the inside of my Define XL R2 that you asked for. I did have to extend both the 24-pin ATX and the 8 pin 12-V cables to reach the top of the motherboard. The case looks almost empty, but I wanted to leave expansion room. If I ever fill up my 6-drive RaidZ2 of 4TB WD Reds, I've got a second 4-drive HD cage that'll be put in the case between the PSU and the bottom existing 4-drive cage. That, plus moving the 2 SSDs somewhere else (probably just sticky-tape them to the MB tray below the MB) will give me room for a 2nd 6-drive RaidZ2 vdev. One other build note - I ended up swapping out the two Fractal Design 140mm fans that are in the case front, blowing air through the HDs. Their static pressure wasn't enough to do the job. I swapped them out for this and this. HD temps now run 33-37C (38 during a scrub), even with the case sitting in a small utility closet that gets toasty warm.

Eh, I don't have an account in imgur, pastebin, whatever and don't want to bother right now. So you can find the pics here and here.

 

[NineFingers]

Thanks for the pics. Very nice looking installation! Mine looks nothing like that... and not in a good way. You've given me motivation to clean mine up.

 

[NineFingers]

Philiplu - next question -- did you flash your mobo to convert the other ports from SAS -> SATA? Any issues/difficulties with that?

 

[philiplu]

Flashing is to change the on-board SAS controller from IR (Raid) to IT (JBOD) mode., not so much SAS to SATA. I did flash it, though it looks like some people didn't bother and are OK. Might as well flash it, though, since you'll probably have to reflash once 9.3 comes along. The FreeBSD driver version needs to match the controller firmware version. That means v16 in the 9.2.1 series, and I think something later in 9.3.

I found the V16 IT-mode firmware here. Download PH16.0.1-IT.zip. I followed jyavenard's instructions in this post, flashing from UEFI, not DOS. This was a couple months ago, so I don't recall the details, though I remember it took some fumbling on my part before I could get my USB stick with the firmware properly mounted under the UEFI shell. Actually, I can't remember if I ended up using a USB stick or just a virtual device under IPMIView. Once I was able to access the .nsh shell script, though, the actual flashing was easy.

 

[NineFingers]

I recall seeing folks had issues performing the flash with certain CPUs... did you perform the flash with the 1241 installed?

 

[philiplu]

Yep - the people complaining about needing an older CPU had an old version of the BIOS. My board came with the latest BIOS (V2.00) and IPMI Firmware (V1.42), so the SAS controller was the only thing that needed flashing.

 

[NineFingers]

excellent! Thanks so much for your assistance.