SuperChassis 847E16: MB/CPU/HDs recommendations

[jgreco]

Very true... unfortunately, in my case, not an option. The closet is just wide enough for the servers to slide out from their rails through the doorway, and just deep enough for the rack and servers (they actually end up with their noses in the hallway if fully extended). Thus, my hope is for the back wall to act like a "chimney" of sorts, pulling the hot air up the wall and out the exhaust fan. I'm hoping 110CFM is sufficient, but I do have ceiling space available for a second unit if required, which would get me to 220CFM.

It's hard work needing lots of horsepower, in a small space, with a very high WAF (that's Wife Acceptance Factor - the most critical part of this build!)

All I want is true high speed broadband and then I could shove everything out to the data center.

 

[tvsjr]

I actually had everything in a data center... but, it wasn't cheap. If you have a freebie hookup, that's different! The home connection is 50/8 (and could be up to gigabit if I want to pay) and the servers are backed up with a big UPS (about 2 hours of runtime) and eventually with a generator. Plus, it's less servers to maintain... I've always had a fairly large lab environment at home (plus the media server, home automation, etc. etc. etc.), so having the data center is another large collection of systems to maintain. Just not enough hours in the day.

 

[jgreco]

Wish I could get anything past 50/5 here. Upstream speed is in short supply.

 

[tvsjr]

Wish I could get anything past 50/5 here. Upstream speed is in short supply.

I had to go to a business-class service to get the static IP and more upstream, plus no cap. $94.99/mo. for 50/8 and one IP, with pretty decent reliability. I can bump it up, but it does get expensive... the next step is 100/10 for about $150/mo., then 200/20 for about $200/mo. I wish they'd sell me 50/50 for a small upcharge... no one uses upstream bandwidth anyway.

 

[jgreco]

I had to go to a business-class service to get the static IP and more upstream, plus no cap. $94.99/mo. for 50/8 and one IP, with pretty decent reliability. I can bump it up, but it does get expensive... the next step is 100/10 for about $150/mo., then 200/20 for about $200/mo. I wish they'd sell me 50/50 for a small upcharge... no one uses upstream bandwidth anyway.

Well, until about 4 years ago, the business-class service was 15/2 here. Now they're finally talking about higher speeds of maybe up to 20Mbps upstream. Maybe before I die.

 

[tvsjr]

Well, until about 4 years ago, the business-class service was 15/2 here. Now they're finally talking about higher speeds of maybe up to 20Mbps upstream. Maybe before I die.

I can still remember the rush when I upgraded from modem to IDSL (DSL over ISDN... "always-on" and combined the 2B and D channels, for 144Kbps of blistering throughput). Now, a DS3 seems slow...
It is funny talking to ISPs - you start asking about upstream and they're like "why do you care about that? did you mean downstream?" No... some of us would gladly sacrifice downstream for more upstream!

Damn whippersnappers, get off my lawn. :D

 

[jgreco]

Damn whippersnappers, get off my lawn. :D

Don't go there. Both @joeschmuck and I fondly remember the days of 110 baud. I think I may have a 300 baud acoustic coupler around here somewhere. There's a pair of old USR 2400's (one is the error correcting model!) sitting on the coffee table here, and some Telebit Trailblazers on the chair. A USR Total Control MP/8 V.34 is in the closet along with several Cisco 4700M's and a pile of DigitalLink Solo Encore CSU/DSU's, just in case you want to get your T1 up and running and run a small dialup ISP (still have a dual T1 smartjack on the plywood too). The ISDN stuff was nice for about a decade, but wasn't that hot for Internet access; IDSL gear was just not that good... but it was GREAT for voice telephony. Still have a nice Adtran 550 around here somewhere that used to gateway ISDN into our PBX. DS3 was kinda the suck, but I still have a pair of Ascend GRF 400 ("Goes Real Fast") here with ATM OC3 linecards and 100Mbps linecards that'll do line rate routing at any packet size, the damn things cost $65K-$80K new IIRC.

 

[tvsjr]

You win the esoteric phone hardware competition :)

I got my start at 2400baud - using Waffle and uucp. I'm probably one of the younger people you'll find (32) who knows what a bang path is...

 

[jgreco]

Fascinating. I ran two of Milwaukee's four major UUCP hubs, solaria and spool.mu.edu.

I didn't actually get into listing the esoteric, however. There's some of that too. This was mostly stuff that got used for Internet or UUCP stuff in the Milwaukee area; the GRF's were handling at least 25% of Milwaukee's Internet traffic at one point.

 

[Tenek]

I finally ran prime95. One CPU gets much hotter, about 86C. Another stopped about 69C. I guess it gets ventilated much better.

 

[tvsjr]

Something's jacked... Tcase for that family of processors is 75C. If one CPU is hitting 86, it's likely getting throttled. I would suggest inspecting your heatsinks, thermal compound, and mounting hardware. Remember, you only need a VERY small amount of thermal compound. You may also want to explore the active heatsinks (with fans). They're available on eBay quite cheaply, and seem to do a much better job cooling the CPUs instead of relying on just the big fans and the shroud.

 

[Tenek]

tvsjr, thanks. Inspecting\Reapplying thermal paste is a good idea.
One CPU sits closer to power supply units. And air from that side comes much hotter. I assume that CPU gets hotter.
If changing thermal paste will not fix, I will buy a better heat-sinks. Can you send me some link for a good heat-sink on ebay?

 

[tvsjr]

Supermicro part SNK-P0048AP4 is the active heatsink for the LGA2011 chips. You'll need two FAN headers on your MB to drive them... they are PWM fans. However, Supermicro did build these servers to run wide open without the need of the extra fans. As long as you're feeding the systems decently cool air, I wouldn't expect to see 86C on a chip. There's definitely something going on. The usual suspect would be someone who thinks that they should use the entire syringe of thermal paste when installing a heatsink :)

 

[jgreco]

Something's jacked... Tcase for that family of processors is 75C. If one CPU is hitting 86, it's likely getting throttled. I would suggest inspecting your heatsinks, thermal compound, and mounting hardware. Remember, you only need a VERY small amount of thermal compound. You may also want to explore the active heatsinks (with fans). They're available on eBay quite cheaply, and seem to do a much better job cooling the CPUs instead of relying on just the big fans and the shroud.

That's not usually good advice. The passive heatsinks should be just dandy as long as the air baffle is properly installed and configured. With the use of the air baffle, you get redundancy in air from your fans, which you don't usually get with the single fan on an active heatsink.

Pull the existing heatsink. Once pulled, you cannot reinstall using the existing thermal material. Use a proper heatsink cleaner (not just alcohol) on both the face of the CPU and the base of the heatsink, then follow the vertical line method for application. Arctic Silver 5 is almost certainly fine for a Xeon running at spec speeds, and should easily exceed the performance of the thermal material that had previously been present. You can get the nicer stuff if you want, but there's minimal value to doing so.

If you're unclear on what constitutes too much:

This example is of someone who actually managed to INSULATE their processor. Too much thermal paste does not conduct heat, it actually insulates.

So if you follow the proper procedure, it looks like this:

prior to install (though I'm not sure what idiot would put their CPU on top of what appears to be a paper towel?!?!) Then install it, and pull the heatsink back off, to inspect the results, if you're worried. There's not much reason to be worried, but for learning purposes it isn't bad, and you've got all the materials there to clean it up again and do it a second time, so why not.

This is an example of just a little too much compound used:

And just a little too little:

You want that magic balance where the heatsink grease has spread to all four edges but not all four corners. If in doubt, it is better to have a tiny bit too much rather than too little.

 

[jgreco]

tvsjr, thanks. Inspecting\Reapplying thermal paste is a good idea.
One CPU sits closer to power supply units. And air from that side comes much hotter. I assume that CPU gets hotter.
If changing thermal paste will not fix, I will buy a better heat-sinks. Can you send me some link for a good heat-sink on ebay?

I'd definitely try redoing the heatsink first. Also inspect it to make sure it isn't dirty, sometimes dust bunnies like to get lodged in them.

 

[Tenek]

Use a proper heatsink cleaner (not just alcohol)

I have only alcohol and 7 years old Arctic Silver 5 at the moment :).
So I'm going to order new stuff now and will try to reinstall heat-sinks on Tuesday or Wednesday.

I investigated air shroud a little and it seems installed appropriately. I noticed that one part is shifted inside a little, that leaves a small gap (yellow square pic). I believe they did it because network card is installed on its way. I don't need this card right now. I can remove it and try to align both parts, that should improve air flow I guess.

Also, looking on CPUs' placement I think one CPU will be a bit hotter by design. Since first CPU partially blocks fresh air from fans and warms it up before it reaches the second CPU (blue/red arrows).
Hopefully using a better compound will bring second CPU to a reasonable temperature.

 

[jgreco]

No worries about your yellow gap. Yes, the second CPU in that sort of configuration is expected to run warmer, but not by more than maybe 10-15'C.

 

[Tenek]

Just rebooted heat-sink, didn't help. Now CPU heated to 87C (in 5 mins of Prime95). Couple degrees actually hotter, but room temperatures was about 5 degrees hotter than last time, so, I think it stayed at the same level.
I just did reinstalled one heat-sink only (the one is blocked). That is how it looked when I opened it:

So, I cleaned it up:

The best I could do with thermal paste application:

and finally Prime95 output:

I cooled the room and retested. It stopped at 85C exactly same level as it was before. Max difference in the temperature between CPUs is 19C. Without load difference is about 4C.
So I can try to reinstall the other heat-sink. But, I'm pretty sure that one I touched is hotter one.

Question: do I need to worry about it? Honestly, I can't think of situation when both CPUs will be stressed up to 100% for a long period of time.
Primarily that will be a backup solution + media server (music/video streaming). How ZFS will be CPU hungry if I will feel all 36 bays?
I will be running about 3 guest OSes in VirtualBox templates, nothing intensively used (just for internal usage).
It might be stresses with HD movies streaming/encoding but for a few hours max.

Is it critical enough to get active heat-sink or replace CPU? Although I think CPU is fine.
Does it make sense to assemble the server, and monitor CPU to see where the temperature will settle-down in the "real live" environment?
Anyway, I'm open to suggestions. But, if it is not critical I would leave it. Please let me know.

 

[depasseg]

I'm no expert but that looks like A LOT of thermal paste.

 

[Bidule0hm]

Ok, I think I know where's the problem: the IHS is maybe not level and/or there's maybe not enough/too many thermal paste between the IHS and the die.

There's a solution but it's not for the faint hearted... basically you remove the IHS, clean everything, put new thermal paste and re-glue the IHS. The fun part is that before removing the IHS you don't even know for sure if the problem comes from that or something else... :D

@depasseg yep, too much paste on the first picture. But the quantity he put after cleaning looks ok.