Cooling for a Dell PERC H310

Status
Not open for further replies.

Evertb1

Guru
Joined
May 31, 2016
Messages
700
I am about to buy a Dell PERC H310 SAS controller. It is offered to me for a nice price, the seller has taken care that it is flashed with the LSI logic 9211-8i P20-IT firmware and 2 SAS to SATA cables are thrown in. So nothing to complain about. Only this card comes with a warning that LSI Logic specifies forced cooling for this card of at a minimum 200 LFPM (?). This because the processor is supposed to generate quite some heat.

So my question is if anybody out there is using this card and what is done about the cooling? At the first sight the heatsink of the thing is not suited to mount a fan. I would be grateful for some tips.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
You'll really need to provide more information here. What kind of case are you using this in? In typical server environments, I could see this card overheating without sufficient airflow. However, in a more open desktop/tower case, I wouldn't worry about cooling.
 

bigphil

Patron
Joined
Jan 30, 2014
Messages
486
The H310 does run a bit hot. Adequate cooling is advised for that card, but I'd also recommend removing the heatsink from the card and applying some new high quality thermal paste to it. @Gerk has a nice thread here about this helping out when he experienced overheating issues with his HBA.
 

Evertb1

Guru
Joined
May 31, 2016
Messages
700
You'll really need to provide more information here. What kind of case are you using this in? In typical server environments, I could see this card overheating without sufficient airflow. However, in a more open desktop/tower case, I wouldn't worry about cooling.
It will be running in a midi size tower case (Fractal Design Define R5). The case has a good airflow so I hope for the best.
 

Evertb1

Guru
Joined
May 31, 2016
Messages
700
The H310 does run a bit hot. Adequate cooling is advised for that card, but I'd also recommend removing the heatsink from the card and applying some new high quality thermal paste to it. @Gerk has a nice thread here about this helping out when he experienced overheating issues with his HBA.
Applying some new thermal paste sounds like a good idea.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
They also make fan "cards" that fit in the expansion slot spaces to provide additional cooling.
 

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You'll really need to provide more information here. What kind of case are you using this in? In typical server environments, I could see this card overheating without sufficient airflow. However, in a more open desktop/tower case, I wouldn't worry about cooling.

Huh? I think you got that backwards. In a typical server environment you should always have sufficient airflow and you should never see the card overheat. It is the typical open desktop/tower case that is more likely to fail to make sure that there is sufficient airflow past the card. Servers typically have a fan bulkhead that forces air fairly evenly through the chassis. It's possible to muck that up, of course. However, it is the standard PC cases where I'm always having to make some sort of accommodation for cooling for RAID cards.
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
The case has a good airflow so I hope for the best.
Take it from a heat transfer guy of many years, you need to adequately distinguish between "bulk air flow" through the case that can be "good" and the localized air flow at the heatsink that needs to be >200 feet per minute velocity apparently. @jgreco's message immediately above is right on point - the open space in a typical PC case is not conducive to "channeling" air to points of high demand. Effective convective cooling needs (a) sufficient mass of air with the capability to accept the heat, (b) delivered at a high enough velocity to exceed that required to achieve the required rate of heat transfer from the hot device - the finned heat sink in most of our situations. Blow your hot breath onto the back of your hand to simulate the effect of a convective cooling air stream based on velocity in order to visualize why the channeling of the air flow adjacent to the hot source is critical.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Take it from a heat transfer guy of many years, you need to adequately distinguish between "bulk air flow" through the case that can be "good" and the localized air flow at the heatsink that needs to be >200 feet per minute velocity apparently. @jgreco's message immediately above is right on point - the open space in a typical PC case is not conducive to "channeling" air to points of high demand. Effective convective cooling needs (a) sufficient mass of air with the capability to accept the heat, (b) delivered at a high enough velocity to exceed that required to achieve the required rate of heat transfer from the hot device - the finned heat sink in most of our situations. Blow your hot breath onto the back of your hand to simulate the effect of a convective cooling air stream based on velocity in order to visualize why the channeling of the air flow adjacent to the hot source is critical.

Oh there you go, listen to @Redcoat ...

Inside servers you do need to manage airflow. Consider this chassis here. It's a 1U UIO chassis from Supermicro, a clever but horrible idea that involved reversing the component side of one of the PCIe cards, so that the back of the bottom card is facing the bottom of the chassis, kinda like a "mirror-mirror PCIe". Of course, years later, UIO cards are nearly unobtainium and unsupported anyways, so I've been ripping out the UIO risers and putting in a WIO riser instead, that gives two standard PCIe, with the bottom card's PCB roughly in the middle of the chassis. For wiring reasons, I like to put RAID cards in the bottom, leaving the top accessible for ethernet/10G/etc, but a combination of issues means that you have to carefully engineer airflow. First, the chassis comes with only four of six possible fans, and the two missing fans are over in the area of the expansion slots. So we install two more fans. Then, we install a "low-flow" slot blank in the top slot, and a "high-flow" in the bottom, and don't worry about plating the card itself, but rather just make sure the card is cabled and seated so that there is airflow on all sides, because the RAID controller and cabling are such that the card is not removable without pulling it along with the riser. The high-flow blank creates some nice airflow over the RAID controller heatsink, sourced from more than a single fan, and the fans are all monitored by IPMI, so a cooling fan failing does not turn into an immediate overheat and crisis.

Along those lines, I cringe every time I see @wblock proudly describing how he tacked a crappy 40MM fan onto the top of a HBA heatsink, because when that fan invariably fails in a few years, it will actually act as an airblock/insulator, and actually cause his HBA to cook much worse. The LSI's are notorious for kinda-working but horribly corrupting data randomly when they're cooking, so in my view this is a Really Bad Fix, because once it starts puking corrupted blocks into your pool, that could be a pool-killing event. Yes it works great right UP to that point, no doubt. But so does riding in your car without a seatbelt, right up to the crash. A much better fix would have been to replace the heatsink, sacrificing the next slot (which the fan does anyways), and then make sure of "good airflow."

Anyways, the only "real" fix I've found for normal PC cases is to make sure that there's airflow over everything important, and that things are actually cool, and that failure of a single fan doesn't compromise that. This is WICKED HARD, and @Redcoat is right in that this isn't JUST about "oh I have twenty 120MM fans therefore I must be OK" but it is also about localized airflow management, and also making sure that a single fan's failure doesn't result in the airflow stopping. Over EACH COMPONENT that needs to be cooled. WICKED HARD. So a handheld anemometer, temperature probes, and a handheld IR thermometer are good tools to have. I can't justify the cost of a FLIR camera, although they've admittedly come down in price quite a bit... the shop here just doesn't build enough stuff in normal PC cases to warrant that.

I have found some "neat" PC cases, such as the Corsair Vengeance C70 "ammo-can" which include a fan on each side of the 3.5" HDD cages ("redundant!"), which I strongly endorse for keeping your drives cool, but these do not really solve the issue of PCIe slot cooling for RAID etc.
 

Evertb1

Guru
Joined
May 31, 2016
Messages
700
To all: Thanks for your input. @jgreco: whow that is some response. But a lot of usefull information is offered. While this card will not be installed in my production server it's clear to me now that I need to take care for a good dedicated cooling solution.

The case of the lab PC where this card will be used, is fitted with sufficient intake and outake fans to keep the system pretty cool with a fair workload (rsync operation as we speak) and without this card for now. The highest temp I see is from the PCH and that is 39 degrees celcius with the current ambiant temp being 20 degrees celcius. And the fans are not exactly running full speed.

So my take on it is, that if I have an extra fan pulling fresh air and blowing over the PCI slots (from the side), I will be good to go. I just take care that the H310's heatsink is postioned in the middle of that flow. Of course I need to monitor the results (and see to it that the other temps stay OK as well) but I guess it's word trying.
 
Last edited:

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
Along those lines, I cringe every time I see @wblock proudly describing how he tacked a crappy 40MM fan onto the top of a HBA heatsink, because when that fan invariably fails in a few years, it will actually act as an airblock/insulator, and actually cause his HBA to cook much worse.
Thanks for your eventual constructive feedback. I disagree with your characterization of the fan and my description of it, though. Many people are unaware of the problem, and this is a way to address it actively at the heatsink, rather than passively with case airflow. My H310 measured 38C at the heatsink under load with no fan at all, so adding it was mostly a cheap form of insurance. The H200s I have tried ran much hotter and I would not use them without a fan.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Thanks for your eventual constructive feedback. I disagree with your characterization of the fan and my description of it, though. Many people are unaware of the problem, and this is a way to address it actively at the heatsink, rather than passively with case airflow. My H310 measured 38C at the heatsink under load with no fan at all, so adding it was mostly a cheap form of insurance. The H200s I have tried ran much hotter and I would not use them without a fan.

"eventual constructive feedback?" Well, byte my bits, I have sufficient years in the hardware biz that I am entitled to an actual professional opinion, which I have learned through the school of hard knocks, not to mention formal education in electrical engineering and physics. You can disagree all you want, but 40MM fans, especially ones tied to items with high heat loads, are subject to failure. Once they fail, systems that are dependent on them can turn into little ovens. Your "insurance" cools your card now, but when it fails, what's going to happen is that the system's normal airflow, which would normally cause a modest amount of air to flow over the heat sink and through the fins on an un-fanned HBA, will instead cause a large area where that airflow is prevented from reaching the heatsink (because there's a dead fan covering that area) and there isn't a lot to encourage the system's normal airflow to flow *through* the heatsink, so basically the heatsink stops working. So when the fan inevitably fails, you've mostly killed the airflow over the surface area. Heatsinks work by creating large surface area to interface with moving air. Blocking the surface area and blocking the air causes the heatsink to be unable to function correctly. Disagree with my characterization all you want, but if you choose to do so, please try this experiment first: De-energize the fan for an hour while the system is running, and then measure the temperature. Then try it again with the fan removed, and measure the temperature.

The average ZFS pool is designed for long-term storage, and even if we just charitably limit it to an average lifetime of the first set of disk drives, let's say five years, I see about a 20% failure rate on high quality 40MM GM1204PQV1-8A's over that period. These are top of the line 28MM deep fans rated for 100% duty cycle. By comparison, you've *probably* tacked on a crappy 8 or 10MM thin-profile fan of the sort that might allow you to screw it to the heatsink, which are more prone to developing vibration/buzzing issues due to the short hub, and once it starts doing that, they rapidly deteriorate. These tend to last only a year or two under 100% duty cycle, with a really high failure rate. On top of it, what I've seen happen with the LSI controllers under high heat stress is that they tend to continue "working" but do random bad things. We saw a server come in the shop that appeared to have intermittent SATA connectivity, figured "bad drive", replaced the drive, still bad, so started to look at cabling and backplane, all seemed fine, and then while putting all that back together, noticed that IPMI was reporting a fan failure, sure 'nuff, only one fan in the expansion bay, so the "fix" was to replace the fan and install the other one that should have been there too. We've seen broken HBA's send bad stuff to storage, and while I would love to think ZFS is resilient enough to cope with that, I don't actually trust that to be the case. Placing a 10MM high 40MM fan on your HBA seems like it is destined to suffer a burnout at some random point in the future and I do not encourage users to set up their systems as crash test dummies.

So what I really don't want to see is for your "clever insurance" to fail and to start spewing bad bits into your pool. I heartily approve of improving the cooling inside your FreeNAS system, a topic any of the regulars will recognize as a near-and-dear topic of mine, but I recognize what you're proudly describing to other users as a potentially hazardous practice, and your response tags you as someone who is not a professional server builder, so yes, I do cringe each of the times I've read about your fan hack, because I understand the risk and see it as dangerous, and have now even fully explained and justified that opinion.

I'm going to go so far as to suggest that you would be better served finding an appropriate heatsink to upgrade the heatsink on that H310, and I'd really also like to see you avoid suggesting to users that this is a good idea.
 

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
Along those lines, I cringe every time I see @wblock proudly describing how he tacked a crappy 40MM fan onto the top of a HBA heatsink,
This is what I meant by "eventual": I appreciate you finally mentioning it rather than just cringing. I'm not sure why you say "proudly". It's a warning about a potential problem with the LSI controllers that people should know about.

If I could make a suggestion, your post would be improved with links to replacement heatsinks you find appropriate, and what the temperature measurements were with the original and new heatsinks on the H310 under load.
 

Evertb1

Guru
Joined
May 31, 2016
Messages
700
This is what I meant by "eventual": I appreciate you finally mentioning it rather than just cringing. I'm not sure why you say "proudly". It's a warning about a potential problem with the LSI controllers that people should know about.

If I could make a suggestion, your post would be improved with links to replacement heatsinks you find appropriate, and what the temperature measurements were with the original and new heatsinks on the H310 under load.
I appreciate any and all responses on this and other questions I posted on this forum. Just the fact that people take some time to give me an answer to their best knowlegde should make me feel gratefull. Some answers are from the perspective of a good willing hobbyist some from the perspective of a die hard professional and everything in between. I love them all and hope I am clever enough to pick up on those things that are usefull for me.

That being said I would never dream of including a 40 mm fan to cool something I value. I remember the time that I bought some external USB housings for a couple of internal CD burners. You never touched something hot until you touched a running burner with a failed fan in the housing. Those fans were 40 mm as well and the fail rate was awfull.

Finaly I agree that some links to replacement heatsinks would be usefull.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
I'm curious what kind of insane conditions "high heat load" means. Broadcom's spec sheet shows a maximum operating environment temperature of 55*C (during which, at best with perfect heat transfer, the card's chip is 55*C), and in this application the RAID capabilities of the cards aren't even being used, so I'm not sure how we get to a spot where these cards are overheating.

This really is an honest question, because the proffered wisdom conflicts with my existing understanding of processor and chip temperature limits.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
This is what I meant by "eventual": I appreciate you finally mentioning it rather than just cringing. I'm not sure why you say "proudly". It's a warning about a potential problem with the LSI controllers that people should know about.

Please try re-reading what you wrote. This is not the first time you've described this with a tone that seems to suggest that this is some sort of "good solution."

My H310 runs noticeably cooler than an H200. I still added a 40mm 12V fan to it. It was a little too large, so it's turned slightly andonly two screws are holding it, but solidly. See https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/ for an example.

I see nothing there that comes across to the average newbie as anything but a suggestion that adding a fan is a good idea.

If I could make a suggestion, your post would be improved with links to replacement heatsinks you find appropriate, and what the temperature measurements were with the original and new heatsinks on the H310 under load.

You can suggest whatever you'd like, but I'm not likely to do your homework for you for free, and I've expanded upon that in a private message, because I expect you don't really understand how I find this offensive. I'm just going to leave it at that, except to note that it's all good as long as you don't try to ask me to do any more homework for free.
 

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
This is not the first time you've described this with a tone that seems to suggest that this is some sort of "good solution."
Certainly. I just thought it was odd that if you had a complaint, you would have let me know earlier.

My suggestion was just that, a suggestion. Unfortunately, it would likely be difficult to compare any numbers you might have with the ones I posted anyway, due to differing system setups.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Certainly. I just thought it was odd that if you had a complaint, you would have let me know earlier.

Despite what it might appear from my post count, my time is not infinite and I do not go out of my way to touch on every vaguely interesting topic that passes by. If you review this thread, you will note that I only brought this up in the context of airflow management in an environment where some of the design assumptions for a card ("constant server chassis airflow") are not valid. We were discussing localized airflow problems, something I've seen in so many ATX towers, especially with LSI RAID chipsets, some 10G network cards, etc. You may not have understood that I was using your up-thread endorsement of your fix as a poster-child-quality bad fix, but there you have it.

My suggestion was just that, a suggestion.

You're also an "Administrator"-badged "ixSystems"-badged employee. As some of "you" have said in the past to some of us "mere moderators," you should probably be aware that this may carry with it some extra authority, and I am not about to sit here and have it implied that my post wasn't "good enough."

Unfortunately, it would likely be difficult to compare any numbers you might have with the ones I posted anyway, due to differing system setups.

I do not gather these numbers because I don't think we've ever built ATX chassis systems with H200's or H310's. Oddly enough I just did a data center deployment of an R510 with both an H200 and H310 in it last week...

We've built ATX midtowers that included other LSI RAID controllers, where the last comment for a system we built last year had it measured at 103'F. LSI's products can usually be dropped around 5'C simply by removing the heatsink, removing the thermal pad or other material, and then applying new material such as Arctic Silver 5 after a full cleaning. It's worth noting that you can create monster temperature swings on some of the LSI products, easily 20'C, just by modifying the airflow. My basic conclusion is that the heatsink is too small, but for most server applications it is impractical to modify. In a larger chassis, where you could afford to lose the next slot over, bumping up the heat sink size makes more sense.
 
Status
Not open for further replies.
Top