Norco RPC-4224 and hard drives that are too hot

[cyberjock]

Summer has arrived. Naturally my server is a bit too hot in my living room. I don't have central A/C and its common to see living room temperatures over 90F. As such, my 24-bay RPC-4224 is too hot. I did buy the "upgraded" backplane that has 3x120mm versus 4x80mm and i currently have 3 Gelid 120mm fans in the backplane. They seem to have good air flow unless there is backpressure or obstructions. I bought these fans because they are basically bearing-less. As such, I thought they'd provide enough cooling. In past years the server was in the basement, but had a different case and different fans. As an idiot check I did verify that the little vents for the hard drives are all open fully so there is maximum airflow. I do know that the other components (CPU, video card, etc.) are a bit warmer than I'm used to, so I tend to think that this isn't an issue limited to just the hard drives. I do have every available fan slot filled with a fan along with a PCI exhaust fan.

Anyway, I moved my server to the basement 2 weeks ago after having 4 drives fail(out of 24) within about 14 days. Thank goodness for redundancy. They are all Green drives so I can't really buy "lower powered" hard drives. Previously those 24 drives have had only 1 failure in over 3 years of uptime. I try to keep the hard drives below 40C, but I'm getting emails that 1-3 hard drives are hitting 45C. So I consider this a situation that requires rectification.

My first thought was Corsair SP120 fans. But I figured I'd see if anyone else that actually pays attention to the reviews has any good recommendations before I click "buy". Those fans have a rating of 62CFM each but seem to be rated for high static pressure. Since their CFM rating is less than the fans I have(but my fans likely don't have great flow with high D/P) I'm not really sure how "much" of an upgrade this is, if it is at all. I'd really like to see temps below 40C. Especially since the basement is typically below 70F and I've never seen it above 75F.

A second thought was something like an industrial fan. They are usually very loud, but push a very high amount of air. Something like http://www.amazon.com/dp/B009BHRZTM/?tag=ozlp-20(252CFM, also notworthy is that each fan is rated for 48W at 12V.. wow!) but then I get a little worried about excessive vibrations. I do have those rubber things you can use to mount the fan to absorb vibrations.

Any recommendations for fans? Remember that this is a high density server with significant airflow obstruction. Each hard drive seems to have only a small amount of airflow into it as the hard drives almost fill the bay completely on their own. I was thinking I was looking for a fan with a high D/P rating to help ensure there is adequate flow. I've never really paid attention to fan reviews because I've been buying these Gelid fans for all of my computers for more than 3 years and never had a single failure and they are very quiet being they are bearingless. Cost is only an issue within reason. I don't mind paying $30 a fan if I'm sure it'll solve my problems. Noise is only a concern because when I move I'd like to think my new house will have my server in a living space, but it does sit in a 20U rack which muffles some noise.

 

[jgreco]

Do the "quick and stupid" airflow check: take a sheet of 8.5x11 paper, preferably something a little heavier paperweight, and see if it sticks to the front of the server, and how well.

 

[cyberjock]

A regular sheet of paper won't "stick" to the front of the server. If I support the weight of the paper with my hand under the sheet it will suck onto the case, but if I move my hand it falls to the floor. So pretty shitty airflow. :P

 

[jgreco]

Yeah, probably a fail. You don't necessarily need to ramp up the CFM, it may merely result in a more-rapid burnout, which is basically how your fan options all work out, because there'll never be enough air in.

You may, however, want to look at thicker fans. The general idea is that rotating a fan faster generates higher CFM, so lots of fans do that. But a thicker fan "pushes" harder, so you wind up with a lower pressure in front of the fan bulkhead, which hopefully encourages more air to be sucked in the front. Or it adds a lot of burned watts to an already hot chassis plenum...

The depressing thing is that you have to throw power at the problem, and power is what's causing the heat in the first place. Ironically, you might be better off with 80mm fans. The amount of power you need to toss at a 120mm fan in order to forcibly move air usually seems to end up being impractical, but you can look at it yourself.

So you're looking at this Corsair product? I'd say forget it. 1.1mm/H2O static pressure.

Get yourself some of these. The V80E12HBA5 is a 73CFM fan which can handle up to 33.7mm/H2O static pressure; at 20mm/H2O it is pushing only about 20CFM. ;-) I'm cheating because I just happened to pull this out of the 4U 24 drive Supermicro chassis here - and the -57 variant is an inexpensive PWM fan! But it is an impressive fan, and with the 4-wire PWM capabilities, it can be made to run only as fast as it needs to, if your motherboard supports that.

We've got some 1U (40mm) fans, SanAce40 9CR0412S513, that are actually two fans stacked. I can't find the darn spec sheet for them anymore but they're amazing at moving air in a 1U chassis, except they're also loud like a jet plane and they suck power like there's no tomorrow.

 

[cyberjock]

So you're looking at this Corsair product? I'd say forget it. 1.1mm/H2O static pressure.

I'm not looking at that fan. That's the AF120. I'm looking at the SP120 High Performance Edition which is 3.1mm water. Yours do have a much higher rating for pressure. The SP120 are designed for pushing air through a water filled radiator for water cooled CPUs. The reviews people give say they work well for that, and some say it helped case temperatures. But I can't find anyone that put them in a 4U 24-bay case and was happy(or not happy). Hmm...

Get yourself some of these. The V80E12HBA5 is a 73CFM fan which can handle up to 33.7mm/H2O static pressure; at 20mm/H2O it is pushing only about 20CFM. ;-) I'm cheating because I just happened to pull this out of the 4U 24 drive Supermicro chassis here - and the -57 variant is an inexpensive PWM fan! But it is an impressive fan, and with the 4-wire PWM capabilities, it can be made to run only as fast as it needs to, if your motherboard supports that.

While the flowrate and pressure graphs are pretty impressive, that is also pretty darn loud. 60dba! WOW! I'm not sure that's something I'd actually want in my living room. :P Also, looking at the assembly link you provided from ebay I'm not sure that actually has a fan wire that I can plug in. It looks like it plugs into a chassis assembly. Those fans are up to 16watts...each!

Just as a comparison:

	Corsair	Nidec
Watts	2.16	16.8
CFM	62.74	100
Pressure(mmH2O)	3.1mm	44mm
RPM	2350	9500
dBA	35	61

I'm not gonna lie.. I feel like what you are proposing is overkill at first. But, then I have to stop and think about it. Supermicro makes good stuff. They are very reliable and trustworthy, so maybe its not so much overkill as I think. But now that the 2 fans are being compared side-by-side I really wonder if those Corsair fains are going to be good enough. I feel like what was my "high standard" was just lowered to "ghetto" status.

My concern with the PWM option is that:
1. If my motherboard controls temperature it might not control well enough to keep the hard drives cool.
2. My motherboard may not be able to handle that kind of wattage through the PWM. It's a Gigabyte motherboard but is high end and supports ECC and Xeon(which is how I'm using it now).

Naturally I'll have to buy 3 of them. I'm thinking it would be a very bad idea to put 48watts of fan up to 7v via the 12v to 5v bus crossover. I'd almost certainly burn out the 5v bus, which isn't good for anyone.

I'm wondering if there is some kind of box that I can hook all 3 fans up to and dial in a temperature and put it on the back side of the hard drives to regulate the exhaust air temperature from the hard drives and can handle that kind of wattage.

There was a funny video of a guy that accidentally stuck his finger in some high speed 120mm fan on Youtube. It dug a pretty deep gouge into his finger and he bled all over. Wowzers! Pro-tip: Do not stick your finger in a running high speed fan!

Damn, I thought this was gonna be easy... /sigh

Edit: Boo. I just noticed those are 80mm fans. I do have the old 80mm plate so I could go back to that.

 

[cyberjock]

One thing I hate is that a lot of these companies call their pressure mm/H2O. It's not mm per H2O. It's mm of H2O. The proper way to list the pressure would be mmH2O(as I showed above).

 

[jgreco]

While the flowrate and pressure graphs are pretty impressive, that is also pretty darn loud. 60dba! WOW! I'm not sure that's something I'd actually want in my living room. :P

Yes, when ramped up, they are <screaming over the noise:> L-O-U-D. From my own point of view, I'm generally fine with noise if cooling is needed, but it is nice to have quiet too.

Also, looking at the assembly link you provided from ebay I'm not sure that actually has a fan wire that I can plug in. It looks like it plugs into a chassis assembly. Those fans are up to 16watts...each!

Gotta burn watts to cool watts, at least in this nonoptimal situation. Pretty sure you already understand the basics of all this, but for the benefit of others reading this thread:

Rackmount servers are designed to bring air in the front and expel out out the back, to work with data center cooling strategies. These 24-drive in 4U chassis basically stick four 3.5" drives side-by-side. A 19" rackmount chassis is actually about 17" wide; in this case 14" are fully occupied by drives, leaving just 3" of space for chassis metal, drive trays, etc. Vertically, 4U is 7", and with six 1" drives stacked, that means only 1" to allow for chassis metal, drive trays, etc. in that direction. Unfortunately, drive trays generally require some tracks on each side of a drive, so the theoretical up-to-3" is mostly eaten up. A 24-drive chassis tends to have very poor airflow characteristics as the best choice for cooling involves being built to have a millimeter of airspace above and below each drive, and there just isn't any other place to flow air in. But moving air around corners and in tight spaces usually involves pressure differential, because air doesn't naturally want to do that. Your average computer case fan doesn't generate much pressure differential, because it isn't expecting to be meeting significant resistance... most computer cases just need to move air, not force it through little 1mm gaps. And needing force, that becomes watts, electrically.

I'm not gonna lie.. I feel like what you are proposing is overkill at first.

Well, you see, you want it to be, and you want to use PWM if you can, because then you get a beautiful self-regulating system. Overkill means drives cooled (at least if it is possible). But you can mitigate and make it quieter and more energy-efficient with the PWM. So then you get all three, just not at once...

But, then I have to stop and think about it. Supermicro makes good stuff. They are very reliable and trustworthy, so maybe its not so much overkill as I think.

You can find plenty of what I'd call "thick fans" all throughout the world of rack-mount gear. Supermicro has no particular monopoly on the good design aspect. But it does help to look specifically at the other 4U-24's because it is particularly problematic.

But now that the 2 fans are being compared side-by-side I really wonder if those Corsair fains are going to be good enough. I feel like what was my "high standard" was just lowered to "ghetto" status.

That's why it's fun to toss ideas around.

My concern with the PWM option is that:
1. If my motherboard controls temperature it might not control well enough to keep the hard drives cool.

Certainly possible. Most PWM fans seem to be able to operate as variable voltage fans, so check that, and fallback to a fan controller is an option.

2. My motherboard may not be able to handle that kind of wattage through the PWM. It's a Gigabyte motherboard but is high end and supports ECC and Xeon(which is how I'm using it now).

So an opportunity for you to either check with Gigabyte support, or to just do it, burn out the board, and need a nice Supermicro one ;-)

Naturally I'll have to buy 3

four

of them. I'm thinking it would be a very bad idea to put 48watts of fan up to 7v via the 12v to 5v bus crossover. I'd almost certainly burn out the 5v bus, which isn't good for anyone.

I'm wondering if there is some kind of box that I can hook all 3 fans up to and dial in a temperature and put it on the back side of the hard drives to regulate the exhaust air temperature from the hard drives and can handle that kind of wattage.

You don't need it to handle any "kind of wattage". With PWM, the ground and 12V are wired to the power supply, the controller is only pulsing the PWM. You can theoretically even cheat with the motherboard that way, but I don't recommend it. I haven't needed to look for such a thing though so I don't have any specific recommendations.

I've been rebuilding some of our old 1U storage servers into FreeNAS+ESXi boxes and one thing I noticed was this: they're pretty aggressive about revving up the PWM fans. Adding just some moderate load gets them going.

There was a funny video of a guy that accidentally stuck his finger in some high speed 120mm fan on Youtube. It dug a pretty deep gouge into his finger and he bled all over. Wowzers! Pro-tip: Do not stick your finger in a running high speed fan!

Don't pee on the third rail...?

Damn, I thought this was gonna be easy... /sigh

Edit: Boo. I just noticed those are 80mm fans. I do have the old 80mm plate so I could go back to that.

Yes, as I said, for 120MM, while you can get high pressure fans, it seems like they're bigger and more expensive and much more power-hungry, at least what I've seen in the past.

So. If you want to try the Supermicro fans (the eBay link is probably MUCH cheaper than buying them without the Supermicro badging) then yes you kind of need to think about cabling. Supermicro used standard 4 pin connectors but they're probably not long enough to reach your mainboard. You can either get them from the vendor of your choice, or if you drop me a note, we can make them here and send them to you.

 

[titan_rw]

I don't know how they compare to 'server' fans, but these fans were at my local computer store, and worked good cooling my two nas boxes:

[url]http://www.scythe-usa.com/product/acc/022/dfs1238_detail.html[/URL]

They're 120x120x38, so they're thicker than the more 'normal' 25mm thick fans. I'm running the 3000 rpm ones. I'm not sure how they compare for static pressure.

I initially had them running with the bios's 'auto' fan speed turned on. Getting into the summer though, because the room they're in isn't air conditioned, I had to turn auto fan control off, and let them run full speed. They are a bit loud at full speed, but in my situation, that doesn't matter. They're in a room dedicated to servers and stuff.

I only have them in regular PC cases, but the cases don't have room for fans in the front. These pushed enough air out the back that it got the hard drive temperatures down to about 40c worst case scenario when the room is 30c ambient.


Cyberjock, please let us know which fans work the best for your Norco case. I see one of those in my future.

 

[cyberjock]

I had read that those fans were tested to 26mmH2O, which is far better than the corsairs... I'm thinking I'll buy 3x Scythe Ultra-Kaze's. They're only rated for 30k hours, but personally I'm a little fearful of putting 5kRPM+ fans in my computers. I know hard drives aren't too keen with vibrations and putting 24 of them next to each other like they are in a 4U case is already a less than ideal situation.

I'll report back on how well they do. I have a friend with the exact same case so I might get him to buy 3 of a different fan and I can do a comparison. I'm sure the Scythe's won't work as well the Supermicro fans that jgreco mentioned but...

1. I can't find a 120mm versions. If I could(or someone posts a link before tomorrow afternoon) then I'd buy the Supermicro fans.
2. I'm not sure how good a 5kRPM+ fan will be vibration-wise with 24 drives already shoveled into a relatively small 4U case. I assume that a 120mm fan won't spin nearly as fast as the 80mm, but I don't have a Supermicro 120mm fan model number to search for. :(
3. I have WD Green drives, so I don't expect to need the same kind of cooling power that is needed in a server with all 7200RPM or faster drives.
4. Noise is a concern for me since I'd like to keep the server in a living space someday.

If someone links a 120mm Supermicro fan with similar specs to jgreco I'll convince my friend to buy those and compare for all to see. It would be nice to be able to recommend some fans that do work well and not have to guess.

Edit: And poop. Those Scythe fans were discontinued years ago. They are still available, but the bearing type used typically doesn't handle long shelf life well. :( Guess I'll do more searching.. but it looks like there may not be a 120mm Supermicro fan that has even 1/2"H2O and I might have to switch back to 80mm. Lots of disappoinment in the last 30 minutes of searching....

 

[cyberjock]

WOW. Things are just getting uglier and uglier. I was going to get those Supermicro fans with the PWM support, but:

1. The only PWM support on my motherboard is the CPU. I doubt its very smart to hook up 4 of those high wattage fans to the CPU header. :P
2. I looked for a voltage regulating fan controller that I can stick in the case(preferably if it were PCI slot mountable that would be perfect, but I can't really find any. Most are 3.5" or 5.25", which the case doesn't have. So it looks like whatever I buy might have a single speed.

So I'm back to getting the Supermicro fans even though they are 80mm. I dug around the basement and found the original plate. It turns out that the fans the case comes with are Delta AFB0812H and are rated to 4.04mmH2O at 3000RPM and a max wattage of 3watts per fan. It seems that the corsairs are more "inline" with what Norco includes in their case.

Based on the location of my server it would be easy to reinstall the 80mm backplane, so later today I'll do the switch just to compare the temps.

As for the Supermicro fans I'm still thinking of getting them. I'm very torn with what version to get.. PWM or non-PWM. Edit: You can use PWM in a non-PWM circuit, so I don't see any reason not to get PWM capable fans.

I'm thinking I'll setup a little box to test the static pressure of the 120mm fans I pull out of my case just for curiosity. Should be relatively easy to do and I'm very resourceful.

 

[jgreco]

2. I'm not sure how good a 5kRPM+ fan will be vibration-wise with 24 drives already shoveled into a relatively small 4U case. I assume that a 120mm fan won't spin nearly as fast as the 80mm, but I don't have a Supermicro 120mm fan model number to search for. :(

http://www.supermicro.com/support/resources/Thermal/#FAN

Basically none of the options look all that great. Supermicro seems to avoid 120mm fans, and none of the few offerings seem to be applicable except for maybe the FAN-0077L4.

That doesn't mean they're not out there.

Part	Watts	Speed	CFM	inwg	dBA
PFB1212UHE-F00	48	7160	250	1.4	66
TFB1212GHE-F00	30	4600	220	.9	65
AFB1212GHE	40	5200	240	1.3	62
9CR1212P0G03	87	6200	300	1.9	70

Basically it works out that you need a lot more watts to generate pressure differential.

 

[jgreco]

1. The only PWM support on my motherboard is the CPU. I doubt its very smart to hook up 4 of those high wattage fans to the CPU header. :P

So, decide whether you maybe want to cheat. Wire the fans directly to the power supply 12v. Connect the PWM from the fans to the CPU header. That only leaves a question of how to deal with tachy.

But let me make another point here. If you find yourself needing to spend a substantial sum on fans and controllers, you might want to consider the economics of a different fix: replace the system board. It might be more economical. Probably not, but just in case, think about it.

 

[cyberjock]

So here's my plan. I've ordered the Supermicro fans(80mm) and they should be here next Monday. Tomorrow evening I plan to take a "before" and "after" temperature check of my hard drives. So we'll see how my non-stock fans did versus the stock fans. When the Supermicro fans arrive I'll swap out the stock with them and provide more data.

The 80mm fans are installed in the backplane and they have a sliding tray they slide into. This allows for fan replacement without shutting down the system. Very nice and almost identical to the Supermicro cases I've looked at. They all share the same connection(a single Molex) connector so I'm going to look at getting some kind of voltage regulator to help regulate the fan speeds. My plan is to manually adjust the fan speeds(I can't find a voltage regulator that also works by temperature input to regulate temperature) but I figure since the summer is the hottest part of the year if I manually set them conservatively high they should be good enough for year-round use.

 

[cyberjock]

I have installed the OEM 80mm fans versus my Gelid 120mm fans and temps dropped 4C across the board in less than an hour. A sheet of paper sticks to the cover of the case now, so airflow is definitely improved. Now I wait for the PWM controller and the higher D/P Supermicro fans to arrive. Then we'll see how those do!

 

[cyberjock]

So, here's the final results and links for the hardware used:

Case: Norco-4224 case

Installed hardware:
CPU - E5606-2.13Ghz with powerd off
Motherboard - Some Gigabyte 1366 board - Forgot the exact model
RAM- 20GB ECC DDR3
RAID - Areca 1280ML 24-port in JBOD mode

The temperatures I'll be comparing will be the CPU, RAID controller on-chip(as read from areca-cli) and the hard drive temperatures as read from smartctl. I have had issues in the past where the RAID card overheated(very loud audible alarm) because of insufficient cooling(forgot to plug in my PCI slot fan.. whoops!) so I am very conscious of my RAID controller's temperatures now.


Summer hit and this was the first summer I've kept the server running. Since I have no central A/C the computer naturally got warmer, but hard drive temps went over 40C, then some were hitting 45C. Based on my reading of the Google white paper entitled "Failure Trends in a Large Disk Drive Population" Figure 5(page 6) for longevity of hard drive life it is preferred to keep temperatures between 30C and 40C. Below 30C and above 40C and your hard drive failure rates go up, sometimes rapidly. At 3 years the annual failure rate(referred to as AFR for the remainder of this discussion) for a hard drive between 40-45C is over 10% and approaches 15% at as you pass 45C+. Based on this I want the hard drive temperatures to stay 30-40C for the longest lifespan. If you enjoy replacing drives at a higher frequency because you enjoy the exercise of resilvering a zpool, then you are definitely in the minority.

When I bought the Norco 4224 case I installed 24 drives. All are WD Green drives that are now 2-3 years old and a combination of 2TB and 3TB drives. There are 2 zpools of drives, with the associated drives bought at the same time.

When I bought my case I immediately dismissed the 4x80mm fan backplate and replaced it with the upgrade 3x120mm fan backplate and 3xGELID 120mm fans that are effectively bearingless. The case was also outfitted with 2x80mm Gelid fans in the back of the case for exhaust air. The choice of fans was a personal one based on prior experience with these fans and their unusually long lifespan. I prefer fans that I install and never look at again. The case comes with 4x80mm Delta(model AFB0812H) fans for comparison.

Unrelated sidenote(if you don't have WD Green drives you can skip this section): Because I bought WD Green drives I have used the wdidle3.exe tool to change the Intellipark technology(head parking timeout) from the default of 5-8 seconds to 300 seconds. A video showing how to do this from DOS(the tool is not available in any other OS) can be found at View: https://www.youtube.com/watch?v=gqY5zlCXQmU
. Keep in mind that the latest version of the tool is 1.05 and hasn't been updated for a few years. If you choose not to use this tool you can cause premature failure of your hard drives(and a small performance penalty) due to excessive load cycling of the hard drives. They are rated for 250,000 to 350,000 cycles. In a desktop environment this can take several years, but in a RAID environment you can allegedly hit the rated lifetime in a matter of months. There have been rumors that some people have had RMAs denied due to very high load cycling values in the SMART data on the drive, so be warned. It's a foregone conclusion that if you rack up 200k cycles in 3 months then you weren't using the hard drive within its designed function and your warranty may not be honored. Because I'm using WD Green drives I don't consider it an option to buy cooler drives in order to achieve the 30-40C temperatures, but I can assure you that using high performance drives will result in higher temperatures(duh) and may require you to take actions. A friend that had 16 7200 RPM Seagate drives had major temperature issues with the drives idle(sometimes hitting 50C) with the 80mm fans that the case comes with. So keep that in mind when considering if you want to upgrade your fans. I'd wager that if you choose to use anything except "green" drives you will need better cooling than the case can provide with the pre-installed hardware.

Earlier this summer I found that I started having hard drive failure. Up until this point I had one failure in February and none in the previous 3 years. Within about 2 weeks I had 4 more drives fail(thank goodness for redundancy). Looking at SMART data I determined that my hard drives were getting far too hot and I needed to do something about it. The information below is based on that information.

I have 5 sets of data for you: The 120mm backplate with the GELID fans, the 80mm backplate with the Delta fans that came with the case (AFB0812H) and Supermicro FAN-0126L4(NIDEC # V80E12BHA5-57). As some extra datapoints I have paired the Supermicro fans a PCI fan controller an provided temperatures at minimum fan speed and 50%.

I chose to use a fan controller for several reason: I believe these fans are far too powerful and will cool the hard drives below 30C, they are extremely loud(and I mean excruciatingly loud), and I prefer to minimized excessive vibrations of the case and therefor the hard drives. The fans at full speed are so loud I can hear them on my ground floor with the basement door closed. Going down in to the basement I feel like I need earplugs in my ears. Note that in the below values I have normalized the temperatures to 20°C(68°F) since the basement stayed between 67F and 70F. I verified that no scrub was in progress or any other heavy loading tasks for at least 2 hours before obtaining temperatures. All temperatures are in °C unless otherwise specified.

A few notes before I jump to the temperatures:

1. Running a zpool scrub(or resilver) will result in a measurable increase in temperatures. Because of this you ideally want to stay in the lower end of the 30-40°C range I'm attempting to hit.
2. A single fan failure with multiple fans in parallel can result in losing most of your cooling capacity because the air will take the path of least resistance. Using fans that are reliable is a major plus. I don't consider the Delta fans included to be particularly reliable as many people have complained that the fans died after less than 3 months of use.
3. The Supermicro fans are thicker than the Delta fans. It will be a tight fit, but they will fit.
4. The Supermicro fans I purchased are the 4 pin PWM fans. I chose not to use the PWM at this time, but I bought the PWM fans in case I have a need for PWM in the future. PWM controlled fans will work in a VR fan controller, but their speed control will be more logarithmic than linear. I purchased my fans from ebay for $12 each plus shipping. Total cost was just over $50 for all 4 fans on my doorstep.
5. The 80mm Norco fan mounting plate has slots for the fans to slide in and out. The fans have 4 pin connectors, but only 3 leads are attached. The connectors on the Delta fans are the industry standard while the Supermicro have slightly different dimensions and not as long. The fan will easily slide into the fan slot if you remove the bracket from the Delta fans and install them on the Supermicro fans. Despite the deeper Supermicro fans the fan will fit, but you will not be able to use the Supermicro connector with the backplate connector. You will have to run your own power to each of the fans. Since I used my PCI fan controller I had no issues using a 1 foot 3-pin extension cable to connect the fans to the controller. Also you cannot simply replace the plastic connector on the 2 fans to make the Supermicro fan work with the Norco plate. If you wanted to do some splicing you could accomplish this task, but all 4 fans share the same molex connector so you'd need a fan controller capable of handling all of the fan load. Because of this I recommend you use the Norco fan slot hardware to mount the fan but use your own power source and appropriate 3-in fan extension cables.

Now onto the temperatures:

Temperatures with the 3x 120mm GELID fans as it was in my living room: (Estimated living room temperature was about 26C(78F) at the time)

Hard drives(min/avg/highest) : (35/40/43)
CPU: 43°C
RAID: 51°C

Temperatures in the basement with the 120mm fans:

Hard drives(min/avg/highest) : (35/40/43)
CPU: 42°C
RAID: 50°C

Notice that although the environment is much cooler, temperatures are pretty much the same.

Temperatures in the basement with the 80mm OEM(Delta) fans: (257watts at the wall for my system)

Hard drives(min/avg/highest) : (30/34/37)
CPU: 38°C
RAID: 45°C

Note that hard drive cooling seems pretty adequate for idle. When running a scrub the temperatures creep up and about 1/2 the drives go over 40°C.

Supermicro fans without a controller: (303watts at the wall for my system)
Hard drives(min/avg/highest) : (23/26/27)
CPU: 32°C
RAID: 37°C

WOW, talk about a "cool" server! The top and bottom row of hard drives are the warmest, with all 8 drives being 27°C. I never would have thought these temperatures possible with a high density case such as this.

Finally, I installed a fan controller(Sunbeam PCI fan controller PL-RS-PCI). These are discontinued and I can't find these for sale anywhere except Amazon, and they want $90 each! This is better than last week when they were asking just $133 each. Naturally, any fan controller will do, but the PCI controllers are the best option I could find since there are no 3.5" or 5.25" bays with which to install most fan controllers.

Warning: The fan controllers are rated for 20W each and the fans are rated for 12W each. I used 2 controllers to control 2 fans each. Since I don't have any intentions of using the controller above about 1/2 way I'm not worried about damaging anything. I did try turning the fans up to 100% load and system load only went up to 288 watts and the fans weren't nearly as loud as they were without the controller. My guess is there is a regulator that limited power to 20w per controller since the unregulated 12v to the fans yielded a system load of 303 watts. Whether this was by design or because the controller was overloaded I don't know.

Couple of notes:

- On minimum fan speed the airflow feels roughly the same as the Delta fans when they were at full speed.
- On about 50% fan speed the airflow is fairly high. Far higher than the Delta fans could have ever achieved. About 50% is where the fan noise become high enough to begin to be a concern if I had this server installed in its closed rack in the back of my living room.
- Even at the minimum fan speed on the controller a sheet of heavy stock printer paper sticks to the case. This is a significant improvement over my "upgraded" 120mm fans which couldn't hold up a standard sheet of paper.

Supermicro fans at lowest setting on the fan controller: (I had an issue with this setting.. more information is below) (255 watts at the wall)
Hard drives(min/avg/highest) : (33/35/38)
CPU: 41C
RAID: 39C

Supermicro fans at 50% on the fan controller: (272 watts at the wall)
Hard drives(min/avg/highest) : (27/30/31)
CPU: 34C
RAID: 39C

At minimum fan speed I had some very odd problems with my server. I left the server to run for 5 hours idle to allow temperatures to stabilize. I went to a friend's house and when I came home I found out that I couldn't obtain SMART data for any of the drives on my RAID controller. The zpool showed all drives as online. I can't explain this condition because I had been running the temperature script for 3 weeks and never had this problem. After a reboot I got my temperatures for the hard drives. Within an hour 4 more drives wouldn't report SMART data from my Areca controller but the zpools still showed as online and my data was available. If 4 drives had actually been disconnected the zpool wouldn't have been able to function. Additionally, the areca-cli would not run. Attempting to run the program would simply result in a blinking cursor and the SSH session would become unresponsive. The system has an OCZ gold rated 850W PSU and the fans aren't electrically connected to any other hardware except potentially internally to the power supply. I checked the cabling is tight and nothing seems out of order. I really can't explain this behavior, but as soon as I increased the fan speed to 50% and rebooted I have yet to see a drive drop. It could be that at minimum fan speed there's some kind of electrical feedback from the fans or controller resulting in my hardware misbehaving. It's been about 5 hours since I changed the fan speed back to 50% and my intention is to keep the fans at this speed for the forseeable future(temps and noise are acceptable). I do have some upgrades for the FreeNAS hardware that are likely to happen in the next 2-3 weeks. If the issue reoccurs I will update the forum, but as I'm about to gut my system and do some upgrades we may never know the exact cause.

So let's talk about fan theory for a minute. There's a relationship between RPM, fan diameter, fan power consumption, noise, thickness of the fan, D/P(differential pressure) of the fan, and flow rate(there's more than this, but these are easily understood and usually available to the purchaser). In particular, D/P and flow rate matter the most for situations where high air resistance plays a part in cooling. As the fan diameter gets bigger you need the square of the depth of the fan to attain the same differential pressure assuming all other factors are the same. Unfortunately, all factors are not the same because a larger diameter fan running at the same RPMs will have its own centripetal force increase due to the increased mass of the fan blade related to the larger diameter. Fans that blow themselves apart when running at full speed generally aren't sold to the public so 120mm fans typically have a lower RPM than an 80mm fan of similar characteristics.

Put simply, bigger fans are great because they can move more air at a lower RPM with low air resistance(keyword: low air resistance). If you start dealing with situations where you have higher air resistance(such as a 24 disk 4U rack case) then you need to avoid 120mm fans because they will become excessively thick(remember that square of the thickness thing) and won't physically fit in a case. In my opinion, buying the "upgraded 120mm" fan backplate is no upgrade at all. I couldn't find any 120mm fans of any reasonable depth that had even 1/2 the D/P of the Supermicro fans I purchased. I had found 1 120mm fan that was something like 70mm thick and had 1/2 the D/P of the Supermicro fan. If you want better cooling than the 80mm fans that the case comes with, look at higher D/P fans such as the Supermicro fans I purchased.

I hope this helps out someone that is considering buying a 24U case and are concerned about the drive temperatures. Personally, i think my configuration is pretty ideal for "green" drives. If I were to go to 7200RPM(or higher) drives I'd highly recommend you buy better fans than the case comes with.. I think that the Delta fans were acceptable (but not ideal) for"green" drives, but if you want extra margin you need to get better fans.

This whole thing is brought to you by me... since I had 4 disks fail in short order due to hard drive temps. Yes, this was an expensive lesson learned since only 1 of the drives was in warranty still. But I'd rather spend $50 on fans than spend even more money on lots of RMAs, buying new drives when they are out of warranty, and resilvering disks regularly. I value my data and I don't see any reason to skip on cooling.

Anyone that wants to tell me that temperatures of a hard drive don't matter.. well, I'll keep my opinion of you to myself. :p

 

[jgreco]

Well, that's one hot message...!

 

[KMR]

I have an IStarUSA d-416 case which don't come with a backplane like the Norco units but the two cases are the same width and height. Do you think the back plane from the Norco unit would fit in another case of similar dimensions? I know they wouldn't be a direct drop in but it would make my project a lot easier of I could hammer one of those in place. Just a thought.

 

[cyberjock]

Literally, the backplane is mounted via screws and holes in the case. Very simple design. If you are capable of using a drill I'd bet you could easily buy the backplane, drill small screw holes in your case and mount the backplane in your case without any issues. The screws just go through the case. The threaded part is in the backplane. I actually considered moving my backplane back 2 inches or so because of space limits for the fans. All I'd need to do is drill some holes in my case just like you'd need to do for yours.

 

[jgreco]

Are we talking the backplane or the fan bulkhead here? Because the backplane is probably not useful except in a Norco case, because the mounting is a three-dimensionally-precise thing to line up with the drive trays.

 

[KMR]

Sorry, I think I'm getting my terminology confused here. I am referring to the piece of sheet metal that the fans mount to in the case. I guess that bulkhead would be the proper word.

Cyberjock: Thanks. That is what I was hoping for. I think I will order one of these up and see if I can get it working.