SOLVED SuperMicro SC836 HDD Temps Too High / Fan Replacements / Your Experiences?

[svtkobra7]

Hi,

As my first foray into a non-COTS server, I purchased a used SuperMicro Super Chassis SC836E16-R1200B via ebay. Having sold my QNAP NAS appliances to offset the cost, I was left with 16 HGST Deskstar NAS HDDs - 6TB that needed to be repurposed. I spent considerable time researching and planning my build and it is well noted that the Hardware Guide explicitly recommends NOT using 7200 RPM hard drives due to heat produced; however, in my scenario it would not have made sense to source 5400 RPM drives when I already had several $k worth of 7200 RPM NAS HDDs looking for a home.

I've thought for a number of weeks that I had some sort of hardware or software misconfiguration, as the only fan profile that I'm able to achieve sub-40°C HDD temperatures with is the "Full Speed" fan profile. Needless to say, running the server with that fan profile in a home environment is not a sustainable solution. As of last night, I believe I may have identified the issue, but I'd like to share my issue and data with the community for your thoughts / confirmation.

• Specifically, I'm curious what HDD temperatures other forum members have encountered with the same chassis,
• I'd like your thoughts as to whether not having the correct fans installed is the cause of high HDD temperatures,
• and finally, your recommendation on the the most appropriate (a) chassis middle fan and (b) exhaust fan for this chassis.

Without further preface ...

1. Define: Achieve HDD temperatures of 40°C or lower at a "reasonable" sound level and at desired ambient temperature, 78°F.

• 78°F is the ambient temperature I find most comfortable during the summer and 72°F is far too cold.
  
• I don't want to incur the additional cost of lowering ambient temperature to 72°F to achieve the goal (I'm sure I'm already in for a treat when the power bill arrives with dual E5-2670s and 16 spinning disks @ 7200 RPM ).
  
• Modifications to hardware configuration are absolutely acceptable solution (and one I propose at that). If air shroud removal and usage of active CPU coolers would help, please let me know (I've read differing opinions as to whether the air shroud should be left in place).

2. Measure: HDD temperatures were recorded for each drive at each fan profile with ambient temperatures of 72°F / 75°F / 78°F using Spearfoot's "get_hdd_temp" script.

• At each intersection of (a) fan profile and (b) ambient temperature combination, 3 measurements were recorded at t0, t+15 min, and t+30 min. For each of those intersections, the mean of those 3 measurements was used for analytical purposes.
  
• Where ambient temperature was decreased or fan profile was changed, a given profile was allowed to operate for a minimum of 30 minutes once ambient temperature was stable.

3. Analyze: The full speed fan profile is the only profile that achieves sub-40°C drive temperatures at the desired ambient temperature, 78°F. The standard and optimal profiles produce results with a maximum variation of 0.4°F at any measured ambient temperature, but fail to achieve sub-40°C drive temperatures even at the lowest ambient temperature used, 72°F. A summary of my data follows, and I'm happy to pass along the Excel file if you would like to review.

4. Improve: To be completed. 5. Control: To be completed

Proposed Explanation

• I believe the explanation as to why I cannot keep my HDDs cooler and why only the "Full Speed" fan profile can achieve sub-40° temperatures has been staring me in my face, I just have been too blind to see it. Other items that suggest fans are the issue: (a) CPU2 (rear) measuring 20°C warmer than CPU1 (front), and (b) realizing that my fans where not spinning at 100% of what I thought should be their maximum rotational speed. Yet, I didn't connect the dots until last night.
  
• After much (over) analysis, I looked at the fans installed in the server (as purchased from ebay), and compared them to the SuperMicro's specifications for the SC836E16-R1200B and noticed that they differed.
• Those familiar with the chassis / backplane know that the "cut outs" in the backplane for airflow are quite small and the "chassis middle" fans pull air from the front of the chassis / hot swap drive bays through the backplane via these "cut outs". As such, I would assume a fan rated at as high a static pressure as possible is needed to effectively cool the HDDs.
• Per the SuperMicro System Fan Matrix, the 3 chassis middle fans fans that came with the system are listed at 0.85 in. H20 static pressure, vs. the specified fan for the SC836E16-R1200B which is rated at 1.09 in. H20 static pressure. The installed fan's speed is lower by 700 RPM and airflow is 17.8 CFM higher than the specified fan.
• Again, per the SuperMicro System Fan Matrix, the 2 exhaust fans that came with the system are listed at 0.49 in. H20 static pressure, vs. the specified fan for the SC836E16-R1200B which is rated at 0.68 in. H20 static pressure. The installed fan's speed is lower by 1.7k RPM and airflow is 0.5 CPM lower than the specified fan.
• In summary, I assume those static pressure and air flow specifications are at maximum speed, where I'm missing 0.24 in. H20 at the backplane (per fan x 3) and I'm missing 0.19 in. H20 at the rear (per fan x 2). The "Optimal Speed" Profile uses a base fan speed at 30% PWM duty cycle, so perhaps the variance in static pressure is much larger at that duty cycle, which would account for its ineffectiveness? My question to you: Does this proposed explanation make sense?
• As an ancillary question, I have the backplane fans connected to FAN2-4 and the exhaust fans connected to FAN5-6, is this the correct? My understanding is that since all fans are deemed "CPU / System" fans, they belong in Zone 0 (FAN1-6) and there is no need to use FANA-B, Zone 1, for Peripheral Zone Fans.
• Note: A full summary of my hardware configuration including fan specification follows at the end of this post.

Proposed Solution

• Replace 2 x Exhaust Fan (80 x 80 x 32 mm). For this fan size, the Fan Matrix shows the only alternative to the incorrect fan I have (FAN-0062L4) to be the same one listed in the specifications for the SC836E16-R1200B. That correct fan is the FAN-0125L4.
  
• Replace 3 x Chassis Middle Fan (80 x 80 x 38 mm). For this fan size, the Fan Matrix shows 17 fans. Should I procure the one specified for the SC836E16-R1200B, 3 x FAN-0126L4, or is there a better alternative?
• Note: I've summarized the Fan Matrix in a shared spreadsheet located here: Fan Matrix - Extract.

Thank you in advance for your review and feedback.

*** EDIT: Results with replacement fans available in post #25 of this thread: SuperMicro SC836 HDD Temps Too High / Fan Replacements / Your Experiences? ***
____

[Chassis Middle Fans] / {Specified, per Supermicro's website} or (Installed, as Purchased)

• {Specified} = 3 x FAN-0126L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 7.0K, Airflow (CFM) = 72.5, Static Pressure (in. H20) = 1.09, Noise (dBA) = 53.5
• (Installed) = 3 x FAN-0094L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 6.3K, Airflow (CFM) = 90.3, Static Pressure (in. H20) = 0.85, Noise (dBA) = 51

[Exhaust Fans] / {Specified, per Supermicro's website} or (Installed, as Purchased)

• {Specified} = 2 x FAN-0125L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 6.7K, Airflow (CFM) = 59.6, Static Pressure (in. H20) = 0.68, Noise (dBA) = 47
• (Installed) = 2 x FAN-0062L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 5.0K, Airflow (CFM) = 59.1, Static Pressure (in. H20) = 0.49, Noise (dBA) = 47

Hardware Configuration

• Chassis: SuperMicro Super Chassis SC836E16-R1200B - 3U 16-bay
• Motherboard: SuperMicro X9DRi-LN4F+, 2 x Intel Xeon Processor E5-2670 (passively cooled), 48GB ECC RAM
• Backplane / HBA: SuperMicro BPN-SAS2-836EL1 / LSI SAS 9211-4i
• Storage Drives: 16 x HGST Deskstar NAS HDD - 6TB (8 in service + 8 in burn-in + 4 in transit)

[FDD/DVD Size Drive Bays Above 3.5" Drive Bays - Position] / {Original} / (Replacement)

• [LEFT SLOT] / {DVM-PNSC-DVD-SBT3: DVD-ROM Drive} / (Open, to be replaced with MCP-290-00036-0B: DVD Dummy Tray for 2.5" HDD)
• [MIDDLE SLOT] / {CP-220-00007-01: USB/COM Port Tray} / (MCP-220-00023-01: USB Dummy Tray for 2.5" HDD, populated with Intel 320 Series SSD - 40 GB)
• [RIGHT SLOT] / {MCP-290-00015-01: FDD Dummy Cover} / (MCP-220-83601-0B: FDD Dummy Tray for 2.5" HDD, populated with Intel 320 Series SSD - 40 GB)

[Cross Posted @ STH - I've discussed the issue at length with members from both forums and would like to give all the opportunity to view replies]

 

[BigDave]

In summary, I assume those static pressure and air flow specifications are at maximum speed, where I'm missing 0.24 in. H20 at the backplane (per fan x 3) and I'm missing 0.19 in. H20 at the rear (per fan x 2). The "Optimal Speed" Profile uses a base fan speed at 30% PWM duty cycle, so perhaps the variance in static pressure is much larger at that duty cycle, which would account for its ineffectiveness? My question to you: Does this proposed explanation make sense?

IMHO You have struck the nail on the head. Using those 7200 rpm drives (with that ambient room temp.) means the higher static pressure fans (at lower rpm) are
going to be the only way to keep 'em cool.
I live in the Texas heat and have nothing but sympathy for you ;)
Also I would mention that the front end of your server must be populated with dummies where you might have "holes' caused by missing components such as optical drives,
floppy drives, usb/serial panels AND empty hard drive bays. All or any of these "open" spaces cause the air to take the path of least resistance and reducing the amount
of air flow over your drives. Take some blue painters tape and plug up all the "holes" and watch the drive temps go down (as a temporary solution of course).

It's currently 76F in my home and and my script report emailed me this morning with the highest drive temp 35C.
When it's 105F outside in the summer, my Air Conditioner runs 24hrs a day to keep the house at 80-81F, at which point
I must move the fan profile to FULL to keep those drives under 40C. I run a rack mounted Supermicro SC836 chassis with
all five fans plugged into the motherboard (X9-SCM-F) with the stock shroud installed and a passive heatsink on the CPU.

 

[Spearfoot]

As you know, we have the same motherboard, and my experience matches yours w/ respect to CPU2 running hotter than CPU1. Makes sense, owing to the physical location of CPU2, which is behind CPU1 in the air stream.

You might try installing these active CPU coolers, which ought to work in your 3U chassis:

http://store.supermicro.com/heatsink/2u-active-cpu-cooler-snk-p0048ap4.html

 

[svtkobra7]

@BigDave

Thanks for the feedback. Believe it or not, I did try that previously, but didn't notice an improvement. Early days before I approached methodically, so I will give it another go.



To confirm:

• Should I cover just the DVD-ROM slot which is wide open at the moment?
• Or are you suggesting to cover the entire top "strip" above the 3.5" HDD bays?
• If those holes in all three slots need to be perfectly plugged, it would be rather easy to pull those dummy trays and drives and line the inside of the grated face with black felt or something similar, completely restricting airflow, while not being aesthetically unpleasing.

Appreciate the sympathy! I actually spent a year in Texas working on a project so I know exactly what you are going through too. I'm assuming you have the correct fans in your 836: 3 x FAN-0126L4 + 2 x FAN-0125L4?

Thanks again.

 

[svtkobra7]

@Spearfoot

You know I have my eyes on a pair of those coolers. Not telling you anything you haven't imparted upon me, but the upside to active CPU cooling would be: (a) more uniformly and better cooled CPUs and (b) the ability to decouple System cooling (backplane / exhaust) from CPU cooling.

My only hesitancy in doing so: I've seen a SuperMicro recommendation not to remove the air shroud. That hesitancy is a bit strengthened having most likely determined the root cause here, as it suggests everything was engineered very precisely and with consideration for the system as a whole. Inadvertently using fans with a lower static pressure than designed has a considerably negative impact on HDD temps, but also an imbalance in CPU temps, and the rear CPU not being adequately cooled.

To my point, that shroud serves a purpose: to ensure adequate airflow across the CPUs and memory. With those nice coolers, you can remove the CPUs from that concern, but what are your thoughts as to any negative impact to RAM temps?

Thanks again for your offline assistance and contributions generally, your scripts are extremely helpful and made it quite easy to collect data here. :):):)

 

[Dice]

@svtkobra7 Kudos for a great post with lots of effort and valid conclusions.

 

[BigDave]

I'm assuming you have the correct fans in your 836: 3 x FAN-0126L4 + 2 x FAN-0125L4?

Actually I run FAN-0094L4 and Fan-0062L4
My server began life as a TQ model with 500w (read noisy as hell) PSUs. I changed to a SAS2 backplane and 920w SQ power supplies.

If those holes in all three slots need to be perfectly plugged

I would not say that, but plug up the now open DVD slot for sure. The other two SSD adapter trays (if now installed) won't make enough of a difference.
Just to double check, are all your drive trays populated with drives or plastic dummy trays? Empty bays will let in a lot of air, tape them off
or install dummy trays for sure.

This is my hottest running drive of the 12 in this chassis

Code:

[root@freenas] ~# smartctl -l scttemp /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SCT Status Version:  3
SCT Version (vendor specific):  258 (0x0102)
SCT Support Level:  1
Device State:  Active (0)
Current Temperature:  34 Celsius
Power Cycle Min/Max Temperature:  25/38 Celsius
Lifetime  Min/Max Temperature:  17/43 Celsius
Under/Over Temperature Limit Count:  0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:  2
Temperature Sampling Period:  1 minute
Temperature Logging Interval:  1 minute
Min/Max recommended Temperature:  0/60 Celsius
Min/Max Temperature Limit:  -41/85 Celsius
Temperature History Size (Index):  478 (7)

Index  Estimated Time  Temperature Celsius
  8  2017-05-19 04:58  35  ****************
 ...  ..( 41 skipped).  ..  ****************
  50  2017-05-19 05:40  35  ****************
  51  2017-05-19 05:41  34  ***************
 ...  ..( 96 skipped).  ..  ***************
 148  2017-05-19 07:18  34  ***************
 149  2017-05-19 07:19  35  ****************
 ...  ..(335 skipped).  ..  ****************
  7  2017-05-19 12:55  35  ****************

 

[svtkobra7]

Actually I run FAN-0094L4 and Fan-0062L4
My server began life as a TQ model with 500w (read noisy as hell) PSUs. I changed to a SAS2 backplane and 920w SQ power supplies.

OK so same fans - did it come with them or you select them (just curious)? The PWS-920P-SQ is on my list, but at the moment, the fans are MUCH louder than the PSU.

I would not say that, but plug up the now open DVD slot for sure. The other two SSD adapter trays (if now installed) won't make enough of a difference.

10-4, thanks. Yup, the other two (USB Dummy + FDD Dummy) are installed and have SSDs in them.

Just to double check, are all your drive trays populated with drives or plastic dummy trays? Empty bays will let in a lot of air, tape them off
or install dummy trays for sure.

Take a look at this - as it shows what is populated and what is not. Columns 1 - 3 / Slots 0 - 11 are populated. Column 3 / Slots 8 - 11 are currently in the burn-in process running the 4th and last bad blocks pattern. Column 4 / Slots 12 - 15 is not populated, but has empty drive caddies inserted. I RMAed 4 drives and they are on their way back from HGST and will populate that empty column once received.


Your drive temps rebut my theory as to the cause of my high HDD temps if you are running the same fans ... hrm ...

 

[StephenFry]

Interesting thread. It's impressive to me that the fans you've got are managing to get the (high but not crazy) temps you're getting.

For my 7200rpm builds that are even a little noise sensitive, I've completely ditched all rackable chassis as a base.
Not ever going to get them decently cool outside of a 15C data center, nor reasonably sounding. For homebrew machines of up to 16 drives (and even more), there are enough cases that provide a ton more room and as such can be easily and quietly cooled.

 

[svtkobra7]

@BigDave

Take a look at this - as it shows what is populated and what is not.

I'd be happy to pass this Excel file along if it would be of any use to you. It parses the text output of @Spearfoot 's "get_hdd_temp" script perfectly and allows you to dynamically compare profiles / ambient temps using combo boxes in Excel. But it doesn't look like you have too much need for analysis looking at the SMART data you posted.

 

[BigDave]

OK so same fans - did it come with them or you select them (just curious)?

I purchased my chassis used and those models were installed at that time.

Slots 12 - 15 is not populated, but has empty drive caddies inserted.

This is all good then, you should have good air flow through your backplane area (no "holes").

But it doesn't look like you have too much need for analysis looking at the SMART data you posted.

Your offer is much appreciated, but I'm good!

Your drive temps rebut my theory as to the cause of my high HDD temps if you are running the same fans ... hrm ...

A couple of things we might take into consideration.

1. My server at idle vs. yours running 4 instances of badblocks
2. My ambient 76 vs. yours being higher at 78
3. My chassis is near the bottom of a 42U enclosure with the top and door venting. vs. yours???

 

[svtkobra7]

My chassis is near the bottom of a 42U enclosure with the top and door venting. vs. yours???

Nothing but the sweetest, finest Georgia air for my baby. 12" off the ground where the densest, coolest air hangs about.

A compounding issue, I believe, is that the server is in my home office, where that room is served by only one supply vent and there is only one return in the entire condo, on the air handler in a utility closet. Point being, this condo provides less than optimal air flow and although the thermostat may be registering 78°F, I highly doubt the ambient in that room is 78°F. My ultimate plan was to vertically rack mount it in the "server closet" (OK - coat closet with structured media enclosure for Cat5e + Coax termination). Then create a vent nearly at floor level for cool air intake into the closet from the living area and also create a vent at ceiling level for warm air exhaust out of the closet. However, I'm not sure if convection just "works" in that scenario and I'm better off there despite the smaller size of the closet.

Air Flow:
Living Area >> Server Closet >>>> Utility Closet (with Air Handler and sole air return in condo)
where >> is a vent allowing cool air intake to the Server Closet, at floor level,
and where >>>> is a vent allowing warm air to exhaust out of the Server Closet, at ceiling level,
and by vent I mean a hole in each side of the drywall, covered by register that allows as much air through as possible.

Interesting: I made three changes that netted the best results I've seen yet:

• I received notification that a drive hit 45°C at 78°F, so I turned it down to 75°F;
• I taped off the entire top row of DVD/FDD drive size slots with painters tape;
• I removed the empty 3.5" caddies in Column 4 and taped off that column with painters tape.

I've held steady at 36.5°C for the past hour at 75°F "ambient", where I previously averaged 37.5°C, so I dropped 1°C. When I tested previously, I didn't even hit 36.5°C when ambient was at 72°F, rather I was at 36.8°C. Despite not seeing it previously, forcing those fans to pull air from 12 bays instead of 16 + 3 drive slots above seems to have a favorable impact.

 

[melloa]

One question:

- Are you measuring the temperature @ your server room or getting from the ap thermostat located somewhere else?

Also:

• - Move to the closet: If temps are already too high, inside the closet will probably be hotter;
• - Vertical mounting: I'm sure you know that:

1. You will be pulling from the front of the chassis, exhausting from the back;
2. Hot air goes up;
3. Will increase the intake temperature as it goes;

• You will need a pretty good support on the sides to support the weight;
• Removing from the rack for maintenance will be hard.

Excellent post, by the way. Detailed with lots on information. That shows how much you've researched.

 

[svtkobra7]

Mr. Mello, quite nice for you to make an appearance here. :) I hope that your brief travel was relaxing. To your question, if I may quote myself:

Point being, this condo provides less than optimal air flow and although the thermostat may be registering 78°F, I highly doubt the ambient in that room is 78°F.

• I unfortunately do not have any other means to measure temperature at the moment.

- Move to the closet: If temps are already too high, inside the closet will probably be hotter

• Probably, but I would think airflow would be improved with the planned inclusion of the intake vent @ floor (cool air enters) and and exhaust vent @ ceiling (warm air exits). I'm by no means an expert on fluid dynamics or thermodynamics, but I think the fact that the exhaust vent will direct air to the utility closet which contains the sole return vent in the condo will have some favorable impact on air flow. Your thoughts?

1. You will be pulling from the front of the chassis, exhausting from the back;
2. Hot air goes up;
3. Will increase the intake temperature as it goes;

• Technically, warm air, being less dense than cool air, is displaced, but I know what you mean. ;)
• My (potential) mitigation strategy for the concern of the exhaust air impacting intake temps includes removing that exhaust air from the equation by way of using ducting to the top of the closet where the exhaust vent is located. Of course, I won't know if that option need be exercised until I get the server in the closet.

Excellent post, by the way. Detailed with lots on information. That shows how much you've researched.

• Thank you very much for the compliment.
• Thank you moreso for all of your offline help - you have certainly been a lifesaver for me and are an asset to the community.

I hope your weekend is going well.

 

[svtkobra7]

As you know, we have the same motherboard, and my experience matches yours w/ respect to CPU2 running hotter than CPU1. Makes sense, owing to the physical location of CPU2, which is behind CPU1 in the air stream.

You might try installing these active CPU coolers, which ought to work in your 3U chassis:

http://store.supermicro.com/heatsink/2u-active-cpu-cooler-snk-p0048ap4.html

How would you rate the noise level of these coolers which I'm strongly considering?

I was reading AMZN reviews on them and there were a number of complaints about noise, but those individuals may not have a "calibrated" opinion if they haven't spent time next to a server, so I'm curious as to your thoughts.

Also, if I may be lazy, and since you know the answer with the same board, (a) do brackets need to be purchased separately (I've seen them being sold as such)? (b) did your install require extenders for any of the fan cables?

Thanks and I hope you are having a good weekend.

 

[BigDave]

Probably, but I would think airflow would be improved with the planned inclusion of the intake vent @ floor (cool air enters) and and exhaust vent @ ceiling (warm air exits). I'm by no means an expert on fluid dynamics or thermodynamics, but I think the fact that the exhaust vent will direct air to the utility closet which contains the sole return vent in the condo will have some favorable impact on air flow. Your thoughts?

Favorable impact on air flow when the air handler blower is running in A/C mode, but this will not be the case 24/7 over an entire calendar year right?
You could install a stand alone fan in the top of the closet blowing out and causing cooler air to be
drawn in at the bottom vent. If said fan was large enough in dia. and ran at slow speeds (read silent).

 

[svtkobra7]

Favorable impact on air flow when the air handler blower is running in A/C mode, but this will not be the case 24/7 over an entire calendar year right?
You could install a stand alone fan in the top of the closet blowing out and causing cooler air to be
drawn in at the bottom vent. If said fan was large enough in dia. and ran at slow speeds (read silent).

Favorable impact if HVAC fan mode = Auto (and air or heat turns fan On) OR if fan mode is set to On. So leaving fan mode = On 24/7/365 does indeed provide 'round the clock relief to stagnating air flow, right? It may not be using the condenser in that case, but it is certainly expelling the air outdoors. I'm assuming the cost to run that fan year round is marginal, but I really don't know. But any cost increase will be offset by the fact that I won't need to run the heat during the winter. [emoji1]

What I should really do: Put a hole in the ceiling with an exhaust fan, add a duct, cut through 4 sheets of 5/8" drywall and let my neighbors enjoy the free heat. JK

I received my first power bill lsince the server has been running 24/7 last night and there was no discernible impact (I will have to look at it more closely to see how that could be).

I was thinking same on fans and have pre-wired for them if needed.

Haven't done much other than demo that drywall, move the structured media panel higher in the closet, and the electrical (added a light to the closet too). Can't recall if I mentioned the reason for removing the drywal, but it was to add additional support for a wall mounted rack. Metal studs FTL.

Thanks for your continued input.

Sent from my HTC6545LVW using Tapatalk

 

[Spearfoot]

How would you rate the noise level of these coolers which I'm strongly considering?

I was reading AMZN reviews on them and there were a number of complaints about noise, but those individuals may not have a "calibrated" opinion if they haven't spent time next to a server, so I'm curious as to your thoughts.

Also, if I may be lazy, and since you know the answer with the same board, (a) do brackets need to be purchased separately (I've seen them being sold as such)? (b) did your install require extenders for any of the fan cables?

Thanks and I hope you are having a good weekend.

Not having an appropriate meter, I can only give my subjective observations...

My 'new' server is a Supermicro X9DRi-LN4F+ motherboard with dual E5-2660 Xeons housed in a 4U Supermicro CSE-846 chassis. The system came with dual 920W SQ power supplies - which really are very quiet - and had 5 fans installed: 2 rear exhaust fans and 3 fans in the mid-chassis 'fan wall'. I added a Zalman FB123 bracket w/92mm fan to blow directly on the PCIe adapters slots, which are populated w/ (3) LSI 9310 HBAs, a Dell H200, and an Intel X520-DA1 NIC. I also removed the air shroud and added two of the Supermicro CPU coolers (part number SNK-P0050AP4). Though I'm running the CPU fans at full speed, they have had no noticeable impact on the overall noise level of this server. (To answer one of your questions: I didn't need cable extenders in order to plug in these fans.)

While I wouldn't want it located next to my desk, the server's noise level isn't too bad at all. It lives out in my shop, and is completely drowned out by my shop's AC unit when the later is running. I suffer no discomfort whatsoever tinkering in my shop, despite multiple servers running there.

Earlier you asked about keeping the RAM cool... this hasn't been an issue for me. It seems that the 3 middle fans provide plenty of air flow over the DIMMS - with or without an air shroud.

 

[Stux]

You know, there are only two modes which spin the peripheral fans at 100%, Max and Heavy IO.

All the other modes spin the fans in relation the CPU temperature, which is not what you want.

If a supermicro system can only be cool enough on max, you should check what fans are connected to where, and run a custom fan controller script.

Here's mine: https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/

Should work on x9 but will probably need a BIOS update.

 

[svtkobra7]

Not having an appropriate meter, I can only give my subjective observations...

Thank you, I'm sold. Your subjective observations are more than good enough for me.