Server Closet Cooling - Intake Fan Recommendations?

Status
Not open for further replies.

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
(I have no idea what the static pressure would be of course)
You should have such because I sent you the spec sheets that have the characteristic curves.

Anyway, this all seems moot based on your recent post. 80F ambient, huh? I keep my office at 69 presently to keep my equipment cool. Sounds like you are good to go.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
There is one return air "vent" in the condo, and that sits on the underside of the air handler in the utility closet next to the server closet.
Things are not as I had imagined. Your air cond/furnace is an "updraft" unit and therefore draws all return
air from the lower plenum at floor level. Your mention of concrete above the open closet also tells
me that your place is single story, again no reason for return air to be located above the ceiling.
I don't follow the comment about the handler drawing air from the closet with a ceiling in place? Perhaps you could explain that to me.
I was making assumptions about this mysterious draw of air under the closet door, knowing what I've learned
from you about your system, I can now say with confidence that this has to be natural convection with the
heat produced by your electronics and server.
I think you should install the fan you purchased and duct it into the space as you have described. Hang your
sheetrock and I agree with @Redcoat , you are good to go! At worst case scenario, you will need to possibly
install the second through the wall fan at the bottom of the wall. Stay Cool :cool:
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
You should have such because I sent you the spec sheets that have the characteristic curves.

To be honest, I've have a very cursory understanding of fluid dynamics. The extent of my understanding: I realized my SC836 did not have the correct fans installed as purchased (used via ebay) and the proper ones rated at a higher static pressure dropped temps significantly. OK well maybe that and I know that you can increase the static pressure with square instead of curved elbows in ducting, etc. I could give you the actual definition without googling, but I still don't really "get it." To that point, I don't understand a fan curve any better now than the first time I looked at one. Example: To me, it seems like air flow * should * be a function of static pressure not vice versa and I don't understand why fan curves present static pressure on the y-axis as a dependent variable.

2017-07-07_21-42-18.jpg


Anyway, this all seems moot based on your recent post.
Regardless of whether it appears solved or not, I actually would still like to understand this (and other questions from your earlier response). If you have the inclination, may I take this offline, I would actually genuinely love to start to understand fluid dynamics. Literally, if an outcome related to this genre can be binary, and I pick 0, it is always 1. Whatever I think is/will be the case, is always wrong.
80F ambient, huh? I keep my office at 69 presently to keep my equipment cool.
I got cold last night at home with 78F and bumped it up to 80F (which is comfortable for me). I know that would be far to hot for some. I shiver at the thought of 69F all day long (no pun intended), but I would have to wear a jacket.
Sounds like you are good to go.
Call me a pessimist, but if this is the case it came "too easy" and I'm hesitant to consider case closed. It took so much effort to get to the point where I was keeping the HDDs at 40C (and without a deafening sound pressure level) it isn't even funny, thus my hesitancy here.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
I was making assumptions about this mysterious draw of air under the closet door, knowing what I've learned from you about your system, I can now say with confidence that this has to be natural convection with theheat produced by your electronics and server.I think you should install the fan you purchased and duct it into the space as you have described. Hang yoursheetrock and I agree with @Redcoat , you are good to go! At worst case scenario, you will need to possiblyinstall the second through the wall fan at the bottom of the wall. Stay Cool :cool:
Gotcha. I wouldn't be able to feel the stack effect, would I?

Final thought (I will of course report back with final measurements): While I have the opportunity, do you think I would see any benefit to adding this between the drywall: http://www.roxul.com/products/roxul-safensound/ If I did so, I would hit all three vertical walls and apply as well as possible to the "ceiling" before fitting it (by adhering it in advance or something), leaving the only un-un-soundproofed proofed vertical plane to be the hollow door and the only un-soundproofed horizontal to be the hardwood floor.

Standing at the same distance in the closet behind I can hear the fan noise as well as, if not better, than at the door. Partially driven by the fact that there is a missing 5/8" sheet of rock.

Thank you (and everyone else) for your feedback here. Much appreciated. Have a great weekend.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
While I have the opportunity, do you think I would see any benefit to adding this between the drywall
Sure, the noise will be reduced for sure. There's 12 pieces of 16 x 48 in that bundle you linked to. If the cost
of that bundle is under $47 that puts the cost per piece @ <$4 so I would say go for it:)
Thank you (and everyone else) for your feedback here. Much appreciated. Have a great weekend.
Thank you, it was fun, and please keep us updated.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Thank you, it was fun, and please keep us updated.
You remember the HDD temp feedback, eh? When I report back I will include true ambient (not thermostat readout) at floor and ceiling, CPU temp and HDD temp, and more than just an hours worth.:)

The air handler and server are now conversing via IOT and apparently they have come to the consensus that I should feed the server a real shot of A/C considering their proximity and the fact that I could tap directly into the supply plenum, without the risk of screwing the balance of the whole system by tapping a trunk. I told the server to STFU otherwise I would use it as a counter balance tonight after I have a few beers and decide to rappel off the balcony. I told the handler to STFU otherwise I wouldn't give it a new air filter for 2 years. Then all the lights went off and I swear i heard the server laugh at me when the UPS kicked on.

damn machines.jpg

OK, even I, with my horrible sense of humor don't think that one was funny, which means its time to catch some much needed sleep.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Some further remarks on this topic as I hit the home stretch with the install and in this post I present: (1) a closet diagram (proposed configuration), (2) discuss mounting orientation, (3) dive a bit deeper into mounting bracket selection, (4) question whether two brackets are needed, (5) question which cage screws are appropriate for my rails, and (6) discuss whether a final optimization to the system should be implemented to enhance efficiency.

1. Closet Diagram

I took a moment to diagram out what this will look like when complete (I needed to this for myself as I start installing). Sorry for the large images, and there are probably better ways to do this; however, I've always done this sort of thing in MS Excel setting the cells to squares with values for measurement purposes.
  • Current proposed onfiguration: Kendell Howard 3U V-Rack with SC836 mounted chassis bezel towards the floor, Monoprice 3U Hinged Wallmount Rack for current Cisco switch, patch panel (to be sourced), and 1U for future expansion (perhaps a 10 GigE switch).
  • Rationale for not getting a 6U bracket and mounting everything together: (a) "Isolate" the SC836 from other components that produce heat or impede airflow and (b) I like the idea of a swing out patch panel.
  • More on the brackets in a moment, but it goes without saying that if you are aware of a better solution, please, please let me know.
  • Further, I do need a recommendation for a Cat 5e 20 port patch panel for the Hinged Wallmount Rack.
  • Side View
  • Diagram - Side View.jpg
  • https://ibb.co/g6RfZa
  • Edit: I couldn't zoom into this guy as I would like, so I've appended externally hosted links to those images.
2. Mounting Orientation
  1. Latest discussion with @BigDave suggested mounting with the chassis bottom flush against the wall, chassis front/rear facing left/right was the best approach in my case; however,
  2. as @Redcoat and @MatthewSteinhoff pointed out, the ideal configuration would be chassis bottom flush against the wall, chassis front/rear facing floor/ceiling.
  • I had taken a myopic approach being stuck on that particularly open frame rack and the clearance needed to perform "upside down, vertical mounting," but considering bracket mounting appears the appropriate path, I believe #2 is the way to go.
  • This seems to make sense to orient in this fashion as warm air is already being forcefully expelled upwards and convection is going to carry it further, so maybe this configuration creates less resistance in the flow path (perhaps I'm misapplying @Redcoat 's remarks though)? Further, to @MatthewSteinhoff 's point about attempted emulation of typical rack airflow (front to back), I think this configuration best aligns with that approach (although using intake/exhaust | floor/ceiling, instead of intake/exhaust | left/right).
3. Server Bracket
  • @MatthewSteinhoff was kind enough to note his company's use of Startech 4U or 6U vertical wall mount brackets. As I started to look at sourcing options, it appears as though a 3U Vertical Wall Mount Rack made by Kendell Howard is superior compared to StarTech's offer. I make that assessment just looking at specifications (lesson learned from @Redcoat). It is both heavier (3.47 lbs v 2.08 lbs) and rated for a higher load (150 lbs v 125.3 lbs).
  • While I'm sure either would get the job done, considering they are just about the same price, do you think the Kendeall Howard 3U bracket is the way to go? If you have another option, I'm all ears.
  • Further, I take it there is nothing gained by going with a 4U instead of a 3U? If I need more RUs for future expansion, I will buy another bracket and put it on the side wall. The only thing that I can think of is that if a 4U bracket were used, and the server was mounted on the 3 RUs furthest from the wall, that 1 RU buffer may aid upward air flow. I don't think this would have much, if any, impact though.
4. Chassis Rails - 2 Brackets Needed?
  • Latest discussion with @BigDave suggested use of two brackets, but that suggestion was made with the understanding that a horizontal mounting approach would be used.
  • Beavis.jpg
  • Using a vertical wall mount orientation, I don't believe that two are needed, unless the rails I have can't be locked to prevent the bezel from hitting the floor.
  • I've attached a picture below, but I believe the inner/outer rails I have are Repon W-778-B-0 / W-778-A-0 Rev B Rails.
  • Rails.jpg
  • Unfortunately I can't move the server to check at the moment, but is one of the two options possible: (a) Lock rails so they don't slide, or (b) Reverse them such that the end which would slide is facing up? If so, I believe only 1 bracket is needed, but if not 2 brackets are needed.
5. Chassis Rails - Cage Screws
  • Please realize the below question is coming from someone who has never racked a server in his life. [Internal dialogue: Why do I always take the much more complicated route? No prior experience with FreeNAS ... oh well, let's use ESXi ... Never racked a server before ... oh well, let's make it more difficult and mount vertically. Slaps self on head]
  • Looking at the rails, it looks like each rail end takes 3 screws / nuts, so I need 6 total if 1 bracket is sufficient and 12 total if 2 brackets are required.
  • Rails.jpg
  • Can anyone confirm M6 is the appropriate screw size and that these will work: http://a.co/6vs2XOg?
  • Alternatively, Rackstuds look kinda neat ... and I think they would make mounting / dismounting the server easier, which is already going to be a PITA with its heft by one person.
6. Cooling "System" Review
  • What we have effectively done with the system is take one box (closet) and place it in another box (overhead area), but I wonder if another "box" should be added, i.e the entire condo.
  • While there was no ceiling in place during my "test", lets suggest that fan actually does move 100 CFM, then the entire volume of the closet (59^3 ft) is evacuated nearly twice per minute. That means the capacity of the overhead area (454^3 ft) is fully utilized in less than 5 minutes.
  • Of course, it doesn't really work like that as the warm air is going to cool, become more dense, and gravity is going to pull it back towards the earth between the walls and that air ultimately escapes one way or another, and the cycle repeats.
  • Maybe not in 5 minutes, but given long enough, if that warm air doesn't cool and escape naturally, air flow stagnates and the system loses efficiency right? Where I'm ultimately headed here: Would it make sense to add a passive vent connecting the overhead area to the utility closet with the air handler, better allowing warm air to escape and prevent that loss of efficiency? As an added bonus, when the air handler fan kicks on, I can piggyback off that free active airflow movement, further increasing efficiency.
  • Finally, will the "whiffer" fan impede airflow due to lack of positive pressure in the closet? It is well noted that it is not the optimal fan for the job, but it has been purchased and installed, and isn't returnable so is the worst of my worries that the motor will fail and considering a 1-year warranty I have to pay for a new motor every year?
7. Miscellaneous
  • I think it would be really cool to install thermometers close to the floor and also at the ceiling to monitor ambient. Any cool ideas on "connected" thermometers?
  • Sound absorption: To be very clear, I'm not looking for a way to sound proof, but I did take the extra step of installing Roxul Safe 'n' Sound before reinstalling the drywall. It may not accomplish much, as most sound will probably escape through the hollow core door. Of course, I could install a solid core door, but I can't ever really "sound proof" the room anyway as I'm relying on the gap underneath the door for cool air intake (which would have to be sealed with any serious attempt at sound proofing). In the interest of sound deadening, has anyone ever seen someone attempt to deaden sound at the server, perhaps by entombing it in mass loaded vinyl or similar material. I've read varying remarks on the effectiveness of that, but could find nothing on the internet about an attempt with a server. Most looking to quiet sound would go with a sound proof enclosure, which is not an option in my scenario. Or better yet, understand in advance that servers make a lot of noise.
  • In my design, I have the the server front bezel 12" above the floor with plenty of room to add another bracket and position fans underneath it which would provide additional airflow for the drives (note as designed / proposed, that airflow would come from air in the closet). Would such an idea provide any benefit? Of note, I did add an extra hot j box on the side of the closet in the event that this solution does not work, so I already have a power source in-wall if needed.
  • mo fans mo problems.jpg
Conclusion / Open Items
  • This post ended up being a bit longer than expected, so to recap, a list of open items follows:
    • Closet Diagram: Recommendation for a Cat 5e 20 port patch panel for the Hinged Wallmount Rack?
    • Server Bracket: Validation of selection / 3U appropriate size?
    • Chassis Rails - 2 Brackets Needed?
    • Chassis Rails - Cage Screws: Recommendation on Cage Screw / Nut?
    • Cooling "System" Review: Is system efficiency increased by adding passive exhaust from enclosed overhead area to utility closet?
    • Miscellaneous: Recommendation on connected thermometer? Any experience with sound deadening? Fans beneath bezel?
Thank you in advance for your continued assistance.
 
Last edited:

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
This seems to make sense to orient in this fashion as warm air is already being forcefully expelled upwards and convection is going to carry it further, so maybe this configuration creates less resistance in the flow path (perhaps I'm misapplying @Redcoat 's remarks though)?
No, you have it right. And, if mounted bezel up the momentum of the warm air being expelled downwards would reduce the natural air flow upwards in the closet.
Is system efficiency increased by adding passive exhaust from enclosed overhead area to utility closet?
Not sure - but what it seems likely to do is give a lower resistance path for intake of air that bypasses the equipment in the closet...
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Not sure - but what it seems likely to do is give a lower resistance path for intake of air that bypasses the equipment in the closet...
Thanks for your prompt reply!
  • Can you explain that point please, I'm unclear as to what you mean (I'm sure it is me, not you)?
  • Curious (if you have time): If that enclosed space was a hermetically sealed chamber, what does happen after it has been pumped full of warm air? Perhaps, that isn't so hypothetical after I think about it, as at some point it does become full, and considering pressure isn't going to build and make it explode, "release" of cool air would have to equal "input" of warm air for the system to have a positive effect, right?
 
Joined
Feb 2, 2016
Messages
574
I'm not sure why you're committed to mounting the server vertically instead of horizontally. That makes everything more difficult. You'll have to lay on the floor to see the front panel lights. Pop a drive latch and there's a good chance it'll hit the floor. Dirt and dust you draw into the closet (because of the ventilation fan and convection) is primed to get sucked into the server because the intake is close to the floor.

Your server is 26 inches deep and your closet is 41 inches wide. Because you're using an open-side rack mount, there is no issue here. You're not sliding it on the rails. You're setting it into the rack and pulling it out straight towards the closet opening. For internal access, you'll just pop the top - you don't even have to remove it from the rack. There will be plenty of clearance (15 inches) to add and remove drives. About the only time you'd ever need to remove it from the rack is if a power supply needed a replacement because you'll cheat it toward one side to give yourself plenty of drive clearance. Put the exhaust fan in the ceiling above the exhaust side of the server.

If you do insist on mounting it vertically, mount it higher. It will produce the bulk of the heat in the closet. Put it at the top so it is nearest the exhaust port. That way, if you do add equipment, it'll be below the heater. Plus, mounted higher, it won't suck in so much dirt/dust from the floor.

The Supermicro 836E16 weighs 52 pounds without drives. Sixteen drives adds 26 pounds. That puts you at 78 pounds. Either of the brackets mentioned above will support more than 100 pounds and the Startech comes with all the hardware you need to mount it to the wall and the server to the rack. So, get that one instead of the Kendell. I always recommend buying more U than you need: the cost difference between the 3U and 4U, 5U or 6U Startech is trivial and you never know when you're going to add more hardware.

The rated operating temperature for the server is 95°F while the drives are rated for 140°F. If you ever see those temperatures, your house is on fire. As long as you have good airflow through the server, you'll be okay. Just as a point of reference, the intake temperature on our 12-bay Supermicro SC826E16 is 68°F and the output temperature is 84°F. That's a 16°F increase front to back. We could be 10°F worse off and still be within spec. (Though, I'd sleep less well at night. If I had my way, everything - servers and me - would be 65°F year round.)

You keep the ambient temperature at your house between 78 and 80°F? The biggest obstacle to your success isn't the closet or the equipment, it is that you have your house at 80°F.

Cheers,
Matt
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Matt,

First and foremost, thank you for your continued assistance.

I'm not sure why you're committed to mounting the server vertically instead of horizontally. That makes everything more difficult. You'll have to lay on the floor to see the front panel lights. Pop a drive latch and there's a good chance it'll hit the floor. Dirt and dust you draw into the closet (because of the ventilation fan and convection) is primed to get sucked into the server because the intake is close to the floor.
  • I'm absolutely not committed to mounting it vertically, but I am committed to attempting to mount it in the most optimal manner, given a sub-optimal situation.
  • To be honest, when I created this thread I really did want to mount vertically, butt down, but you and other's wisdom made me realize that wasn't the best play. Although, that was my "going in position, just to confirm, you do realize that the diagram I created uses a vertical, butt up configuration, right?
  • I must have misinterpreted your prior remarks as I thought that you were advocating either (a) horizontal wall mount, or (b) vertical wall mount, butt up.
  • Generally I thought that vertical, butt up was the "consensus" approach but recent dialogue with @BigDave centered on the horizontal approach. In my head, perhaps not supported by the evolution of this thread: (1) Vertical, butt down was proposed / discredited, (2) Horizontal was proposed and analyzed, and (3) There wasn't much dialogue on vertical, butt up. My point here: My post today changed the orientation from recent discussion of horizontal as I thought that was skipped over (to an extent).
  • I'd like to understand why you think horizontal mounting is superior compared to vertical, face down. It seems like this configuration aligns with the laws of physics better than a horizontal mount. As you point out, there is downside to that approach, but how often are you looking at the system indicator lights or swapping drives? IMPI shows you much more than 6 LEDs.[1] Further, I'm not going to suggest that no particulate matter will make its way inside the server, but that concern is somewhat mitigated considering I have installed the optional front bezel and I really like the fact that it has an air filter inside: https://forums.servethehome.com/index.php?threads/neos-freenas-build.5907/page-2#post-52567.
[1] Trust me I do understand the allure of LED lights, though. To be candid, I couldn't look away at first when I had 16 blinking blue LEDs. That love affair was rekindled and I spent a week using sas2ircu <controller_#> locate <Enclosure:Bay> ON to turn on the red LEDs. OK, I'm kidding, but it was kind of neat (for a few seconds) to see red instead of blue.

Your server is 26 inches deep and your closet is 41 inches wide. Because you're using an open-side rack mount, there is no issue here. You're not sliding it on the rails. You're setting it into the rack and pulling it out straight towards the closet opening. For internal access, you'll just pop the top - you don't even have to remove it from the rack. There will be plenty of clearance (15 inches) to add and remove drives. About the only time you'd ever need to remove it from the rack is if a power supply needed a replacement because you'll cheat it toward one side to give yourself plenty of drive clearance. Put the exhaust fan in the ceiling above the exhaust side of the server.
  • I'm really not looking forward to mounting it (vertically or horizontally) as it is a PITA to move. It isn't so much that the weight is excessive, it is just unwieldy.
  • If you care to come help, I'll pay for your plane ticket. Seriously ... the fiance works for Delta.
  • Your post has thrown me for a bit of a loop though, as all circumstances considered, I thought vertical, butt up is the optimal mounting orientation. You have given me a bit to think about. You don't happen to know the answer to my questions regarding the chassis rails, do you? Not the hardware, but if they can be "locked."
If you do insist on mounting it vertically, mount it higher. It will produce the bulk of the heat in the closet. Put it at the top so it is nearest the exhaust port. That way, if you do add equipment, it'll be below the heater. Plus, mounted higher, it won't suck in so much dirt/dust from the floor.
  • Only for the sake of understanding, you have probably come to understand I like to research and ask questions, is this statement made assuming vertical butt down or butt up mounting?
  • If butt up, I'd disagree and think you would want the intake to pull the densest, cool air at the floor.
The Supermicro 836E16 weighs 52 pounds without drives. Sixteen drives adds 26 pounds. That puts you at 78 pounds. Either of the brackets mentioned above will support more than 100 pounds and the Startech comes with all the hardware you need to mount it to the wall and the server to the rack. So, get that one instead of the Kendell. I always recommend buying more U than you need: the cost difference between the 3U and 4U, 5U or 6U Startech is trivial and you never know when you're going to add more hardware.
  • Thanks for addressing my question on this.
  • Take no offense to my alternate suggestion, as I very much appreciate you making a recommendation. As well as your thoroughness in your replies, as you might guess I try to take a similar approach. Only because I work in Finance, I had to check your figures: 16 x 715 grams = 11,440 grams / 453.592 = 25.2 lbs. You are high by 0.8 lbs. Just giving you crap, I have a dry sense of humor. :)
  • As I noted, and as you confirmed, either work, but I will take your advice. When I started to source one, I was surprised to see alternatives that appeared much sturdier. As a personal quirk I tend to over-engineer.
  • Valid points on not needing to separately source hardware and on buying more RUs than needed.
The rated operating temperature for the server is 95°F while the drives are rated for 140°F. If you ever see those temperatures, your house is on fire. As long as you have good airflow through the server, you'll be okay. Just as a point of reference, the intake temperature on our 12-bay Supermicro SC826E16 is 68°F and the output temperature is 84°F. That's a 16°F increase front to back. We could be 10°F worse off and still be within spec. (Though, I'd sleep less well at night. If I had my way, everything - servers and me - would be 65°F year round.)
  • Thanks for the link. I've seen that document before, but wasn't aware HGST made a 10TB variant now.
  • Everyone seems to have a differing opinion on optimal HDD temperatures, but the consensus on the FreeNAS forum seems to be 40°C max.
  • To your point, I was aware that the recommended operating temperature max is 60°C, but I think you reference that document just to make a point as you later note you wouldn't sleep well at 78°F, so I imagine you wouldn't sleep at all at 140°F (and hopefully not because you burned in a fire).
  • I would love to see the failure rates for a statistically significant sample of HGST Deskstar NAS drives run at 60°C for 3 years straight (warranty period).
  • If I let my drives get anywhere close to that, I would be looking for new ones the first day out of warranty.
You keep the ambient temperature at your house between 78 and 80°F? The biggest obstacle to your success isn't the closet or the equipment, it is that you have your house at 80°F.
  • I know this sounds bizarre but anything lower than that is too cold for lounging around at home comfortably in shorts and a t-shirt. I was a bit chilly last week so I ratcheted it up to 80°F temporarily. On the flip side, I never run the heat in the winter.
  • To ensure I don't come across the wrong way, let me preface my remaining remarks by noting how thankful I am for your thoughts and guidance. I don't provide the below data in the interest of starting a debate, but simply to suggest that there isn't an obstacle to success. I spent a good bit of time optimizing my system getting here and I think the results are quite solid considering 16 7200 RPM drives throw off a bit of heat and ambient is 78°F.
  • During the period of time that data is presented for, the server has been working away downloading data from ACD and replicating offsite and my 100/70 connection has been 100% utilized during that duration with vSphere CPU utilization around 40%
  • Over the last 24 hours, the average HDD temperature was 37.2°C. Over that same period of time, I held 40°C or lower HDD temps (for all drives) 99.4% of the time.
  • Perhaps more importantly, the "Full Speed" fan mode was only engaged for 8 minutes (of 1440). Also, only for 8 minutes during that 24 hour period was the max temperature greater than 40°C.
  • I consider Optimal, Standard, and Heavy I/O modes to be tolerable, whereas the Full Speed mode is not. Provided you have the fans specified for your SC826E16-R1200LPB installed, you know exactly what three of the FAN-0126L4 fans sound like at 100%. Completely unrelated to the topic of discussion, I'm curious if you think what fan modes are tolerable and what are not.
  • Nice case choice by the way. I love my Fractal Node 304.
temps-1min-drives.png


Fan Script Log Pivot.jpg


Thanks again!
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
I'd like to understand why you think horizontal mounting is superior compared to vertical, face down. It seems like this configuration aligns with the laws of physics better than a horizontal mount. As you point out, there is downside to that approach, but how often are you looking at the system indicator lights or swapping drives?
Not having tested out the difference between horizontal mounting and "butt up" mounting, I just can't see the
drive temp difference between the two methods to be more than a couple of degrees at most! I understand the desire
to have the drives run as cool as possible and IF the "butt up" method gains 4-5 degrees, go for it! If the cooler
temps are not realized, I would highly recommend a horizontal mounting.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
"butt up" mounting

I got tired of saying:
1. "Vertically wall mounted in an orientation such that the server chassis face is facing the floor (server chassis rear is facing the ceiling)"
OR
2. "Vertically wall mounted in an orientation such that the server chassis face is facing the ceiling (server chassis rear is facing the floor)",
SO
"Butt up" and "Butt down," respectively, it is. I think we can all agree I use way too many words as is. :):):)

I understand the desire to have the drives run as cool as possible and IF the "butt up" method gains 4-5 degrees, go for it!
  • I appreciate that. Thanks for understanding my "passion" here as well as all of your help. Murphy's law plagued my first foray into utilizing enterprise-grade servers and while I learned a heck of a lot, it was extremely frustrating at times and I * thought * about giving up.
  • I'm a planner by nature and have to understand activities I put significant effort into, but got tired of waiting and pulled the trigger. If I were to do it again, I would manage this project just the same as one at work and notably spend much more time researching. But again, admittedly, I do have a tendency to over-analyze. I think I'm being a little too trigger shy due all the issues I encountered at first.
  • Hopefully my presentation of HDD temps did not come off the wrong way (my writing style is often perceived with an unintended, non-existent tone), but I'm proud of the temps presented in the post immediately prior to yours (considering the environmental variables). It took a lot of time, research, and effort to: (1) determine what was causing my high HDD temps, (2) deploy a hardware fix, (3) empirically analyze the improvement and report back to the community, and (4) and finally deploy a script that manages those temps in an "automated" way.
  • Initially, I couldn't break below 40°C with the fans at 100% and endured them screaming in my ears for weeks and then next thing you know, I'm producing (hopefully) insightful 1,777 word count posts with nice visuals comparing stock heatsinks and active CPU coolers, with the results surprising even some of the most knowledgeable members of the community: https://forums.freenas.org/index.ph...permicro-snk-p0048ap4-passive-heatsink.55799/
  • My ultimate point with this monologue: I just don't want all that effort to be all for naught.
I think the path forward is as simple as:
  1. Use two sheets of plywood. One that I can drill into as many ways as I want, whether it be "butt up," "butt down," "sideways", etc. Once the optimal position is empirically supported, simply remove the top sheet and mount permanently on the bottom sheet which I haven't drilled into 1,000 times. I too suspect that we aren't looking at a huge difference here and likely using a more appropriate exhaust fan would have a larger impact. But there is always the next build (or home) to get it right, eh?
  2. Also, I would like to hear back from @MatthewSteinhoff as clearly he has admirable and significant experience with this.
  3. You wouldn't happen to know the answer to how to use the rails with a butt up config would you (I'm still at work and curious)?
Thanks again and I hope you have a great evening! :)
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
There's a lot of good info here, but I'll add my 2 cents. I did a very similar thing... I've got a 96ft3 closet (3x4x8) with one rack within. The rack has:
Cyberpower 3KVa online UPS
Supermicro 36-bay 4U chassis (FreeNAS)
3x Supermicro 2U chassis (all dual E5-2670s, vSphere nodes)
48-port PoE switch
Patch panel
Tivo
PDU
A few other things

Total load is about 11A @ 115VAC. I tried using an exhaust fan (Panasonic WhisperQuiet @ 150CFM), no dice. Added an intake fan, same problem. Right now, the door stays open... in the near future (once I install the infrastructure) I'll be adding a Mitsubishi ductless mini-split AC unit. It's the only way to get reasonable temps.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Total load is about 11A @ 115VAC. I tried using an exhaust fan (Panasonic WhisperQuiet @ 150CFM), no dice. Added an intake fan, same problem. Right now, the door stays open... in the near future (once I install the infrastructure) I'll be adding a Mitsubishi ductless mini-split AC unit. It's the only way to get reasonable temps.
  • Thanks for weighing in here. Man that stinks your solution didn't work.
  • I think you are right about that amount of gear needing real infra, A/C, and a dedicated circuit. Perhaps that fan didn't work because it didn't create the pressure differential needed and is a "whiffer" as @Redcoat calls them?
I'm curious:
  • Do you have a before / after temp measurement for the any of those servers? Did the fans have any impact?
  • Where were you exhausting to?
  • What kind of intake fan were you using?
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
There are a couple of earlier-raised points to which I didn't yet respond:

I don't understand a fan curve any better now than the first time I looked at one. Example: To me, it seems like air flow * should * be a function of static pressure not vice versa and I don't understand why fan curves present static pressure on the y-axis as a dependent variable.

I believe that you may be thinking of the manner in which a flow system would respond, rather than about the behavior of the fan.

Air flow and static pressure loss in a flow system are related - the more the flow the greater the pressure loss will be. In a system where the final pressure is zero (= to atmospheric pressure) the pressure loss due to flow will be numerically equivalent to the maximum static pressure in the system.

The fan characteristic curve of static pressure versus flow is a plot of the operating points of the fan assembly - in this case a single curve for a constant speed unit. What's shown in the data sheet is only the portion of typical interest to the user/specifier - the "full curve" would have looked something like the "SP" line below (they cut the top off the one in the literature).

upload_2017-7-12_13-17-20.png


The relative shapes of the various curves are inherently a function of the specific fan design. But the indicated fan, run at the speed for which the curve is drawn, will run at that point on the curve that the system resistance dictates. In the case of the diagram here, with little to no system resistance the flow will be around 300,000 CFM but no pressure will be developed. As resistance is increased in the system the flow will drop and the pressure differential created by the fan will increase to about 105" and then drop back to about 100' at zero flow - at which point the fan will be dissipating energy heating up the air in it as the impeller rotates.

For more info, "best" ref I found in a quick Google search is here.

Next was
Redcoat said:
Not sure - but what it seems likely to do is give a lower resistance path for intake of air that bypasses the equipment in the closet...
Thanks for your prompt reply!
  • Can you explain that point please, I'm unclear as to what you mean (I'm sure it is me, not you)?
  • Curious (if you have time): If that enclosed space was a hermetically sealed chamber, what does happen after it has been pumped full of warm air? Perhaps, that isn't so hypothetical after I think about it, as at some point it does become full, and considering pressure isn't going to build and make it explode, "release" of cool air would have to equal "input" of warm air for the system to have a positive effect, right?
I understood you to say that you were proposing to make a flow path between the top of your server closet and your air handler "room" and expected that additional space's volume to accept some of the warm air from the server closet and dissipate its heat. If that was your proposal, my expectation was that it was likely that the air handler room space would actually become a new "fresh air" inlet to your exhausted space and the resulting inflow of air to the top of the closet would reduce air flow past the server.

But based on your "Curious" bullet (which I can't put into context) I'm guessing I didn't correctly interpret the scenario ...
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Thank you for providing additional detail here. Between your reply and that reference link, I think I'm starting to "get it" but I really need to study this more when I'm not multitasking and able able to dedicate myself completely to it. It isn't difficult, it is all based on a simple set of related equations, I'm just having trouble wrapping my head around it and applying it.
I understood you to say that you were proposing to make a flow path between the top of your server closet and your air handler "room" and expected that additional space's volume to accept some of the warm air from the server closet and dissipate its heat. If that was your proposal, my expectation was that it was likely that the air handler room space would actually become a new "fresh air" inlet to your exhausted space and the resulting inflow of air to the top of the closet would reduce air flow past the server.
  • I 100% agree - your understanding of my proposal would not be effective, but my proposal was misunderstood. Sorry for not being more precise.
  • Let me attempt to clear this up, but I'm not suggesting anything that was previously discussed be changed. Rather, only adding an additional path for warm air to escape at the end of the system.
  • Server closet would still exhaust to overhead area by whiffer fan. Server closet is not connected to utility closet.
  • I was only asking whether that overhead area eventually becomes "saturated" with heat, the system loses efficiency, and if adding a passive exhaust from the overhead area to the utility closet would be beneficial. That passive airflow from the overhead area to utility closet would ultimately be disposed of by evacuation from the air handler or move back into the living area becoming part of a loop.
To recap with visual aid …

There are 4 contained spaces here:
  • Living Area
  • Server Closet (without fan is not connected to overhead area)
  • Overhead Area (without a passive vent is not connected to utility closet)
  • Utility Closet (air handler)
Flow Path *could* consist of:
  1. Living Area to Server Closet (intake = under door) #no change
  2. Server Closet to Overhead Area (exhaust = whiffer) #no change
  3. Overhead Area to Utility Closet (passive exhaust) #potential addition
Flow.jpg
Hopefully I did a better job explaining, but let me know if not. Further, as I didn't appropriately relay the proposal previously, I'm curious to your thoughts on this.
But based on your "Curious" bullet (which I can't put into context) I'm guessing I didn't correctly interpret the scenario ...
To explain the context of the "Curious" bullet, let me rehash the conversation (my brain works best with lists):
  • @svtkobra7: Is system efficiency increased by adding passive exhaust from enclosed overhead area to utility closet?
  • @Redcoat: Not sure - but what it seems likely to do is give a lower resistance path for intake of air that bypasses the equipment in the closet...
  • @svtkobra7: [I'm] Curious (if you have time): If that enclosed space was a hermetically sealed chamber, what does happen after it has been pumped full of warm air? Perhaps, that isn't so hypothetical after I think about it, as at some point it does become full, and considering pressure isn't going to build and make it explode, "release" of cool air would have to equal "input" of warm air for the system to have a positive effect, right?
Perhaps with proper explanation of my proposed addition (Overhead Area to Utility Closet flow path), this doesn't require clarification. I was attempting to understand * conceptually * what occurs in a system where you have:
  • Container 1 (representative of Server Closet)
  • Warm Air is exhausted into
  • Container 2 (representative of Overhead Area)
  • Where Container 2 is "sealed" and pressure isn't significant enough to make it explode.
I presented it as such as you noted you weren't sure if system efficiency was increased (but my understanding is that the proposal wasn't properly conveyed, making the "Curious" bullet moot).
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
Thank you for providing additional detail here. Between your reply and that reference link, I think I'm starting to "get it" but I really need to study this more when I'm not multitasking and able able to dedicate myself completely to it. It isn't difficult, it is all based on a simple set of related equations, I'm just having trouble wrapping my head around it and applying it.

  • I 100% agree - your understanding of my proposal would not be effective, but my proposal was misunderstood. Sorry for not being more precise.
  • Let me attempt to clear this up, but I'm not suggesting anything that was previously discussed be changed. Rather, only adding an additional path for warm air to escape at the end of the system.
  • Server closet would still exhaust to overhead area by whiffer fan. Server closet is not connected to utility closet.
  • I was only asking whether that overhead area eventually becomes "saturated" with heat, the system loses efficiency, and if adding a passive exhaust from the overhead area to the utility closet would be beneficial. That passive airflow from the overhead area to utility closet would ultimately be disposed of by evacuation from the air handler or move back into the living area becoming part of a loop.
To recap with visual aid …

There are 4 contained spaces here:
  • Living Area
  • Server Closet (without fan is not connected to overhead area)
  • Overhead Area (without a passive vent is not connected to utility closet)
  • Utility Closet (air handler)
Flow Path *could* consist of:
  1. Living Area to Server Closet (intake = under door) #no change
  2. Server Closet to Overhead Area (exhaust = whiffer) #no change
  3. Overhead Area to Utility Closet (passive exhaust) #potential addition
View attachment 19432
Hopefully I did a better job explaining, but let me know if not. Further, as I didn't appropriately relay the proposal previously, I'm curious to your thoughts on this.

To explain the context of the "Curious" bullet, let me rehash the conversation (my brain works best with lists):
  • @svtkobra7: Is system efficiency increased by adding passive exhaust from enclosed overhead area to utility closet?
  • @Redcoat: Not sure - but what it seems likely to do is give a lower resistance path for intake of air that bypasses the equipment in the closet...
  • @svtkobra7: [I'm] Curious (if you have time): If that enclosed space was a hermetically sealed chamber, what does happen after it has been pumped full of warm air? Perhaps, that isn't so hypothetical after I think about it, as at some point it does become full, and considering pressure isn't going to build and make it explode, "release" of cool air would have to equal "input" of warm air for the system to have a positive effect, right?
Perhaps with proper explanation of my proposed addition (Overhead Area to Utility Closet flow path), this doesn't require clarification. I was attempting to understand * conceptually * what occurs in a system where you have:
  • Container 1 (representative of Server Closet)
  • Warm Air is exhausted into
  • Container 2 (representative of Overhead Area)
  • Where Container 2 is "sealed" and pressure isn't significant enough to make it explode.
I presented it as such as you noted you weren't sure if system efficiency was increased (but my understanding is that the proposal wasn't properly conveyed, making the "Curious" bullet moot).

Ah... so there is currently no significant outlet from the volume into which the bathroom exhaust fan discharges? If so, that's not good. From the curves discussion and assuming that the curve for your installed fan won't look much different than the L200L in form, imagine what sort of flow you might have to cool your closet. You need an outlet in that space. Cut a hole from it into the air handler room. Your fan is not a positive displacement device - it won't pump anything up. 1" water column is ~1/27th of 1 psig.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
  • Thanks for weighing in here. Man that stinks your solution didn't work.
  • I think you are right about that amount of gear needing real infra, A/C, and a dedicated circuit. Perhaps that fan didn't work because it didn't create the pressure differential needed and is a "whiffer" as @Redcoat calls them?
I'm curious:
  • Do you have a before / after temp measurement for the any of those servers? Did the fans have any impact?
  • Where were you exhausting to?
  • What kind of intake fan were you using?
Yeah... oh well, cost of doing business. That closet has two 115/30 circuits in it (one's spare, one's feeding the UPS).

I didn't do any significant temp measurements. The big problem with insufficient cooling in such a small space is it dramatically enhances the progress of thermal runaway. If I go close the door and seal the room right now, ambient air temperature will increase 20 degrees in about 60-90 seconds. Here's some good rough number calculations, with the knowledge that these assume standard temperature and pressure, become increasingly inaccurate as the temperatures rise, and assume a perfectly sealed environment (no accounting for radiant heat loss through the walls):
10A * 115V = 1150W = 3,924BTU/hr.
0.24BTU will raise 1lb. air 1 degree F.
0.075lb./cuft air. So 96cuft = 7.2lb.
So, 1.728BTU will raise our ambient temperature 1F.
With the above, we know we're generating 65.4BTU/min. That equates to 37.8F/min. At this rate, even with HVAC, a 1-2min power outage can be a Big Problem. Even if you half that number, at an ambient 72F, in 5 minutes, you're at 166.5F. Keep in mind that wood autoignites between 160-270F depending on species, and sheetrock 450-475F. You've got a fire on your hands pretty quickly.

You can also calculate air flow as: BTU = dT * CFM * 1.08 (the constant assumes 70F at sea level, close enough). If you're willing to accept a 20 degree rise, you need 188CFM. But that assumes perfect airflow, which I think was my problem.

I took an old door, cut a 10x6 vent through it, filled the rest of the door with expanding foam, then mounted a 20" box fan to the inside. The fan was sealed to the door so there was no air reingestion from the hot side. They say 2500CFM, but I don't buy that. That's at zero static pressure. But I would've expected more than 200CFM input.

The exhaust was a Panasonic 150CFM WhisperQuiet fan, ceiling mounted, with exhaust outside via a 6" flexible duct with no kinks. It should have been able to deliver 150CFM through the ductwork.

Yes, this meant that conditioned air was being exhausted to atmosphere. Not the ideal scenario, but worthy of a test. And, as long as the hot air being expelled was warmer than the make-up air from the outside world, it should result in a net savings (but this assumes the ventilation is carefully balanced... if too much ventilation existed and there was minimal temperature rise, which would be best, then energy would be lost).

With all of this in place, I watched temperatures climb from ~72F to ~90F in about 20 minutes before I aborted the test. 90F was too hot for everything and the temperature wasn't showing signs of stabilizing, so I may have still been underventilated and looking at the possibility of thermal runaway.

Now, what I expect was happening (but never proved through the use of colored smoke/etc.) was that air was flowing from the inlet directly to the exhaust, largely bypassing the pocket of hot air being created behind the servers. I was hoping the box fan would introduce air with enough velocity to push this air out, but apparently not. The path of least resistance would be straight out the exhaust.

My potential solution to this (negating the issue of the loss of conditioned air to atmosphere) would be to seal the system in some sort of cabinet. Ensure that the ONLY air in is from the conditioned supply (through the front) and the ONLY air out is through the exhaust vent (through the rear). Assuming you can balance the intake and exhaust fans (variable speed fans would be best here) there should be no static pressure issues to deal with (intake should "push" with the same pressure the exhaust "pulls").

But, with all that said, I'm just going to drop a 1-ton low-ambient ductless mini-split AC in and call it a day. Eventually, I'll add a sufficiently large diesel generator to run the whole house.

Hopefully my ramblings will benefit you somewhat :)
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Ah... so there is currently no significant outlet from the volume into which the bathroom exhaust fan discharges?
  • No, not significant.
  • Of course, that Enclosured Area is by no means air tight, i.e. the warm air will cool and descend between the studs, escaping at outlets, wall switches, light fixtures, and surely other crevices I haven't thought of (as well as other ways).
Cut a hole from it into the air handler room.
  • Create a passive vent to the utility closet from the Server Closet or the Enclosed Area?
  • The way, I read your post is to cut a vent from the Server Closet to the Utility Closet and that the fan is worthless in extracting air and the only air "extracted" would be that which convection drives.
 
Status
Not open for further replies.
Top