60TB+ New Build Need Hardware check

Status
Not open for further replies.

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
My Primary NAS is a similar build. One of its most important tasks is as a Time Machine target for a bunch of macs.

Currently I'm working on a 10gbe upgrade for it, as it means being able to service up to 10 clients simultaneously at full gbit speed each.

Time machine writes are temporarily cached in ram before being flushed to disk. No need for ssd cache.

Would recommend either 6-way or 8-way RaidZ2 vdevs in a 24 bay backup device.


Did you get the 10gbe working?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I know these are extremely loud. I've read about changing out the fans or putting a fan controller on the stock fans. I'll read more into that. I originally wanted to keep this in the office that has AC but the noise would be unbearable. I'm trying to make a judgement call on what would be better. I can have quite fans that might compromise integrity of the machine (if I don't research the hell out of it first) in a 24/7 AC's office or put it in a well ventilated server closet that might reach 110 degrees.
No, 110 degrees is way too hot. You have to keep the input air on the server cool enough that it can cool the internal components. After adjusting the fans in my server, it is quiet enough that I keep it in my office about 8 feet from my desk and it is perfectly tolerable.
I keep my room between 74 and 77 degrees and the server stays plenty cool. The one thing I had to change (other than the fans) was the CPU heatsinks. The original ones rely on high speed airflow from the system fans, so once you put slower fans in that are perfectly adequat to keep the hard drives cool, the CPU will overheat. I had to put heatsinks on the CPUs that had integrated fans and now it works great.

The one I use looks kind of like this, but I have a different socket. This is the one I think you will need:
https://www.newegg.com/Product/Prod..._2011_v3_cooler_server-_-35-184-040-_-Product

I used these with the stock fans in one of my servers to get very good noise reduction by just slowing the fans down.
https://www.ebay.com/itm/5Pcs-Compu...d-Noise-Transform-Resistor-Cable/311838087910

I just stood mine on their sides like they are towers. You might even be able to get a conversion kit because Supermicro does sell a 'tower' server that is nothing more than a rack server with feet screwed on one side and a decorative chunk of plastic on the top.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
or put it in a well ventilated server closet that might reach 110 degrees.
Please let us know what brand of drives you intend to buy. I need to buy some stock in that company - I'll get rich off the number of drives you'll buy.

My home server closet with dedicated A/C stays at 70F. The drives all run around 40C, or 104F. That's all Supermicro gear with their fans, running at whatever speed the system decides is proper. So, the drives are 34F above ambient. In your world, that would put you at 144F or 62C.

Refer to page 6 of this document, and watch the failure rate skyrocket with drive temps north of 50C:
https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Messing with the fan speeds of a chassis that is designed for high static pressure fans is dangerous.

https://forums.freenas.org/index.ph...-hds-recommendations.24453/page-4#post-255614

https://forums.freenas.org/index.ph...n-1st-freenas-build.28814/page-11#post-300857

https://forums.freenas.org/index.php?threads/replacement-of-fans-fan-0126l4.43712/#post-289354

etc

The reason the fans are noisy is because they're pouring a lot of energy into making a low pressure zone that causes air to seep through the minuscule gaps around your hard drives to cool them. The fix isn't to sabotage the fans, because that only sabotages your cooling and slows or stops the air from flowing around some of your drives. The fix is to do something rational to cool. You can use a larger (probably non-rackmount) chassis with larger 120mm fans and drives mounted in 5.25" HH bays, as an example. As an alternative, you can also get a 24 bay rackmount chassis and put 12 drives in it in a staggered pattern that avoids having two drives stacked on top of each other, making sure to include the Supermicro slot blanks, which reduces the heat load and offers larger spaces for cooling, so you can probably mess with the fans on that somewhat and still be safe. This works pretty well but isn't as good as a properly-built system.
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
Please let us know what brand of drives you intend to buy. I need to buy some stock in that company - I'll get rich off the number of drives you'll buy.

My home server closet with dedicated A/C stays at 70F. The drives all run around 40C, or 104F. That's all Supermicro gear with their fans, running at whatever speed the system decides is proper. So, the drives are 34F above ambient. In your world, that would put you at 144F or 62C.

Refer to page 6 of this document, and watch the failure rate skyrocket with drive temps north of 50C:
https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf
Yeah I was being facetious but probably to tired to try and express sarcasm in text. I'll most likely have to get some sort of AC in the closet. :(
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
Messing with the fan speeds of a chassis that is designed for high static pressure fans is dangerous.

https://forums.freenas.org/index.ph...-hds-recommendations.24453/page-4#post-255614

https://forums.freenas.org/index.ph...n-1st-freenas-build.28814/page-11#post-300857

https://forums.freenas.org/index.php?threads/replacement-of-fans-fan-0126l4.43712/#post-289354

etc

The reason the fans are noisy is because they're pouring a lot of energy into making a low pressure zone that causes air to seep through the minuscule gaps around your hard drives to cool them. The fix isn't to sabotage the fans, because that only sabotages your cooling and slows or stops the air from flowing around some of your drives. The fix is to do something rational to cool. You can use a larger (probably non-rackmount) chassis with larger 120mm fans and drives mounted in 5.25" HH bays, as an example. As an alternative, you can also get a 24 bay rackmount chassis and put 12 drives in it in a staggered pattern that avoids having two drives stacked on top of each other, making sure to include the Supermicro slot blanks, which reduces the heat load and offers larger spaces for cooling, so you can probably mess with the fans on that somewhat and still be safe. This works pretty well but isn't as good as a properly-built system.

Thanks for the "work around" I do want to "do this right" but I thought there might be some "tricks" that don't compromise on stability or data safety that I could do and having more space for air seems legit but I again want to do this right so AC it is for the closet.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeah, there aren't really many happy answers. The best answers are expensive and that really sucks. If you build a new system, it isn't that hard to spec gear that will run quiet and cool, but that generally means a large roomy chassis, which probably means a newish mainboard/CPU/RAM, which all translates to dollars.

Basically you want your disks to run cool. Ideally mid-20's to mid-30's. That's probably overly paranoid, but hey... it works.

# for i in 1 2 3 4; do
> smartctl -a /dev/ada${i} | egrep '^ 9|^194'
> done
9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 27381
194 Temperature_Celsius 0x0022 032 041 000 Old_age Always - 32 (0 22 0 0 0)
9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 27342
194 Temperature_Celsius 0x0022 033 041 000 Old_age Always - 33 (0 23 0 0 0)
9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 27317
194 Temperature_Celsius 0x0022 034 041 000 Old_age Always - 34 (0 22 0 0 0)
9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 27931
194 Temperature_Celsius 0x0022 035 041 000 Old_age Always - 35 (0 22 0 0 0)

That's a 1U chassis with some kind-of-noisy fans in an area that's kept around 68-70'F.

In a 4U SC846, 12 disks evenly spaced:

etctl -a /dev/da${i} | grep '^194'; do
194 Temperature_Celsius 0x0022 023 040 000 Old_age Always - 23 (0 17 0 0 0)
194 Temperature_Celsius 0x0022 026 049 000 Old_age Always - 26 (0 18 0 0 0)
194 Temperature_Celsius 0x0022 024 040 000 Old_age Always - 24 (0 18 0 0 0)
194 Temperature_Celsius 0x0022 028 040 000 Old_age Always - 28 (0 19 0 0 0)
194 Temperature_Celsius 0x0022 027 040 000 Old_age Always - 27 (0 19 0 0 0)
194 Temperature_Celsius 0x0022 025 040 000 Old_age Always - 25 (0 18 0 0 0)
194 Temperature_Celsius 0x0022 027 040 000 Old_age Always - 27 (0 20 0 0 0)
194 Temperature_Celsius 0x0022 027 050 000 Old_age Always - 27 (0 19 0 0 0)
194 Temperature_Celsius 0x0022 026 049 000 Old_age Always - 26 (0 18 0 0 0)
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29 (Min/Max 21/42)
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 34 (Min/Max 23/45)

That last drive there's a bit of an outlier. It's a 7200RPM drive and it consistently runs hotter than the other one.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I have already said how I slow down the speed of the original fans in my system.
I have drive temperatures in the low thirty range.


Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Hi, I ended up getting that one just for myself. Should I spend the time to crossflash the LSI 9210-8i or should I just get the LSI 9211-8i and call it a day?
The physical layout of the two cards is different, where the connectors are, but they are virtually identical in the ways that matter. If you flash the IT mode firmware on the one you got, it should be fine.

PS. The biggest difference is that the 9210 was only sold as an OEM part.
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9210-8i#overview
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
The physical layout of the two cards is different, where the connectors are, but they are virtually identical in the ways that matter. If you flash the IT mode firmware on the one you got, it should be fine.

PS. The biggest difference is that the 9210 was only sold as an OEM part.
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9210-8i#overview

I just cannot BELIEVE HOW GREAT YOU ARE! (*AND EVERYONE ELSE) on this forum. I seriously have never been so taken aback by the sheer wealth of knowledge and quickness to answer questions. I'll be building more of these and I can't wait until I have more knowledge and experience so that I too can lend a hand or suggestion when needed. I do that on all my other forums from networking to advanced use of Macs so this feels like I'm getting paid back big time. Thx again.
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
A switch like this would probably do great for you: https://www.neweggbusiness.com/product/product.aspx?item=9b-33-150-165
You might be able to fine one a little less expensive, but this one has four SFP+ ports and all you need to do is put a Chelsio SFP+ adapter in the NAS. Then all the 1GB clients could access the NAS at full speed and you have three more ports for 10GB connections if you have certain clients or other servers that need the speed.

Hi Chris,

I'm finally getting my 10GB setup. If I get a switch with SFP not SFP+ do I need anything other than the Chelsio card? I'm thinking about this Switch. https://www.amazon.com/Ubiquiti-Uni...520213949&sr=8-6&keywords=ubiquity+switch+poe and the S320E-SR-XFP Chelsio card. This means I'd just get an SFP cable and plug and play?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
SFP and SFP+ differ primariy in the fact that 'plain' SFP is only 1gig where SFP+ is 10gig.
If you get a switch that doesn't support SFP+, you might as well not do it.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Hi Chris,

I'm finally getting my 10GB setup. If I get a switch with SFP not SFP+ do I need anything other than the Chelsio card? I'm thinking about this Switch. https://www.amazon.com/Ubiquiti-Uni...520213949&sr=8-6&keywords=ubiquity+switch+poe and the S320E-SR-XFP Chelsio card. This means I'd just get an SFP cable and plug and play?
The Ubiquity switch has two SFP+ (10GB) ports and two SFP (1GB) ports. How many ports do you need to be 10GB is the question?
 

Maelos

Explorer
Joined
Feb 21, 2018
Messages
99
This community is awesome. This and many other threads are quite similar in requirements and help needed, but its awesome to see the community pounce on the opportunities to help. I too had questions about the noise, storage arrangement, etc. While I am a few steps behind you iHeartMacs, but look forward to hearing more and seeing your progress.
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
The Ubiquity switch has two SFP+ (10GB) ports and two SFP (1GB) ports. How many ports do you need to be 10GB is the question?

Hi I was thinking SFP+. Where I get confused is do I need two different transceivers? One compatible with the Ubiquity switch and one compatible with the Chelsio card?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
The Chelsio cards can be a little picky but you should be able to us generic transceivers on both sides.
What is the distance?

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

iHeartMacs

Explorer
Joined
Oct 5, 2017
Messages
56
1 meter. Thx.
 
Status
Not open for further replies.
Top