New 24bay Build, 5 Questions

[craigdt]

Hi All,

After building my own FreeNAS build not to long ago, which has been rock solid and fast, my friend has requested my help in getting a 24 bay monster going.
Right now my friend has 2 NAS's, a Netgear ReadyNAS which is giving him performance problems and a old FreeNAS server which he pieced together with standard desktop components (non ECC ram, yes my friend likes to live on the edge)!
He has about 24TB of data to store and its growing, we are thinking that a 24bay system with 6TB drives should last him a while at least.
I've done some initial component selection and costing, but I do have some questions in regards to some of the components I've picked.

What I have picked so far:

MotherBoard: Supermicro X10SRi-F
CPU: Intel Xeon E5 1620 v3 (3.5Ghz 4 Core)
CPU Cooler: Noctua NH-U12DX i4 (Replace fan with a Noctua NF-12 Industrial PCC 120mm 3000RPM PWM)
Memory: (64GIG in total to begin with, add more later as we add more hdd's) Crucial Memory Crucial 16GB DDR4 2133 MT/ s (PC4-2133) CL15 DR x4 ECC Registered DIMM 288pin [CT16G4RFD4213]
Raid Card: (3x) IBM ServeRAID M1015
Chassis: Norco RPC-4224 4U Rack Chassis

Norco 120mm FAN Wall
6x Norco Mini-SAS(SFF-8087) to Mini-SAS(SFF-8087) cable​

Optional Additional Network Card: Intel I350T2 Gigabit Network Adapter - 2 port
Fans: Noctua NF-F12 Industrial PPC 120mm 3000RPM PWM
or
Delta QFR1212GHE-PWM 120mm Case cooler (Haven't seen anything come close to it in PWM)
HDD Type: SATA
Redundant Power Supply: Zippy MRG-5800V4V


Question 1: Redundant Power Supplies, this is an area I have no idea about, I started looking at brands and prices and I just don't know what is considered bang for buck. Being its important as without power the server will sit there and do nothing, can anyone recommend a list of brands / model range / prices to look at?

Seems the zippy I've specified is a common brand in Australia, the model is also recommended by the supplier website for the norco case.

Question 2: In regards to the X10SRi-F motherboard fan control, does anyone know from experience if this is the same set up as the X10SL7-F where if you use the the setting for Heavy IO, the FAN-A port will be the one that is controlled by the system temperature which I found to be best for the HDD fans and the rest of the fan ports are determined based on the CPU temperature ?
There doesn't seem to be any decent documentation on what the zones are and I ended up working it out on the X10SL7-F by trial and error. I'm hoping there is no differences between the two motherboards so I don't have to do any experimentation on the new motherboard!

At this stage I'm going to assume the thermal zone arrangement is the same between the two motherboards, why would Supermicro do anything different!

Question 3: Can anyone see a problem using the Reverse Breakout cable for the motherboard sata ports to connect to two of the SFF-8087 backplane ports (will be using SATA hdd's only)? I choose this option to save some cash and heat by only having 2 IBM ServeRAID cards vs 3

Friend is happy to go with 3x IBM ServerRAID cards so not going to bother with the reverse breakout cables!

Question 4: In regards to the (120mm) Noctua NF-F12 3000 PWM fans I have chosen, I have the 140mm version on my own server which I find are great for a warm room, but my friends server will be sitting in a garage so he could probably handle something a bit noisier, can anyone recommend any 120mm PWN fans that run a bit faster with more air flow and pressure for about the same value ?

I'm thinking Noctua iPCC fans should suffice, the 140mm version run well in my server and the case I use probably has worse airflow than the norco when comparing the designs
Alternative the Delta fan would leave nothing to chance!

Question 5: Can anyone see any obvious incompatibilities with the hardware I have picked ?

Many thanks,

Craig

 

[Techie Will]

Question 4: Look up Noctua's iPPC line of fans, they offer 2,000 and 3,000 RPM versions, and come in all black.

What CPU cooler are you using for the Xeon? The 1620v3 does not include one.

 

[craigdt]

Question 4: Look up Noctua's iPPC line of fans, they offer 2,000 and 3,000 RPM versions, and come in all black.

What CPU cooler are you using for the Xeon? The 1620v3 does not include one.

Whoops, I meant the iPPC 120mm fan, didn't list it in the list correctly, thanks for the suggestion it made me realise I didn't compose my post correctly.

I totally forgot about those CPU's not coming with a heat sink... I'm now looking for a narrow ILM heatsink to fit the socket. Do you have any recommendation for a narrow ILM cooler for the Xeon ?

 

[jgreco]

You might want to look to see if you can find a SuperMicro chassis instead. Especially once you start trying to integrate stuff like a redundant power supply, there's a nice aspect to being able to just plug things in and have it all be designed to work together. You'll gain stuff like the PMbus just working right with the motherboard as well.

 

[craigdt]

You might want to look to see if you can find a SuperMicro chassis instead. Especially once you start trying to integrate stuff like a redundant power supply, there's a nice aspect to being able to just plug things in and have it all be designed to work together. You'll gain stuff like the PMbus just working right with the motherboard as well.

I was looking at the Supermicro cases, just they are very expensive here in Australia, the cheapest I have seen is about 3x the cost of the Norco case kitted out. I'll talk to my friend as he may be-able to get a SuperMicro case through his suppliers but it might seem a little overkill for home use ;)

 

[SweetAndLow]

You basically parted out the exact build I'm wanting to make. I have been trying to decide between the norco and a supermicro 846E16. The norco might have cooling issues and the supermicro is very loud. I would also look for fans that have high static pressure and probably in the 80mm range. These should pull air through the HDD bays better.

 

[cyberjock]

You basically parted out the exact build I'm wanting to make. I have been trying to decide between the norco and a supermicro 846E16. The norco might have cooling issues and the supermicro is very loud. I would also look for fans that have high static pressure and probably in the 80mm range. These should pull air through the HDD bays better.

I have a norco, and I had to buy the same fans that are in the 846E16 to get good cooling. So save yourself the trouble and just buy the darn Supermicro chassis.

 

[craigdt]

I have a norco, and I had to buy the same fans that are in the 846E16 to get good cooling. So save yourself the trouble and just buy the darn Supermicro chassis.

Are you happy with the norco despite the fact that you had to put some faster fans in it ?, what was the fan model that you used that was in the 846E16 ?

We will look into a SuperMicro case, perhaps my friend can source one from his suppliers at a good price, I had a look online for Australian retailers and found a few examples:

The following one is a good price but it states its individual sata ports instead of sas, ports, thats a whole lot of reverse breakout cabling
http://www.techbuy.com.au/p/171344/CASINGS_SERVER_-_5U/Supermicro/CSPC-846TQ-900B.asp

This one seems to use mini-sas cables which is more preferable:
http://www.auspcmarket.com.au/supermicro-846a-r1200b-4u-rack-chassis/

Can't find anyone selling the 846E16 in Australia, unfortunately NewEgg wont ship here because we have SuperMicro distributors already, plus I imagine the shipping would kill the cost even if we could

 

[cyberjock]

Are you happy with the norco despite the fact that you had to put some faster fans in it ?, what was the fan model that you used that was in the 846E16 ?

I bought a Supermicro chassis in January. I just haven't had a chance to switch the guts of my server to the chassis. Does that answer your question?

As for the fan model number, I don't remember.

 

[jgreco]

The general points against the Norco seem to be lower build quality, and more "DIY" hackiness to solve power and cooling problems.

The SuperMicro stuff is hella-expensive but very solid, with a variety of options (E16, A, etc) for every need. Some people complain the fans scream, but usually on investigation that turns out to be someone not hooking up the fans correctly or omitting the PMbus. It's not quiet even when set up correctly, but you're just not going to get that in a 24 drive 4x6 4U, the air pressures required are a matter of physics, and the fans used by SuperMicro are designed for reliability in the data center, not quietness.

Most of the time, people with Norcos seem to wind up spending a fair bit extra to get the power supply and cooling issues sorted out, and in my opinion that significantly reduces the value of the dollars saved. When you start talking 24 drives at $100 each, mistakes mean you're cooking $2400 in disks, plus risking your data on top of it.

 

[craigdt]

I bought a Supermicro chassis in January. I just haven't had a chance to switch the guts of my server to the chassis. Does that answer your question?

As for the fan model number, I don't remember.

Thanks for info, it looks like SuperMicro is the way to go, in the end it looks like only about a $300 difference going the Norco way and putting PSU and decent fans in it compared to the SuperMicro SC846A-R1200

 

[craigdt]

The general points against the Norco seem to be lower build quality, and more "DIY" hackiness to solve power and cooling problems.

The SuperMicro stuff is hella-expensive but very solid, with a variety of options (E16, A, etc) for every need. Some people complain the fans scream, but usually on investigation that turns out to be someone not hooking up the fans correctly or omitting the PMbus. It's not quiet even when set up correctly, but you're just not going to get that in a 24 drive 4x6 4U, the air pressures required are a matter of physics, and the fans used by SuperMicro are designed for reliability in the data center, not quietness.

Most of the time, people with Norcos seem to wind up spending a fair bit extra to get the power supply and cooling issues sorted out, and in my opinion that significantly reduces the value of the dollars saved. When you start talking 24 drives at $100 each, mistakes mean you're cooking $2400 in disks, plus risking your data on top of it.

I guess if you weren't going redundant PSU, the Norco case could be good value with 3 of those Delta 120mm 210CFM fans in it. But my friend wants redundant PSU, wished the case my own server was in would support redundant PSU, but thats a different story.

So we found a retailer that sells the SC846A-R1200: http://www.supermicro.com/products/chassis/4U/846/SC846A-R1200.cfm just need to verify that the X10SRi-F works in it ok

 

[jgreco]

The Norco case might be a deal if you could get the engineering right, for power supplies and cooling and cabling, but the reality is that most of those are being sold to AV enthusiasts and other hobbyists who probably aren't going to have the technical expertise to be aware of all the issues. Especially if you're going for something like a SAS expander, getting a SuperMicro E16 has a bit of a cost premium, but not as much as getting a discrete SAS expander and all the related cabling, and figuring out how to make that work. The A option is a little more complex because it's more flexible but more complicated.

The redundant PSU (or actually any PSU) option for the SuperMicro is nice because of the PMbus, which gives you realtime and historical monitoring of power utilization. This by itself is "worth something" IMHO.

 

[cyberjock]

If you shop around you can often find used Supermicro chassis for a very good deal on ebay. I got my 4U/24bay Supermicro SAS3 chassis for $310 with shipping in January. Sadly, I haven't had 4 hours to move to it. :(

 

[smit4.hage4]

craigdt, did your friend manage to get all of that working?

I'm in Australia too and the limited parts selection is so frustrating. I'm looking and a very similar build, same Motherboard & CPU, Supermicro Case.

I'd really like to know if the memory worked out fine. The QVL for that board has a lot of sticks not available here.

 

[RichTJ99]

Would this work for you?

http://www.ebay.com/p/Supermicro-SAS-846-TQ-24-Bay-Server-SAS-SATA-HDD-Backplane/1222720500

I cant tell if its sas or sas2, or 3?

 

[jgreco]

The TQ isn't particularly recommended; having 24 individual SAS connectors to paw through when something goes wrong is a bad idea. More discussion of better backplane options at

https://forums.freenas.org/index.ph...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

 

[smit4.hage4]

RichTJ99, thank you but that Ebay unit doesn't appear to be available for shipping to Australia. I'm kind of set on a brand new backplane

jgreco, thank you so much, I read that thread but skipped that particular paragraph, (shame on me).
I'll get the BE16 as I'm looking to go the m1015 route. I had a few Supermicros on my spreadsheet and thought they both had a single 1x sff8087. Clearly I need to get my eyes checked.

 

[jgreco]

jgreco, thank you so much, I read that thread but skipped that particular paragraph, (shame on me).
I'll get the BE16 as I'm looking to go the m1015 route. I had a few Supermicros on my spreadsheet and thought they both had a single 1x sff8087. Clearly I need to get my eyes checked.

No problem. There's a Lot Of Things To Know and as long as you can stomach my saying it like I've said it a hundred times before, I feel that the TQ's in a 24 bay are a bad idea and you'll get to hear me say it again. ;-)

 

[craigdt]

craigdt, did your friend manage to get all of that working?

I'm in Australia too and the limited parts selection is so frustrating. I'm looking and a very similar build, same Motherboard & CPU, Supermicro Case.

I'd really like to know if the memory worked out fine. The QVL for that board has a lot of sticks not available here.

Hi smit4,

Yes my friend got it all working, but he ran into a couple of problems when assembling.
1. The redundant powersupply he had to drill some holes to screw it in.
2. The cpu cool we had choosen was actually to tall for the case, should have gone the shorter one. but he just cut a hole in the case for it to stick out the top.
3. The sas cables he ordered were too short and he had to buy longer ones

Once he overcame these problems though, he got the system up and running. Very fast as well, he even has a small ssd raid in there as well.

However if you can swing the cash, I would recommend the supermicro case, better quality and features...

But if I were to build a 24 bay system, I would most likely go the norco just to save some cash, plus the supermicro case I would like to buy isn't available in Australia