BUILD Hardware Recommendations For Low-Power 16 disk Build

Status
Not open for further replies.

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
Hello everyone,
I'm looking for some hardware recommendations for my FreeNAS server build. Basically, I want to use some parts from my current machine, and buy a few "required" parts to build a low-power FreeNAS server. Once my PC parts start failing I will slowly replace them with what is appropriate for a server. What I currently own and would like to use is:

A.) Custom case with 30 - 3.5" bays, 2 - 5.25" bays.
B.) 16 - 4TB Green drives.
C.) 3 - 2TB drives.
D.) 3ware 9650SE-24M8 Raid Card (I know RAID controllers are heavily frowned upon, but it's what I have).
E.) SeaSonic X850 80Plus Gold PSU.

I need to buy a Mobo/CPU and ram. For fitment, Mobo form factor doesn't really matter for me (case is ATX, but can be adapted).

I intend to use the server to store and stream all my media and documents for my home. I'm planning to use Kodi, VLC and explorer via Windows machines to stream and access, so no transcoding. Right now I have no intention of running any jails or anything else either. I also have about 10 3MP IP cams that will be writing to the server via NFS shares, unless I can get CIFS working in the cams. The cams are pretty low bandwidth. At some point I might try to backup my PCs to the server also (I don't know how to do that yet). All the machines in my house are Windows machines, so I will be using CIFS for everything except the IP cams. I was going to use 4TB drives to make two 8 disk Z2 vdevs striped together for one large pool/volume. If I need more space in the future, I will add a 3rd 8 disk Z2 vdev to the pool. I was planning to use 2TB drives to make a completely separate 3 disk Z1 vdev and pool/volume for the IP cams. I don't intend to ever resilver the vdev for the cams. I just don't want to completely stop recording if 1 drive fails. There will never be more than 2 users (plus the cams) accessing the server at the same time. I also keep a second backup of all my data on external HDDs. I'm trying to decide which motherboard/CPU would best suit my needs. I don't know how much CPU power I really need. I was looking at low-power C2750 boards. I looked a little at the Asrock 2750d4i board, but I will be running an expansion card, so I really don't need 12 onboard sata ports (maybe save a couple watts). Plus I heard one of the sata controllers eats up a considerable amount of power (turns out it's more the added PCIe switch). I particularly like the Supermicro A1SAM-2750F because it does NOT use soDimm RAM and it has an extra expansion slot (if I ever need it). One thing I did notice about that board, is it doesn't look like freeBSD 10 is a "supported" OS. I was also planning to use 32GB of ECC RAM. If I need more CPU power or RAM, I was looking at an e3-1230v3 CPU, since it idles at about the same watts. I haven't looked into motherboards for that CPU yet. It would be cool to have more than 4 RAM slots though (16GB modules are expensive!). I will also be moving toward Red drives as my Greens fail. I've done a lot of FreeNAS reading over the past few weeks, but I'm sure I still fit into the Noob category. As such, any advice and hardware/software configurations would be greatly appreciated. Also, this is my second FreeNAS build, the first was just to learn a little about FreeNAS. I will not go into details on it, for fear of being flamed ;) So my questions are:

1.) Does scrubbing or resilvering take much CPU?

2.) Do you think a C2750 will be powerful enough for up to thirty 4TB disks with no encryption or dedup?

3.) Any idea why FreeBSD 10 (not sure what version FreeNAS uses) wouldn't be listed as supported on the A1SAM-2750F, but is on the other Supermicro C2750 boards?

4.) Is 32GB of RAM enough for now, how about if I add another 8 disk vdev?

5.) What make/model RAM should I buy? The supported list only shows 4 different models.

6.) If you don't think the C2750 is a good choice, how do you feel about the e3-1230v3 and what Mobo/RAM should I be looking at for low power consumption?

7.) Any idea if smart data is obtainable in FreeNAS when using a 3ware controller? The latest firmware suposedly doesn't support JBOD mode. Disks must be exported as single volumes.

8.) Should I flash the 3ware controller back to a previous version that does support JBOD? Although I'm not sure if it would support 2TB+ drives then.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Hello everyone,
I'm looking for some hardware recommendations for my FreeNAS server build. Basically, I want to use some parts from my current machine, and buy a few "required" parts to build a low-power FreeNAS server. Once my PC parts start failing I will slowly replace them with what is appropriate for a server. What I currently own and would like to use is:

A.) Custom case with 30 - 3.5" bays, 2 - 5.25" bays.
B.) 16 - 4TB Green drives.
C.) 3 - 2TB drives.
D.) 3ware 9650SE-24M8 Raid Card (I know RAID controllers are heavily frowned upon, but it's what I have).
E.) SeaSonic X850 80Plus Gold PSU.

I need to buy a Mobo/CPU and ram. For fitment, Mobo form factor doesn't really matter for me (case is ATX, but can be adapted).

I intend to use the server to store and stream all my media and documents for my home. I'm planning to use Kodi, VLC and explorer via Windows machines to stream and access, so no transcoding. Right now I have no intention of running any jails or anything else either. I also have about 10 3MP IP cams that will be writing to the server via NFS shares, unless I can get CIFS working in the cams. The cams are pretty low bandwidth. At some point I might try to backup my PCs to the server also (I don't know how to do that yet). All the machines in my house are Windows machines, so I will be using CIFS for everything except the IP cams. I was going to use 4TB drives to make two 8 disk Z2 vdevs striped together for one large pool/volume. If I need more space in the future, I will add a 3rd 8 disk Z2 vdev to the pool. I was planning to use 2TB drives to make a completely separate 3 disk Z1 vdev and pool/volume for the IP cams. I don't intend to ever resilver the vdev for the cams. I just don't want to completely stop recording if 1 drive fails. There will never be more than 2 users (plus the cams) accessing the server at the same time. I also keep a second backup of all my data on external HDDs. I'm trying to decide which motherboard/CPU would best suit my needs. I don't know how much CPU power I really need. I was looking at low-power C2750 boards. I looked a little at the Asrock 2750d4i board, but I will be running an expansion card, so I really don't need 12 onboard sata ports (maybe save a couple watts). Plus I heard one of the sata controllers eats up a considerable amount of power (turns out it's more the added PCIe switch). I particularly like the Supermicro A1SAM-2750F because it does NOT use soDimm RAM and it has an extra expansion slot (if I ever need it). One thing I did notice about that board, is it doesn't look like freeBSD 10 is a "supported" OS. I was also planning to use 32GB of ECC RAM. If I need more CPU power or RAM, I was looking at an e3-1230v3 CPU, since it idles at about the same watts. I haven't looked into motherboards for that CPU yet. It would be cool to have more than 4 RAM slots though (16GB modules are expensive!). I will also be moving toward Red drives as my Greens fail. I've done a lot of FreeNAS reading over the past few weeks, but I'm sure I still fit into the Noob category. As such, any advice and hardware/software configurations would be greatly appreciated. Also, this is my second FreeNAS build, the first was just to learn a little about FreeNAS. I will not go into details on it, for fear of being flamed ;) So my questions are:

1.) Does scrubbing or resilvering take much CPU?

2.) Do you think a C2750 will be powerful enough for up to thirty 4TB disks with no encryption or dedup?

3.) Any idea why FreeBSD 10 (not sure what version FreeNAS uses) wouldn't be listed as supported on the A1SAM-2750F, but is on the other Supermicro C2750 boards?

4.) Is 32GB of RAM enough for now, how about if I add another 8 disk vdev?

5.) What make/model RAM should I buy? The supported list only shows 4 different models.

6.) If you don't think the C2750 is a good choice, how do you feel about the e3-1230v3 and what Mobo/RAM should I be looking at for low power consumption?

7.) Any idea if smart data is obtainable in FreeNAS when using a 3ware controller? The latest firmware suposedly doesn't support JBOD mode. Disks must be exported as single volumes.

8.) Should I flash the 3ware controller back to a previous version that does support JBOD? Although I'm not sure if it would support 2TB+ drives then.

With that much storage I'd say e5 platform is your only real option. Get a proper HBA and put it in a chassis with sas expander backplane. It's not like you'll be using tons more electricity than the atom, and it will be cheaper in the long run than buying an atom then realizing it doesn't support enough ram and having to buy the e5 anyway.
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
I guess I never really thought that much about the total storage capacity. I was thinking I would have somewhere around 34TB usable, not so much 64TB installed. I thought maybe 32GB of RAM for 34TB usable might be okay with probably only one user at a time. Is the 1GB per 1TB rule for installed space or usable space?
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
I'm pretty new to the FreeNAS game, but I'll throw this in... 16 drives and "low power" are orthogonal. If you want low-power, buy the biggest drives you can and reduce your spindle count. 8x6TB in RAIDz2 would give you 36TB usable.

Does your case move enough air to keep that many drives cool? High HDD temps are associated with substantially reduced lifespans.
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
Yeah, I guess "low power" for me is a relative term. I like the idea of fewer, high capacity drives, and will probably move that way as my drives fail, but I really don't have $1500 to throw at new drives right now. Currently the case probably doesn't flow enough air, although with windows I haven't had any problems with temps. I will add fans/ upgrade them if need be. My case is basically just a big box. With this whole build, I was looking for a somewhat cost effective solution to move to zfs, and get more redundancy than REFS in Windows Storage Spaces has in parity mode (current configuration). I thought I might be able to accomplish my goal with an Atom board and 32GB of RAM for a total of around $600. However, if I have to go with more than 32GB of RAM, it looks like an e5 platform will be required, which for the cheapest CPU/Mobo I can find, puts me at $550. On top of that 64GB of RAM would be another $450, then there is the increased power consumption also. Is there any cheaper combo that will get me above 32GB RAM, without leaving the 22nm CPU lithography or buying $365 UDimms?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
I'm pretty new to the FreeNAS game, but I'll throw this in... 16 drives and "low power" are orthogonal. If you want low-power, buy the biggest drives you can and reduce your spindle count. 8x6TB in RAIDz2 would give you 36TB usable.

Does your case move enough air to keep that many drives cool? High HDD temps are associated with substantially reduced lifespans.


I'm not pretending to have the slightest clue how to calculate usable space on a Z2 pool, but according to Bidule0hm's calculator (thank you for all your hard work!), a Z2 pool with 8x6TB drives gives a usable space of 28TB. Z1 would give 33TB usable. So if I wanted to stay with one vdev I would need 10x6TB drives to get 37TB in Z2.
 
Joined
Oct 2, 2014
Messages
925

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
I'm not pretending to have the slightest clue how to calculate usable space on a Z2 pool, but according to Bidule0hm's calculator (thank you for all your hard work!), a Z2 pool with 8x6TB drives gives a usable space of 28TB. Z1 would give 33TB usable. So if I wanted to stay with one vdev I would need 10x6TB drives to get 37TB in Z2.
FYI, in a multi-user environment, or as vm storage, or in many other use cases, a pool consosting of a single vdev will not provide adequate performance.

Never mind, actually went back to read the original post. Carry on. :D
 
Joined
Oct 2, 2014
Messages
925
I'm not pretending to have the slightest clue how to calculate usable space on a Z2 pool, but according to Bidule0hm's calculator (thank you for all your hard work!), a Z2 pool with 8x6TB drives gives a usable space of 28TB. Z1 would give 33TB usable. So if I wanted to stay with one vdev I would need 10x6TB drives to get 37TB in Z2.
FYI, in a multi-user environment, or as vm storage, or in many other use cases, a pool consosting of a single vdev will not provide adequate performance.
The solution to this would be striped vdevs i believe if its for VM storage or any other high use
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
What effect, if any, does the clock speed of the installed RAM have on a FreeNAS box? More specifically, is less GB of RAM per TB of HDD required, if it is of higher clock speed or lower latency? I saw some server Mobos support ram as slow as 800MHz. I did search the forums and google this time, but I didn't find much regarding RAM speeds and FreeNAS.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Ram speed doesn't really matter much for freenas and actually it doesn't matter much at all anymore.
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
I'm considering a Xeon e5-1620v3 cpu and either an x10srl or an x10sra with 32GB of DDR4 to start with. Any advice on that setup? I would love to hear actual idle power consumption numbers for just the boards, CPU and RAM. Haven't been able to find much on google.
 

DaveFL

Explorer
Joined
Dec 4, 2014
Messages
68
I would take a serious look at X1520 or X1540. High performance and low power.
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
So it looks like an e5-1620v3 with a Supermicro x10srl and 2x16GB DDR4 will total $840. That works out to $181 more than the a1sam-2750f with 4x8GB DDR3. Which is an additional 27% over the total of $659 for the a1sam-2750f. I think the 2750 would be perfect for my application if only I could get affordable 16GB udimms. I'm probably being far too anal about power consumption, but the 2750 boards are supposed to idle at about 15W with RAM. I did find Idle power consumption for two e5-2620v3s with an x10drh board and 8 sticks of DDR4. It was about 65W, but I'm not sure how much lower the power would be for a single CPU, or how exactly the e5-2620 consumption corresponds to the e5-1620 consumption. The Intel datasheet shows the power for various CPU C-States on the e5-1620 as: C6 16W, C3 28W, C1E 53W. Whereas the e5-2620 is: C6 21W, C3 35W, C1E 48W. They talk about P-States as well. So, I'm thinking an LGA2011 mobo consumes about 23W and an e5-1620 (assuming it can go into C6 state, which I doubt) should be about an additional 16W. That brings my total to 39W+ RAM, I'm going to say 3W a stick x2 sticks, so 45W, which I think is VERY conservative. With a 90% PSU, that would cost me an additional $40 a year in electric. My current mITX NAS/NVR uses 35W from the wall while my 10 IP cams are writing to it. It is an N2900 CPU with 2x8GB of RAM and 4x500GB 7200rpm drives in a Z2 configuration and 2-80mm case fans. So, I really do believe it is possible that a 2750 board idles at 15W.
 

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
I would take a serious look at X1520 or X1540. High performance and low power.

I was looking at those before and they seemed really awesome in every way except the price. The motherboards are very expensive and the CPU is higher than a cats butt! Another awesome Intel product though... Thank you for the recommendation too. I still feel like there has to be something out there I haven't found yet.
 

DaveFL

Explorer
Joined
Dec 4, 2014
Messages
68
So it looks like an e5-1620v3 with a Supermicro x10srl and 2x16GB DDR4 will total $840. That works out to $181 more than the a1sam-2750f with 4x8GB DDR3. Which is an additional 27% over the total of $659 for the a1sam-2750f. I think the 2750 would be perfect for my application if only I could get affordable 16GB udimms. I'm probably being far too anal about power consumption, but the 2750 boards are supposed to idle at about 15W with RAM. I did find Idle power consumption for two e5-2620v3s with an x10drh board and 8 sticks of DDR4. It was about 65W, but I'm not sure how much lower the power would be for a single CPU, or how exactly the e5-2620 consumption corresponds to the e5-1620 consumption. The Intel datasheet shows the power for various CPU C-States on the e5-1620 as: C6 16W, C3 28W, C1E 53W. Whereas the e5-2620 is: C6 21W, C3 35W, C1E 48W. They talk about P-States as well. So, I'm thinking an LGA2011 mobo consumes about 23W and an e5-1620 (assuming it can go into C6 state, which I doubt) should be about an additional 16W. That brings my total to 39W+ RAM, I'm going to say 3W a stick x2 sticks, so 45W, which I think is VERY conservative. With a 90% PSU, that would cost me an additional $40 a year in electric. My current mITX NAS/NVR uses 35W from the wall while my 10 IP cams are writing to it. It is an N2900 CPU with 2x8GB of RAM and 4x500GB 7200rpm drives in a Z2 configuration and 2-80mm case fans. So, I really do believe it is possible that a 2750 board idles at 15W.
2750 boards use expensive non standard ram. Get a Xeon 1540D or 1520D depending on your CPU requirements. They used DDR4!
 

DaveFL

Explorer
Joined
Dec 4, 2014
Messages
68
I was looking at those before and they seemed really awesome in every way except the price. The motherboards are very expensive and the CPU is higher than a cats butt! Another awesome Intel product though... Thank you for the recommendation too. I still feel like there has to be something out there I haven't found yet.
The 1520 is under $500.

http://www.wiredzone.com/supermicro...s-embedded-processor-x10sdv-4c-tln2f-10025118

Note above board is only 10GB which may or may not be ok for you.

There is also a rumored ASRock that is expected to ship with LSI 3008 integrated. But it is Micro ITX. The current supermicro boards work great with your standard suggested LSI IT Mode card.
 
Last edited:

paylesspizzaman

Explorer
Joined
Sep 1, 2015
Messages
92
The 1520 is under $500.

http://www.wiredzone.com/supermicro...s-embedded-processor-x10sdv-4c-tln2f-10025118

Note above board is only 10GB which may or may not be ok for you.

There is also a rumored ASRock that is expected to ship with LSI 3008 integrated. But it is Micro ITX. The current supermicro boards work great with your standard suggested LSI IT Mode card.

I'm very seriously considering buying the x10sdv-4c-tln2f. The only place that I've found it under $500 and not sold out, says special order up to 2 weeks before shipment. I know absolutely nothing about 10Gbe, but the pics of the board looks like it has regular RJ45 connectors and Supermicro's page under network connections says "Supports 10GBASE-T, 100BASE-TX, and 1000BASE-T, RJ45 output". Does that mean I can plug in a regular 1Gbe cable and it work fine? I also read that copper 10Gbe uses a lot more power than 10Gbe fiber. Would the 10Gbe controller use less power if connected to a 1Gbe network or not connected at all? If the controller uses too much power, can it be disabled in the BIOS, so I can use a 1Gbe add in card? I also read that there is no driver in FreeNAS for the network controller on the D-1520. Is this still an issue?
 
Status
Not open for further replies.
Top