Hardware recommendation for new SC836

[sovking]

Hi all,
just got a deal for a new Supermicro 836BE1C-R1K03B at only 211 Euro. As you already know it has SAS3 expander and a couple of 1000W Titanium PSU.
Now I've to decide what to put inside for my new home server.
The primary purpose of this server will be: 1) reliable storage for important files, family photos and video, 2) download and upload jails, 3) short term storage for home videosurveillance and maybe for zoneminder 3) storage for external VM, 4) maybe some VM during development work (not sure if I will need that).
I don't need much storage space, Just now I'm using less than 3 TB for important files, and I think it can grow of some TB during the next 5 years. For surveillance files I need another couple of TB.
So, I'm thinking about raidZ2 of 6x4TB disks, and maybe a mirror of a couple ssd to serve VM.
This server will be placed at the basement or under roof depending on temperatures, and I would like to make it a low power server.

What will be your suggestion about motherboard and CPU that suits my needs and make a good use of this new case ?

 

[IceBoosteR]

Hi,

have a look at the hardware recommendation guide:
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

The superchassis is capable for plenty drives thats awesome. I would like to ask you for your budget, according on this we can offer you examples. I would say, normally you are fine with a single socket server with high clock speed and a lot of RAM. But it is the question if you would like to go for new parts, or used ones aswell and if you have concerns about your energy usage.

Cheers
Ice

 

[BigDave]

@sovking
On Supermicro's website, look up that chassis model number and find the list of motherboards that work with that chassis.
The things that a lot of folks miss is how long the power distribution wires are. The mounting for MOTHERBOARDS are standard, but the wire harness for the 24pin power may not reach just any flavor of motherboard. You might need to be prepared to alter/change PDU depending on your MoBo choice.

 

[sovking]

Hi IceBoosteR, you are right: it depends on budget... I've not yet defined exactly: necause I've already case+expander+psu we could start with 600-800 Euro for mobo+cpu+ram before disks ?? But I don't want restrict the choice looking only at the budget, I would like to see the options and then restrict according budget, not forgetting WAF ;).
I'm open to buy used parts if it worth.
Regarding hard-disks, at beginning I'll start re-using some I already have sparse around (mix of green, desktop and enterprise) to move/resilvering gradually to new nas disks after an initial running in period.
While regarding SSD (in all variants) I could follows offers and deals.
I have concerns regarding energy usage: I live in Europe, and here Energy costs... and, even I'm producing some energy from sun, I don't like to waste energy if I don't need it.

@BigDave good observation: I looked at that, but I'm not sure that such list is updated often: supermicro continuosly produce new motherboards, even with the same position of power connector.

I took a look at the hardware reccomandation guide... but I think I have to decide which class of server (motherboard/cpu/ram) is more suitable for my needs, that guide gives me too much options at this stage and maybe it could be updated.

For example, this are my current thought: only 1 CPU (so cut away all motherboards with dual socket). Two cores are ok for general file serving, but I would like at least 4 core to better serve other services (like download/upload programs, rsync with work, cloud sync, etc), zoneminder, etc. Eight cores would offer space some 24h Virtualization, which now are sitting on my desktop when I need it: actually I don't need this... just in case. So I'm thinking about 4 or 8 core CPU.
Do you think I need high frequency (e.g. > 3GHz) ? I will not do transcoding... The bare minimum is to run everything smoothly filling Gigabit Ethernet using SMB (at least for now).

How many CPU fits these requirements ?

• Avoton >= C2550 requires Unbuff DDR3
  
• Denverton >= C3558 requires Unbuff DDR4
  
• Xeon E3 requires Unbuff DDR3 for v1-v4 and Unbuff DDR4 for v5-v6
  
• Xeon D requires DDR4 Ubuff/Reg/LR
  
• Xeon E5 with single socket motherboard requires Reg DDR3 for v1-v2 and Reg DDR4 for v3-v4

Previous version of Xeon E3 and E5 can be found used, sometime at good prices, while the others usually can be found only as new. E5 offers a lot of opportunity for future expansion, but I think that E5+mobo will require more energy...


The memory type impact on costs: current price for new Ubuff memory is around 12 Euro/GB, the same for DDR4 registered. While DDR3 registered can be found for half than that. I can find some deal on used part.. but the ratio it's the same.
So 12 Euro/GB means 190 Euro for 16 GB (minimum for my needs); but to serve ZFS better and to make some virtualization I would like to increase such amount to 32 or 64 (64 GB only in case of DDR4 of course).
While REG DDR3 is widely available both used and new, for Xeon E3 v1-v4 and Xeon E5 v1-v2.

So the choice is quite wide...
What is your advice ?

Thanks.

 

[Chris Moore]

Like @BigDave said, I repurposed a couple of older chassis and put newer boards in them. The power leads were not long enough and I needed to make some modifications to get everything working.

 

[IceBoosteR]

Hi IceBoosteR, you are right: it depends on budget... I've not yet defined exactly: necause I've already case+expander+psu we could start with 600-800 Euro for mobo+cpu+ram before disks ?? But I don't want restrict the choice looking only at the budget, I would like to see the options and then restrict according budget, not forgetting WAF ;).
I'm open to buy used parts if it worth.
Regarding hard-disks, at beginning I'll start re-using some I already have sparse around (mix of green, desktop and enterprise) to move/resilvering gradually to new nas disks after an initial running in period.
While regarding SSD (in all variants) I could follows offers and deals.
I have concerns regarding energy usage: I live in Europe, and here Energy costs... and, even I'm producing some energy from sun, I don't like to waste energy if I don't need it.

@BigDave good observation: I looked at that, but I'm not sure that such list is updated often: supermicro continuosly produce new motherboards, even with the same position of power connector.

I took a look at the hardware reccomandation guide... but I think I have to decide which class of server (motherboard/cpu/ram) is more suitable for my needs, that guide gives me too much options at this stage and maybe it could be updated.

For example, this are my current thought: only 1 CPU (so cut away all motherboards with dual socket). Two cores are ok for general file serving, but I would like at least 4 core to better serve other services (like download/upload programs, rsync with work, cloud sync, etc), zoneminder, etc. Eight cores would offer space some 24h Virtualization, which now are sitting on my desktop when I need it: actually I don't need this... just in case. So I'm thinking about 4 or 8 core CPU.
Do you think I need high frequency (e.g. > 3GHz) ? I will not do transcoding... The bare minimum is to run everything smoothly filling Gigabit Ethernet using SMB (at least for now).

How many CPU fits these requirements ?

• Avoton >= C2550 requires Unbuff DDR3
  
• Denverton >= C3558 requires Unbuff DDR4
  
• Xeon E3 requires Unbuff DDR3 for v1-v4 and Unbuff DDR4 for v5-v6
  
• Xeon D requires DDR4 Ubuff/Reg/LR
  
• Xeon E5 with single socket motherboard requires Reg DDR3 for v1-v2 and Reg DDR4 for v3-v4

Previous version of Xeon E3 and E5 can be found used, sometime at good prices, while the others usually can be found only as new. E5 offers a lot of opportunity for future expansion, but I think that E5+mobo will require more energy...


The memory type impact on costs: current price for new Ubuff memory is around 12 Euro/GB, the same for DDR4 registered. While DDR3 registered can be found for half than that. I can find some deal on used part.. but the ratio it's the same.
So 12 Euro/GB means 190 Euro for 16 GB (minimum for my needs); but to serve ZFS better and to make some virtualization I would like to increase such amount to 32 or 64 (64 GB only in case of DDR4 of course).
While REG DDR3 is widely available both used and new, for Xeon E3 v1-v4 and Xeon E5 v1-v2.

So the choice is quite wide...
What is your advice ?

Thanks.

Hi,

where exactly do you live in Europe? I am living in Germany and I understand the concerns of electric power usage and the costs for it. I am using an Xeon E3-1225v3 with 24GB of ECC memory and I would strongly recommend 32GB. I do have 40TB of RAW disk capacity and server two VMs with 3GB of RAM each. CPU usage is low and normally enough. Don't know how this will be with NVR system (like zoneminder) but I am also looking for options.
If you would like to invest for the next years, maybe keep a look on the newest Intel Xeon Entry Level generation. Xeon E-2146G for example. 6 cores, 3,5Ghz stock speed zp to 4,2ACT and the lowest energy usage you can get. And up to 64GB unbuffered ECC. But this would lead to 1000€ plus 80 bucks for and LSI based SAS controller you will need. I would not got the Avoton or Deverton path.
If you would like to go that E5 path, go with used HW as Chris mentioned, you can save mone here, but I don't know how much energy will be used in idle by these systems.
You can also have my hardware, I need to upgrade :D :D :D
No seriously - there are a ton of options available. I would suggest go the used part of E5 v2 or go with brand new hardware, but go with 6 cores at least to be future proof.
Reg DDR3 is also a good option if you would virtulize, as these are cheep.

Edit: The Denverton CPUs should be good for virtualisation, but I fear the low clock speed for sharing data above 1GBe (just want to make sure the system its future proof.)
I will need to read more about this - but power consumption is awesome on these systems.

Hopefully I get all your points.
Cheers
Ice

 

[sovking]

The cost of electric power usage range from 0.20 to 0.30 per kWh, so it needs to be taken into account.
Ice, your configuration is interesting: how is the system performance and its load ? And how much average power you measure at the wall ?

A system with E3-1220 v2 or v3 with mobo and 32 GB is starting around 800 Euro, with v4 you should add other 150 Euro. With used mobo and cpu, it's possibile to save around 2-300 Euro.
A good combination is E3 v6 + X11SSL-cF which have LSI 3008 integrated + DDR4 ECC UDIMM: as new it's about the same price... :)

I can understand that Avoton is becaming obsolete. Eight core Denverton is interesting for VM, I should check some reviuew but it should sustain 1Gbps transfer... because it is sold with 10Gb Ethernet too :):A2SDi-8C-HLN4F it starts at around 450-500 Euro then you have to add 180-220 Euro for a 16 GB kit: so with 32GB it shoud be got for around 900 Euro.

Going with E5 it's complicated: too many processors, with different versions and motherboard. If I look at v2 version (Ivy Bridge), then 8-core as e5-2650v2, which can be found used around 120-150 Euro, 32GB Reg for 160 Euro and mobo around ~150-250 Euro, so 430-560 Euro (used).

Xeon-D with 8 cores (~800 Euro) with 32 GB DDR4 will be priced around 1200 Euro.

So, price wise, options range from ~500 (used) for E5, to 800-900 for E3 and Denverton, to 1200 for Xeon-D (each filled with 32 GB). Every sistem has 8 core except E3.

 

[Johnnie Black]

You might also consider the recently launched Xeon E-21xx, up to 6 cores.

 

[Chris Moore]

Hi all,
just got a deal for a new Supermicro 836BE1C-R1K03B at only 211 Euro. As you already know it has SAS3 expander and a couple of 1000W Titanium PSU.

That is a fantastic chassis. I have the 4U, 24 bay variant for my ESXi host, but with a non-expander backplane.
This chassis should take just about any system board you want to put in it. The listed boards on Supermicro's website are all dual processor boards, and I am guessing that you probably don't need all that much horsepower.
You are in Europe, so that makes accessibility of hardware a little difficult for you, but you might want to take a look at the list of hardware I put together for another user: https://forums.freenas.org/index.php?threads/new-freenas-build.64095/#post-459293
It is an X9 series Supermicro system board and the main reason for selecting it is the cost of RAM. The Registered DDR3 memory is so much less expensive than DDR4 memory that (in the US) you can save hundreds on the cost of a system that is still very respectable as to what it can do.
It might be worth considering.

 

[Chris Moore]

You might also consider the recently launched Xeon E-21xx, up to 6 cores.

How much do those cost?

 

[Chris Moore]

The cost of electric power usage range from 0.20 to 0.30 per kWh, so it needs to be taken into account.
Ice, your configuration is interesting: how is the system performance and its load ? And how much average power you measure at the wall ?

I just don't get this. I don't even know how much electricity costs and I never consider the cost of electricity when making equipment selection.
Even in the data-center at work, if we need more power, it just delays a project while the electrical contractor comes in to put in another line.

 

[Johnnie Black]

How much do those cost?

They replace the E3-12xx Xeons, so are similarly priced:

https://www.anandtech.com/show/12199/intel-launches-coffee-lake-xeon-e-entry

 

[Chris Moore]

They replace the E3-12xx Xeons, so are similarly priced:

https://www.anandtech.com/show/12199/intel-launches-coffee-lake-xeon-e-entry

I can't look the new processor up, Arc.Intel said it had no results for the part number, so I am stuck with just what Anand Tech is telling us.
The top price they list on that site says $450 for a 6 core processor but those processors are limited to only supporting 64GB of memory.
For $245, you can have a 6 core Xeon E5 that will support up to 768GB of memory:
https://www.ebay.com/itm/Lot-of-1-I...hz-Six-Core-25Mb-8GT-s-Processor/232846722737
Sure, the max turbo of the Intel Xeon E5-2643 v2 is only 3.8 GHz, but that is still a very capable processor:
https://ark.intel.com/products/75268/Intel-Xeon-Processor-E5-2643-v2-25M-Cache-3_50-GHz
Also, for FreeNAS, the speed of the processor isn't as much of a factor. The Xeon E5 also has quad channel memory instead of dual channel, and 25MB of internal cache instead of only 12MB so that might make for better performance.
I seriously don't see the reason to spend for the latest generation hardware. If it is down to trying to save on your electric bill, I think you would find that the difference in power consumption is so small that it would take you more than the life of the system to break even on the additional cost.

 

[Johnnie Black]

I can't look the new processor up, Arc.Intel said it had no results for the part number

https://ark.intel.com/products/series/134861/Intel-Xeon-E-Processor

 

[IceBoosteR]

The cost of electric power usage range from 0.20 to 0.30 per kWh, so it needs to be taken into account.
Ice, your configuration is interesting: how is the system performance and its load ? And how much average power you measure at the wall ?

The load is very small in "idle" (two VMs and one jail running) at around 5%. Performance is mostly more as enough, only when I do backups with GZIP(6) I do see 80-90% of CPU usage. But during the day its more as enough.
When I scrub my pool at around 1.4!GB/s CPU is at around 60%. Power from the wall depends on your hardware. When I have started in the stock T20 I have used around 45 watts (with 4 drives) in idle. With my current system I am
using 59 watts in idle (10 disks, 1 SSD, HBA and 6 fans). With disks in use it is around 10 watts more. In peak it is not using more than 130W from the wall.
Seriously I would like to sell my system, upgrade to have more RAM as I would like to have a ton more VMs on it. But normally I would not need to switch my hardware - it's more like an expensive hobby :D
What you would like to change is maybe an upgrade from 24 to 32GB RAM. But this thread is not a sales show, we can keep it to a PN if you are interested...
Back to the topic:

A system with E3-1220 v2 or v3 with mobo and 32 GB is starting around 800 Euro, with v4 you should add other 150 Euro. With used mobo and cpu, it's possibile to save around 2-300 Euro.
A good combination is E3 v6 + X11SSL-cF which have LSI 3008 integrated + DDR4 ECC UDIMM: as new it's about the same price... :)

I would suggest you will go to the 6 core option which is at the same price, but you may need to wait some weeks until its avaiable. But the option of CPU and board is nice.

I can understand that Avoton is becaming obsolete. Eight core Denverton is interesting for VM, I should check some reviuew but it should sustain 1Gbps transfer... because it is sold with 10Gb Ethernet too :):A2SDi-8C-HLN4F it starts at around 450-500 Euro then you have to add 180-220 Euro for a 16 GB kit: so with 32GB it shoud be got for around 900 Euro.

Yes. Did not know there is a 10GBe option avaialble. If so, these combination is looking good on the paper. I have seen one in the forum using a similar build and he is happy with it.

Going with E5 it's complicated: too many processors, with different versions and motherboard. If I look at v2 version (Ivy Bridge), then 8-core as e5-2650v2, which can be found used around 120-150 Euro, 32GB Reg for 160 Euro and mobo around ~150-250 Euro, so 430-560 Euro (used).

Yes it is. But if you do a deep dive it's easy^^

Xeon-D with 8 cores (~800 Euro) with 32 GB DDR4 will be priced around 1200 Euro.

I would not recommend the XEON-D due to the price.

I think there a quite some good options available for you. It's up to you what you would like to have (also think about the point, that this system may run the next 3 years).

Just fyi: https://forums.freenas.org/index.php?threads/hardware-upgrade-from-dell-t20.61473/#post-437101

You might also consider the recently launched Xeon E-21xx, up to 6 cores.

Thats what I have mentioned earlier :)

I seriously don't see the reason to spend for the latest generation hardware. If it is down to trying to save on your electric bill, I think you would find that the difference in power consumption is so small that it would take you more than the life of the system to break even on the additional cost.

I would say it depends. You are spending a lot of monery on a new system, or less money on used parts. Maybe you are one of those person who only trusts new hardware. On the other hand you will need to have a look how long you would like to own the system.
If you would like to use it for 5 years, think about 100€ per year more electricity costs. In 5 years this is 500€ where you can easily buy the new system. Its like a fridge. MAybe you are lukcy and prices in the US aren't as high as these in Europe.
But at the end of the day it is important to evaluate both sides. There are pros and cons on both...

-Ice

 

[sovking]

Regarding the E-2xxx launch, I think the most interesting cpu is E-2136, having 6 cores/12 threads without graphics, to put on Supermicro X11SCA-F. But both are just lanched, so no real avaibility: E-2136 is listed at 284$, while X11SCA-F in Europe could be found at over 360 Euro: so over 640 Euro + other 400 Euro for 32 GB of ECC DDR4 Unbuff... The only advantage I see over v5/v6 is the two more cores but it still miss hba; at that price we have to compare with E3-v5/v6 + X11SSL-cF with integrated SAS3 controller. Probably there is more than 200 Euro of difference... maybe not much for having the last generation and 6 core instead of 4. (AMD effect ? :D)
By the way, on Epyc side: Supermicro H11SSL-C + Epyc 7251 start at 840 Euro with SAS3008 included, then you have to add ddr4 memory: for 32 GB you will be under Xeon-D 1541 price. I agree with Ice, Xeon-D now is overpriced.

It's very interesting the data offered by IceBoosteR about its E3 configuration: with 10 disks, 1 SSD, HBA and 6 fans the idle power usage is less than 60 W, problably that usage is for 90% of the time. Ice are you sure to have measured well ? ;) 10 Spinning disk even at idle should consume at least 40 W, some W for SSD, 10 W for HBA, and 22W for the CPU not counting IPMI fans and other minor consumption... I would expect around 80 W...

Regarding the owning time of this system: I would like to own, until it suffices my need, 5 years minimum maybe longer by upgrading some parts (disk subsystem and memory mainly): it will be not a computing machine, it is mainly a storage one, bottlenecked on Internet side for cloud operations. Now I'll use mainly hard-disk and a couple of SSD, but I could increase SSD/NVME storage if I will find some deal.
Building a 75W system, instead of 175W one, will save 100Wx24x365 = 876 kWh per year at an average of 0.25 Euro are 219 Euro saved per year, over 1000 Euro in 5 years: you are right a new system!
So because this will be an home system I am leaning towards building a system that hopefully will not exceed 75-80 W with 6 low power hard disk and a couple of SSD. The 836 chassis should be only quite large and future proof to let to be updated over the time.

 

[IceBoosteR]

Hi,
yeah the AMD effect is everywhere - and I like it.

It's very interesting the data offered by IceBoosteR about its E3 configuration: with 10 disks, 1 SSD, HBA and 6 fans the idle power usage is less than 60 W, problably that usage is for 90% of the time. Ice are you sure to have measured well ? ;) 10 Spinning disk even at idle should consume at least 40 W, some W for SSD, 10 W for HBA, and 22W for the CPU not counting IPMI fans and other minor consumption... I would expect around 80 W...

No I am sure this is correct what I have written. These numbers comes directly from my UPS, maybe there is a difference in 1 or 2%, but I don't supect these are incorrect. The WD RED are using 3,6W in idle, which would cause the system without anything else as mobo, CPU and RAM at around 30W. Adding a Platinum PSU and the disks as well as the fans (running at 30%) are causing the 59w, with 100% fan speed it is 66W. Now with load on the disks it is 10W more.
And yes ou are right, my system is using these idle load for about 90% of the time. So efficency is key.

When you would like to use your system over the next 5 year I would agree you choose a new system, or move to the cloud :D
Jokes aside, with 6 disks and a Kaby Lake system you may see around 50 watts on idle.
My i7 6700K (Skylake) has also a quite low idle power consumption, but I do not have numbers for you. But I guess you will find some numbers on the internet with equivalent comsumer grade hardware.
I would personally go for the E-2136 (Coffee Lake) version, you will never know if you would like to have more horsepower in the upcoming years...
Its up to you, but a v6 i nothing bad^^

-Ice

 

[sovking]

I just don't get this. I don't even know how much electricity costs and I never consider the cost of electricity when making equipment selection. Even in the data-center at work, if we need more power, it just delays a project while the electrical contractor comes in to put in another line.

For me, in the data-center at work is the same. At home is different for several reasons, starting from Wife Acceptance Factor ;)... electricity bill cannot ramp too much!

Regarding the X9 configuration proposal:
- motherboard Supermicro X9SRL-F (good expandibility!): I don't find any used interesting one, but I can get new for 320 Euro.
- cpu E5-2650 v2 (8 core/16 thread): now used is around 100 Euro in EU, then better/worse deal can found.
- heatsink: around 30-60 Euro (from passive to noctua)
- memory 4x8GB DDR3 ECC Reg: around 100 used or 160 new
- hba: around 50 Euro for used SAS2, 150 Euro for sas3 (probably we have to add the cable)

So total should range 600 to 800 Euro, less if you can find the mobo used at good price, and this configuration has good expandibility both for devices and memory...
I don't know figures of average power comsuption of this configuration for the proposed scenario, probably it will consume some tens watt more than IceBoosterR configuration, because of cpu prior generation, cpu with double cores even at lower frequencies, cpu with more features, and more complex mobo.

However just now I could restrict the choice between E3-v5/v6 and E5-v2 with x9SRL :)

 

[Chris Moore]

In 5 years this is 500€ where you can easily buy the new system.

I built a system recently where we saved about that much because the components, especially RAM, were so much less expensive going with the older generation hardware. That is why I say you would spend the same over the life of the hardware to get the old hardware vs getting the new. I would say that there are other places to economize on power consumption.
If you spend a little more on power each month for 5 years or spend a lot more on hardware NOW, which hurts more.

 

[Johnnie Black]

Thats what I have mentioned earlier :)

Yep, my bad, missed that.