Specific build changes to upgrade as high as 512GB of RAM

[Chris Moore]

Chris Moore submitted a new resource:

Specific build changes to upgrade as high as 512GB of RAM - Better FreeNAS

This is intended as a modification to the list of hardware I put together here:
https://www.ixsystems.com/community/resources/specific-build-components-list-up-to-32gb-ram.109/

Everything else can stay the same and the function doesn't change, but if you change the system board, processor and memory from what they were to this:


This board uses Intel C602 chipset and can accomodate up to 512GB of RAM, I did a couple builds with these:

System Board: Supermicro X9SRL-F...

Read more about this resource...

 

[racielrod]

Hi @Chris Moore,
In terms of power consumption, how much power these 2 setups would draw with a 4-6 HDD setup?

https://www.ixsystems.com/community/resources/specific-build-components-list-up-to-32gb-ram.109/
vs
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/

Thanks,
R. Rod

 

[Chris Moore]

In terms of power consumption, how much power these 2 setups would draw with a 4-6 HDD setup?

https://www.ixsystems.com/community/resources/specific-build-components-list-up-to-32gb-ram.109/

Honestly, I had 12 drives in it, so I can't say with certainty. It was drawing around 120 watts at idle with 12 drives, so I would guess it to be around 80 to 90 watts with only 6 drives, but that depends a lot on what drives you are using.

vs
https://www.ixsystems.com/community...anges-to-upgrade-as-high-as-512gb-of-ram.110/

When I upgraded the system board, I made several other changes including adding more drives. The system is drawing around 300 watts now, but I have mine in a 48 bay rack chassis with thirty drives in addition to two SAS expander backplanes and 64GB of memory. It is tough to say how much of the power is the system board, processor and memory and how much is the other. Best guess, the more sophisticated board and CPU are going to draw an extra 30 watts.

These builds are geared more to performance than to low power consumption. I wanted to be able to run my Plex and transcode video and also be able to run some virtual machines.

 

[racielrod]

Thanks!
I have been looking at different builds for a long time and what keeps me from pulling the trigger is really finding the right hardware for what I need from FreeNAS.
Initially I'm only planning to use FreeNAS for storage, so looking for a power efficient build makes sense - not planning to run Plex or VMs on it.
BUT, is there a power efficient build that has some room for growth to support a small VM?

Thanks again!
R. Rod

 

[Chris Moore]

Thanks!
I have been looking at different builds for a long time and what keeps me from pulling the trigger is really finding the right hardware for what I need from FreeNAS.
Initially I'm only planning to use FreeNAS for storage, so looking for a power efficient build makes sense - not planning to run Plex or VMs on it.
BUT, is there a power efficient build that has some room for growth to support a small VM?

Thanks again!
R. Rod

The 32GB build could be done with a lower power CPU to save a little power, but the big power draw is the hard drives. General rule is to figure 10 watts running and 15 watts at startup, per drive, on top of what you figure the rest of the system will take.
Have you looked at this:
https://www.ixsystems.com/community/threads/proper-power-supply-sizing-guidance.38811/

 

[Tertius Carstens]

So I built your first suggestion for my offsite server and it's working well, still need to get the old server from offsite to add old drives to already installed drives.

At idle it's doing 70watt.

What would that look like for this suggested hardware? I'm thinking about building this to replace a Synology 916+ locally.

 

[Chris Moore]

So I built your first suggestion for my offsite server and it's working well,

I am happy it is working well for you, but which one are you calling the first one, the one with the X9SCM-F system board?

What would that look like for this suggested hardware?

If you are asking about the difference between the X9SCM-F system and the X9SRL-F system, I can only give my best estimate of the difference because the system board and memory would draw a different amount of power in addition to the CPU itself. I would guess the difference to be a little as 30 or as much as 60 watts, but when I made that change myself, I also made other changes that significantly impacted the overall power draw.
Purely guessing, I would expect the total to run around 100 watts but it could be as much as 130, but it depends on the exact hardware you end up using.

 

[Tertius Carstens]

I am happy it is working well for you, but which one are you calling the first one, the one with the X9SCM-F system board?

Yes, thank you very much!

If you are asking about the difference between the X9SCM-F system and the X9SRL-F system, I can only give my best estimate of the difference because the system board and memory would draw a different amount of power in addition to the CPU itself. I would guess the difference to be a little as 30 or as much as 60 watts, but when I made that change myself, I also made other changes that significantly impacted the overall power draw.
Purely guessing, I would expect the total to run around 100 watts but it could be as much as 130, but it depends on the exact hardware you end up using.

Okay, let me get specific.

Change X9SCM-F to X9SRL-F.
Change Intel Xeon E3-1230V2 to Xeon E5-2650 V2.
Change 8GBx2 to 16GBx1 (for now).

Let's assume everything else stays the same.

 

[Chris Moore]

Change X9SCM-F to X9SRL-F.

I don't have numbers for the difference in power draw between the system boards, but I would guess the X9SRL-F draws a little more because it has a little more hardware. I would guess the difference is less than 10 watts.

Change Intel Xeon E3-1230V2 to Xeon E5-2650 V2.

From the available documentation, the E3-1230 v2 can draw as much as 94 watts, but averages 69 watts, but can draw as little as 34 watts at idle.
From the available documentation, the E5-2650 v2 has a rated TDP of 95 watts, and I am having a difficult time finding data other than that. That should mean that 95 watts is the maximum but it will draw less at idle.

Change 8GBx2 to 16GBx1 (for now).

Memory should be pretty equivalent, stick for stick.
Without having them in front of me to put a tester on, I can't get more accurate than that. The system I built with the X9SRL-F system board has 30 x 3.5" hard drives an NVMe PCI drive and two 2.5" drives for the boot pool in addition to the two 24 slot SAS expander bakplanes and the SAS controller. All that draws around 324 watts for me.

 

[Tertius Carstens]

Question: Why not a GA-7PESH2?

Maybe E-ATX makes getting cases really expensive?

 

[Chris Moore]

Question: Why not a GA-7PESH2?

Part of the reason is that there is not a lot of reason for having a dual socket system in a home NAS. The vast majority of users are just fine with a single processor. There are instances when a dual processor board is less expensive on the used market because of a higher supply vs demand, but that is inconsistent.

Maybe E-ATX makes getting cases really expensive?

This is also a large factor. A standard ATX size system board can fit in almost any old (or new) case you can round up. I listed a case in the parts list, but if you have one already that you like, there are a large variety of options. If you go to a dual socket system board, regardless of vendor, they are almost all larger in ways that make fitting them in a case more challenging. For me, I have both of my FreeNAS builds in full size, 4U rack-mount chassis such that I have almost unlimited options and I still choose the single socket board because there is (for me) no need for the additional processing cores that a second CPU would provide.

This is not to say that there is no reason for it, ever, in all cases. I have a system at work where there are active times that the CPU load (even with two physical processors) spikes to 80 or 90 %; that is not all the time and not that often, but we need the resources for when it does happen.

Choosing the right hardware for your build depends a lot on what you plan to do with it.

 

[Chris Moore]

PS. Operational cost for a dual socket system board are more even if you only populate one socket, but if you have both sockets populated, it doubles the amount of power for the CPU which is often the largest single power draw in the system, other than the drives. If I recall correctly, the CPU in my system is rated for a max TDP around 95 watts and if I had two of them, that would be double. Extra heat and power consumption for very little practical benefit, unless you are doing a lot of virtualization.

 

[Tertius Carstens]

Okay, does having dual-10Gbe change the cost benefit if you're just using one socket?

P.S. One of the deployments I'm doing will be doing some virtualization (but really won't need dual socket performance) and I won't pay for power. But don't let that influence the question above. Maybe just a separate comment.

 

[Chris Moore]

Okay, does having dual-10Gbe change the cost benefit if you're just using one socket?

It does change some, but not a massive amount, and it mostly depends on if the physical size of the board is a non-issue.

Here is how I thought through it... I can get a pair of very good 10Gb cards for about $30, so call that $15:
https://www.ebay.com/itm/Mellanox-M...10G-Ethernet-10GbE-SFP-PCI-E-NIC/223088957450

In fact, that is such a good deal, I am tempted to buy those to replace some Mellanox ConnectX 2 cards I am currently using.

I can also buy a SAS controller that I would prefer to have for around $40 so I am not entirely sure I would want it built into the board:
https://www.ebay.com/itm/660088-001...8I-PCI-E-HBA-HOST-BUS-ADAPTER-US/132757577271

Then there is the board I use for my server, Supermicro X9SRL-F, and that can be had for only $154.99/ea:
https://www.ebay.com/itm/Supermicro...GA2011-System-Board-BIOS-Updated/401678513337

So if I wanted to buy the X9SRL-F, Mellanox NIC and HP H220 SAS HBA; it would run about $210.

The Gigabyte GA-7PESH2 you suggested is listed on eBay for $189.99 with heat-syncs included that may need to be replaced, so not necessarily a value add:
https://www.ebay.com/itm/GIGABYTE-G...RVER-MOTHERBOARD-LSI2008-2X10GBE/333109137334
Looking at the spec on the Gigabyte site: https://www.gigabyte.com/Server-Motherboard/GA-7PESH2-rev-10#sp
we can see that it includes a pair of 10GbE BASE-T LAN ports using the Intel® X540-AT2 chipset, so that is good, and it also includes two Mini-SAS (for 8 x SAS 6Gb/s ports) using LSI® SAS2008, so I am sure that can be flashed with the IT mode firmware.

Just looking at raw cost, the Gigabyte board wins, but you also have an E-ATX board that will not fit in some cases, where the X9SRL-F is a standard ATX board and can be used in many more cases. So, around about $20 extra for a greater degree of flexibility.

You must choose based on your needs though, so if the lower initial cost of the Gigabyte board is your goal, and the extra size of the board doesn't matter, it might be the solution for you. Also, the 10Gb NIC that I pointed to is using SFP+ instead of BASE-T (RJ-45) LAN ports, so that could influence your choice, but I would prefer to have SFP+ for the lower latency.

 

[Vegastrey]

It's worth noting for those of us like me not paying total attention that the socket type for this motherboard processor combo is different than the previous build you posted Monday. I didn't realize that and ordered the cpu fan and all of the other necessary components from the initial build. I'm so far pretty happy with the build. Now I just need to find a LGA2011 fan.

 

[Chris Moore]

It's worth noting for those of us like me not paying total attention that the socket type for this motherboard processor combo is different than the previous build you posted Monday. I didn't realize that and ordered the cpu fan and all of the other necessary components from the initial build. I'm so far pretty happy with the build. Now I just need to find a LGA2011 fan.

Sorry about that. An oversight on my part. I updated the info and this is the one I use:
Dynatron R27 3U CPU Cooler Fan for Intel Socket Narrow 2011 Intel Xeon Series
https://www.ebay.com/itm/321375083251
Price: US $33.90

 

[Backs1de]

Hi @Chris Moore and thanks for making these guides, they have been super helpful on determining what i need to get this project going. I'm in the process of gathering info for building a system very similar to this one but i have a Fractal Design Node 804 which i want to use, what would be a similar build that would fit in a mATX case for around the same budget or a little more.

I was really leaning towards the E5-2650v2 or similar CPU due to their passmark score and lower price but i am now limited to a mATX board, is there one that would work with this cpu? or which combinations of cpu/mobo would you recommend to stay around the same passmark as a 2650v2


Thanks!

 

[pro lamer]

mATX

Did you mean uATX? I'd love one, too... I couldn't find any and unless I missed sth important I guess there are no such... :(

Sent from my phone

 

[Backs1de]

mATX and uATX are the same thing I believe

 

[pro lamer]

Now I can recall I was able to find some X79 chipset based mobos using pcpartpicker but:
1. IIRC they might have been limited to E5-16xx v1/v2 CPUs only (not 26xx) they even work with some E5-46xx...
2. I couldn't find them in my area and was not ready to buy overseas so I gave up and forgot the details such as DIMM slots count (I checked yesterday: 4 slots), LRdimms support, (only UDIMMs so RAM limit seems to be 64GB) exact form factor (mITX or checked: microATX)
3. Update: Haven't paid attention whether they are server or consumer ones.

BTW mini ATX is not microATX and mini ATX is not common and it used to mean two different form factors anyway...

Sent from my phone