suggestions for a low power system to replace that old dual xeon macpro

[Constantin]

Depends in part on the cost of power, peak demand pricing, cooling considerations, and so on.

I needed a new server because I lost faith in AsRock after my second replacement failed. With 8 spinning disks, a SLOG, and a L2ARC, about 90W of power consumption is decent.

 

[jgreco]

One last thought; If you are looking for lower power and good performance, consider CPUs with an "L" at the end. The 2630L V2 (6 cores) or better yet the 2648L V2 (10 cores) offers quite a bit of performance at less the 75w per CPU. Pair that with a SuperMicro board like an X9SRI-F and you get the power savings you want and the cores you need for virtualization at a lower cost.

No, this is just wrong.

The L CPU's are TDP-limited parts that accomplish this by running at a reduced clock rate. To do a fixed unit of work, a normal CPU operates at a much higher speed and generates more heat per unit time, but does it in a short period of time. The L CPU runs at a lower speed for a much longer period of time. It puts out about the same amount of heat, but because it is spread out over a longer time, the heat dissipation requirements are greatly reduced. This is very important in environments where maybe you don't have active cooling (just a huge heatsink) or you're in a constrained environment (1U chassis etc).

These are stupid parts to use in applications where you can get rid of the heat. They're more expensive and they run slower. What you really want is when work comes along, you want the CPU to tackle that workload as quickly as possible and return to idle as quickly as possible. This allows the CPU to use things like turbo boost much more effectively, and in many cases this actually turns out to be more energy efficient for the same workload.

So if you want lower power and good performance, YOU DO NOT USE AN L PART.

 

[hotdog]

maybe a few hints to power consumption:

try to avoid putting more stuff into your machine.

E.G. better to find a cpu and a board which has sufficient amount of sata ports (or whatever you want) Instead of adding HBAs later on.
Or better to have 1x 16gb ram instead of 4 times 4 gb
unbuffered ECC RAM should use less power than reg ECC

that might also apply to 10 gbit nics. there are supermicro boards available with 10 gbit nics. But I have no experience with that (if it works good, reliable, fast etc)

Discs with helium usually run cooler and use less power

lower RPM for spinning HDDs is also beneficial to safe power.

in general; there is a break even point, between saving power and pay higher prices for the components.
Since the price for energy in switzerland is low (compared to other European countries) i would not go for very expensive products in order so safe a few watts.

i felt very comfortable using ssds for the jails and the logs. in order to safe power, i set the HDD drives to keep spinning but the head is in parking position (should be lowest power state without spindown) if you have the jails and log on the HDDs there are continuous writes, so in essence they will not safe energy (that I experienced in my system).

tnx John for your hints.

 

[hotdog]

I am trying to understand why such a focus on CPU power usage for your application. Usually CPU power consumption is a big deal in a data center where there are dozens or even hundreds of servers running. The cost of power and cooling is significant such a high density environment and companies are willing to pay a premium to get similar computes with less power. In an office or home environment with a small number of servers it can be difficult to justify the cost savings of a low power/high performance CPU. Consider the following options:

Dual Intel L5640 CPUs vs one E5-2630-V2 vs one E3-1230-V6. Each of these CPU solutions provide similar performance on the PassMark tests. The annual power costs (using CA residential power rates, 24/7 usage and 25% avg. CPU usage) are as follows: Dual L5640s: $46.32; 2630 V2: $26.88; 1230 V6 $24.29.

The good news is the 1230V6 (assuming 24/7 usage) will save about $22 a year over the dual L5640s and about $1.50 annually over the 2630 V2. There will potentially also be savings in cooling costs as well. The challenge is the cost of newer CPUs, memory and motherboards, make the older hardware a better value. For example (on eBay) a pair of L5640s will run less than $20. A single 2630 V2 will cost less than $25. A single E3-1270 V6 starts at about $220. Newer CPUs are on a steep price curve compared to older CPUs.

One last thought; If you are looking for lower power and good performance, consider CPUs with an "L" at the end. The 2630L V2 (6 cores) or better yet the 2648L V2 (10 cores) offers quite a bit of performance at less the 75w per CPU. Pair that with a SuperMicro board like an X9SRI-F and you get the power savings you want and the cores you need for virtualization at a lower cost.

Not sure where you got your number from, in Zurich where I live I just calculated:
A 24/7 running 200W Energy source (which the Mac Pro is I guess) will cost me about 350 CHF / Year ($ to CHF is about 1:1)
My intention here is to build a machine which draws around 50W or - better - less energy.
I just don't feel comfortable having something running all the time and consuming a fair amount of energy when not really needed.
5 years ago someone on this forum - when I asked something about a logged error - said something like his new hardware would payoff in less than a year.
I don't see it all that way - maybe from his point of view the financial payoff is there, but IMO you should consider much more in the calculations and consequences when replacing a running system. things like how much energy was invested in a part at the moment it is laying on your work bench, how much energy will it cost to be cleanly brought to end of life or could it be recycled etc. etc.
Don't get me wrong, I am not just focusing on get the most efficent system here, I just wanted your inputs/thoughts on this matter.
I don't even know for sure if I need a server standing in my room, I could pay for the service, Zurich has built a very affordable fiber net infrastructure we are using, you don't even feel the remote desktoping lag anymore.
I just like the fact, that in case everthing goes down, there is a box physically standing around which holds our stuff (which then will probably not be that important:) ).

 

[Constantin]

50W might be difficult unless you go with 2.5" spinners and a even lower-power CPU. My system has eight 3.5" drives, each of which pulls about 7W => 56W right there. The PSU conversion efficiency is around 90%, so at the plug that gets you to 61W. I'd consult famerplings excellent hard drive resource page to see if switching to 2.5" spinners helps a lot or not. However, I would also expect some reduction in performance, 2.5" drives aren't typically built for speed.

[EDIT] These days be VERY careful re: any 2.5" spinning drives you may be contemplating in a NAS. Anything in that form factor is likely going to be using Shingled Magnetic Recording (SMR) technology (such as the current WD RED 2.5 allegedly-NAS drive). It's a software tweak that packs in an extra 20% more data per platter, though with severe performance penalties, especially if the drive is a Device-Managed SMR (DM-SMR) drive, which is the only kind of SMR drive currently being sold into the consumer market.

DM-SMR drives have no business in a NAS unless you are OK with severe performance problems, drives getting knocked out of the pool because they've become non-responsive (for example due to a CMR-SMR cache flush) and like issues. The other types of SMR drives (HM-SMR and HA-SMR) have yet to get any support in ZFS. Thus, all SMR drives are currently fundamentally unsuitable for ZFS pool use (they may make an OK, albeit very slow, external backup drive). The lack of HM- and HA-SMR support in ZFS may change in the future, just as the availability of HA- and HM-SMR drives in the consumer retail channels. In the meantime, double-and-triple-check that your drives are not SMR! [End of EDIT]

Thing is, whether you do the storage on site or in the cloud, the energy is getting consumed somewhere. The cloud may be more efficient by aggregating lots of data across a large shared resource but I doubt the delta is that great once you consider everything else that comes along with cloud infrastructure (data transmission infrastructure, and so on). Plus, I prefer having the data physically there, it allows me to run scrubs, make local backups with ease (and speed!), and so on.

 

[jgreco]

50W might be difficult

He said "50W [...] less". Which is trivial to do when you start with a big piggy system. Getting down to 50W only, as you note, is much more complicated.

 

[hotdog]

He said "50W [...] less". Which is trivial to do when you start with a big piggy system. Getting down to 50W only, as you note, is much more complicated.

haha edited my post - just threw in a number, but was really meant absolute.. 70W would be ok also.

Just found a blog entry where 42W were mesured over two days, Atom powered, but 8 x 2.5 inch drives.

DIY NAS: 2019 Edition

A DIY NAS featuring the SuperMicro A2SDI-4C-HLN4f motherboard, a Intel Atom C3558 CPU, 16GB DDR4 ECC RAM, 8x2.5" 2TB HDDs, and the SilverStone Technology CS280B case.

blog.briancmoses.com

With a 6 drive config, I would really like to go with a 2.5 inch solution, but can't find 4TB drives that are really meant as NAS drives. The Barracudas are labeled as Laptop Drives. The better write performance from the 2 VDEVs would make up for the overall slower 2.5 disks.

 

[Constantin]

I’d be surprised if a NAS case with largely dormant data isn’t easier on laptop drives than use in a laptop. :-D

 

[hotdog]

I’d be surprised if a NAS case with largely dormant data isn’t easier on laptop drives than use in a laptop. :-D

Good point :)

 

[John Doe]

my backup system take 69 watts, just measured it recently. Specs in signature.

it is possible to go lower. had an Asrock Q1900ITX board with onboard CPU and 4 drives. Idle was around 31 watts.

Freenas performed somehow okay with lots of playing around with settings, but network speed was only around 80mb/s

 

[joeinaz]

No, this is just wrong.

The L CPU's are TDP-limited parts that accomplish this by running at a reduced clock rate. To do a fixed unit of work, a normal CPU operates at a much higher speed and generates more heat per unit time, but does it in a short period of time. The L CPU runs at a lower speed for a much longer period of time. It puts out about the same amount of heat, but because it is spread out over a longer time, the heat dissipation requirements are greatly reduced. This is very important in environments where maybe you don't have active cooling (just a huge heatsink) or you're in a constrained environment (1U chassis etc).

These are stupid parts to use in applications where you can get rid of the heat. They're more expensive and they run slower. What you really want is when work comes along, you want the CPU to tackle that workload as quickly as possible and return to idle as quickly as possible. This allows the CPU to use things like turbo boost much more effectively, and in many cases this actually turns out to be more energy efficient for the same workload.

So if you want lower power and good performance, YOU DO NOT USE AN L PART.

1. It depends on the application; specifically the ability to drive utilization. Virtualization is an example of an application which can do this.
2. As an example, the 2630L V4 (vs a 2630 V4) provides 92% of the computes while using about 65% of the power.
3. It turns out for this customer, power consumption is everything; in their case, even the 60w 2630L V2 may be too "power hungry"..

 

[jgreco]

2. As an example, the 2630L V4 (vs a 2630 V4) provides 92% of the computes while using about 65% of the power.

Cite? (Not some backwater reddit, either). Intel has explained it as I described in their engineering application notes for low TDP processors, but I'm not interested enough to go chase that down. The gist is that contemporary low TDP and the regular parts are similar at idle and the difference is primarily in how clocking and power management interact with the workload. The design intention isn't to be more power efficient, but to control thermal dissipation issues in hardware design.

 

[joeinaz]

Not sure where you got your number from, in Zurich where I live I just calculated:
A 24/7 running 200W Energy source (which the Mac Pro is I guess) will cost me about 350 CHF / Year ($ to CHF is about 1:1)
My intention here is to build a machine which draws around 50W or - better - less energy.
I just don't feel comfortable having something running all the time and consuming a fair amount of energy when not really needed.
5 years ago someone on this forum - when I asked something about a logged error - said something like his new hardware would payoff in less than a year.
I don't see it all that way - maybe from his point of view the financial payoff is there, but IMO you should consider much more in the calculations and consequences when replacing a running system. things like how much energy was invested in a part at the moment it is laying on your work bench, how much energy will it cost to be cleanly brought to end of life or could it be recycled etc. etc.
Don't get me wrong, I am not just focusing on get the most efficent system here, I just wanted your inputs/thoughts on this matter.
I don't even know for sure if I need a server standing in my room, I could pay for the service, Zurich has built a very affordable fiber net infrastructure we are using, you don't even feel the remote desktoping lag any more.
I just like the fact, that in case everthing goes down, there is a box physically standing around which holds our stuff (which then will probably not be that important:) ).

Having looked closer at the specs of the old Mac "Eight Core" I now see where you are coming from. Two X5482 may need 300w for just the CPUs. By the way, I referenced the site www.cpubenchmark.net for performance and power ratings of just the CPUs. I adjusted the power costs for systems located in California in the US.

You stated you are looking for 50w for a complete solution with the ability to do virtualization. Consider the following:

• Intel 1230L V3; this gives you 4 core and over 7000 on the PassMark scale using only (25w) of power
• SuperMicro Socket 1150 motherboard; (25-40w)
• ECC Memory (4) (10w)
• A quantity (4) of 2.5" SSDs for storage; (10w)
• Boot from USB (to save power!)

In this configuration, you could have 1 perhaps 2 VMs and FreeNAS running. Good news is this entire configuration would use less power than one of the two CPUs in your old MacPro. The reality is even with a 25w CPU; the total power usage of the system (at load) could be over 70w.

The options are to drop your requirements for virtualization or change your requirements to limit just the CPU. for example, no more than 75w maximum power draw for the CPU. This would open many more possibilities.

 

[Constantin]

FWIW, I would not boot from a USB stick for a production server. Use something built for the purpose, i.e. a SSD or a SATADOM. I've read too many stories of woe here related to failed USB sticks to ever want to use one, even in a mirrored boot pool, on a production server. Using SSDs for the data pool is also pretty prohibitive at this point unless the work flow justifies it. If I needed low power and was OK with the related performance tradeoff, I'd stick to the 2.5" spinners the OP mentioned.

That said, the random write performance of that 2019 DIY NAS linked to above is nothing to be particularly happy about. Even with a 2-VDEV system, you'd be looking at maybe 60MB/s per that test. I wonder how applicable Brians testing/setup is to the OP use case.

 

[joeinaz]

Cite? (Not some backwater reddit, either). Intel has explained it as I described in their engineering application notes for low TDP processors, but I'm not interested enough to go chase that down. The gist is that contemporary low TDP and the regular parts are similar at idle and the difference is primarily in how clocking and power management interact with the workload. The design intention isn't to be more power efficient, but to control thermal dissipation issues in hardware design.

Intel Xeon E5-2630 v4 @ 2.20GHz vs Intel Xeon E5-2630L v4 @ 1.80GHz [cpubenchmark.net] by PassMark Software

 

[joeinaz]

Cite? (Not some backwater reddit, either). Intel has explained it as I described in their engineering application notes for low TDP processors, but I'm not interested enough to go chase that down. The gist is that contemporary low TDP and the regular parts are similar at idle and the difference is primarily in how clocking and power management interact with the workload. The design intention isn't to be more power efficient, but to control thermal dissipation issues in hardware design.

Also:

https://www.spec.org/cpu2006/results/cpu2006.html

The CPU 2006 results are used because the E52630 V4 CPUs were not shown in the CPU2017 tests. The spec.org results were similar in showing the 2630L with about 90% of the performance of the 2630. Both the PassMark and spec.org sites are highly respected resources that have been used for years in comparing various systems.

 

[joeinaz]

FWIW, I would not boot from a USB stick for a production server. Use something built for the purpose, i.e. a SSD or a SATADOM. I've read too many stories of woe here related to failed USB sticks to ever want to use one, even in a mirrored boot pool, on a production server. Using SSDs for the data pool is also pretty prohibitive at this point unless the work flow justifies it. If I needed low power and was OK with the related performance tradeoff, I'd stick to the 2.5" spinners the OP mentioned.

While I was trying to stay within a 50w power requirement, I agree mirrored USB is better (more power needed), and SATA DOM is a better choice. As for the disk, I agree there is only a small difference in the power usage of the 2.5" spinning disk and the SSD but if virtualization is done the SSDs will give a much better customer experience. Another option would be to have the SSDs available for VMs and the spinning disks for everything else.

 

[jgreco]

Also:

https://www.spec.org/cpu2006/results/cpu2006.html

The CPU 2006 results are used because the E52630 V4 CPUs were not shown in the CPU2017 tests. The spec.org results were similar in showing the 2630L with about 90% of the performance of the 2630. Both the PassMark and spec.org sites are highly respected resources that have been used for years in comparing various systems.

Ahhh! Okay. I see what you're thinking, and now I can explain your error to you in terms you should understand.

You are comparing CPU benchmarks to each other without understanding what they mean in the context of this discussion.

We are talking about energy burned per unit work.

You have taken some benchmarking numbers and used them without measuring energy burned. Instead, you have assumed a fallacy, which is that a CPU will burn its TDP-rated watts when it is busy running a useful workload. This is simply false. You actually need to measure the energy, not just guess.

See, if you have a 135W TDP CPU, just as an example, that 135W is a theoretical maximum that assumes a bunch of things that are not necessarily true for benchmarks. Benchmarks are designed to test useful and meaningful loads that the CS world has decided are important measures. If you load up a CPU with a given kind of test, let's say a floating point test, there are still significant sections of the CPU that are not doing work, and are not burning watts. If the CPU has an integrated GPU, that's not burning watts. If the CPU has integrated I/O, that's not burning watts. So the problem is that not everything is running during your benchmark, meaning that the CPU is burning less watts than you are thinking it is.

TDP is an engineering value for those of us who actually build servers in order to understand the maximum possible watt-burn a CPU might possibly need to dissipate. Failure to engineer for a sufficient TDP means a server is possibly in a situation where the CPU could overheat. It is not a "when the CPU is running at 100% on a workload it will be taking this many watts" number. Because of that last sentence, your whole argument about CPU efficiency is false.

So what you actually need to cite is a controlled environment where the same workloads are running on the two different CPUs while energy consumption is being monitored. In such an environment, you should find that -- given the same generation silicon and all the other expected qualifiers -- the L CPU's tend to be a little less efficient on watt burn per unit work. Intel's got some application note about it somewhere.

 

[joeinaz]

Ahhh! Okay. I see what you're thinking, and now I can explain your error to you in terms you should understand.

You are comparing CPU benchmarks to each other without understanding what they mean in the context of this discussion.

We are talking about energy burned per unit work.

You have taken some benchmarking numbers and used them without measuring energy burned. Instead, you have assumed a fallacy, which is that a CPU will burn its TDP-rated watts when it is busy running a useful workload. This is simply false. You actually need to measure the energy, not just guess.

See, if you have a 135W TDP CPU, just as an example, that 135W is a theoretical maximum that assumes a bunch of things that are not necessarily true for benchmarks. Benchmarks are designed to test useful and meaningful loads that the CS world has decided are important measures. If you load up a CPU with a given kind of test, let's say a floating point test, there are still significant sections of the CPU that are not doing work, and are not burning watts. If the CPU has an integrated GPU, that's not burning watts. If the CPU has integrated I/O, that's not burning watts. So the problem is that not everything is running during your benchmark, meaning that the CPU is burning less watts than you are thinking it is.

TDP is an engineering value for those of us who actually build servers in order to understand the maximum possible watt-burn a CPU might possibly need to dissipate. Failure to engineer for a sufficient TDP means a server is possibly in a situation where the CPU could overheat. It is not a "when the CPU is running at 100% on a workload it will be taking this many watts" number. Because of that last sentence, your whole argument about CPU efficiency is false.

So what you actually need to cite is a controlled environment where the same workloads are running on the two different CPUs while energy consumption is being monitored. In such an environment, you should find that -- given the same generation silicon and all the other expected qualifiers -- the L CPU's tend to be a little less efficient on watt burn per unit work. Intel's got some application note about it somewhere.

"So what you actually need to cite is a controlled environment where the same workloads are running on the two different CPUs while energy consumption is being monitored."

That is not how the world works. Industry benchmarks measure how much work was done when CPU was stressed with a given test. You can then claim a certain amount of work was obtained with a CPU that has a maximum TDP of x.

1. No overclocked results were presented.
2. The 2630 V4 and 2630L V4 are of the same CPU family and can run on the same motherboard.
3. The maximum TDP of both CPUs is known (2630 = 85w, 2630L = 55w).
4. The tests are designed to get maximum performance out of each CPU.
5. The tests which included both integer and floating point routines showed the 2630L to produce 90% of the performance of the 2630.

The bottom line: Two different sources show the 2630L V4 produced over 90% of the work with a 64% lower TDP compared to a 2630 V4.

"In such an environment, you should find that -- given the same generation silicon and all the other expected qualifiers -- the L CPU's tend to be a little less efficient on watt burn per unit work. Intel's got some application note about it somewhere."

Ok, I will ask you the same thing you asked me: cite;

 

[Jessep]

Is Intel enough of a source for you?

https://www.intel.com.au/content/www/au/en/support/articles/000031072/processors/intel-core-processors.html
Thermal Design Power (TDP) is the maximum amount of heat that a processor can produce when running real life applications. It is used mostly to match up processors with an adequate heat sink that is capable of cooling down that processor effectively.

TDP is NOT performance per watt, which is what you are looking for to compare.

In the end if you think the experts here are wrong, more power to you.

Buy what you want. Being wrong doesn't mean it wont function, only that it will have a higher power usage per work done.

"L" CPUs are for lower TDP cases/enclosures, not that they use less actual power per work. They tend to be higher in price due Intel not making as much volume of these parts.