SOLVED Upgrading system, question about idle power usage

[keboose]

I currently have a setup with four AMD Opteron 6128 processors, and twelve 1TB WD Red hard drives. I have such a processor setup for a few reasons: I planned for multiple Plex transcodes (at least 4 1080P streams,) They encode videos with Handbrake WAY faster than my personal PC, and I got a really good deal (about $300 for the board, all four processors, and some RAM.)

I recently started logging my power usage using the features on my APC UPS, and found that the server is using more than double the amount of electricity I predicted when I first built it, and totals nearly 30 USD per month in electricity. According to the data, my server idles around 260 watts the majority of the time. I did my best to minimize drive usage (moved my data collection from my NAS share to a folder on my boot drives,) but the Plex and owncloud plugins I have running seem to keep the drives on pretty much all the time.

This has prompted me to upgrade to a newer processor. Just going to a new generation alone would cut my power usage significantly, since two lower-end xeon processors would have an equivalent processing power to all four Opterons. I do have a question, though, about how low I can go when idling. Modern processors seem to have several power saving features, namely "states" (p-states and c-states) that allow the processor to downclock or disable cores when they are not being used. I'll admit I don't fully understand what they do, only that they are advertised as reducing power consumption.

My main question is, with FreeNas, would I be able to utilize these features? It's my understanding that whatever "p-states" are, are intel-specific hardware-set states that I can't change myself, and "c-states" are core-specific states that can be set by commands from the BIOS/OS. is that correct? Would I be able to change the c-state of several cores to shut some down when I am away? Are there other power saving features I could utilize besides states? If I were to change c-states on my Opterons, would that have any effect on power usage? Right now, if I query the system for c-state ability, I get this response for every core:

Code:

dev.cpu.0.cx_method: C1/hlt													 
dev.cpu.0.cx_usage_counters: 92752345										   
dev.cpu.0.cx_usage: 100.00% last 6870us										 
dev.cpu.0.cx_lowest: C2														 
dev.cpu.0.cx_supported: C1/1/0

I'm not sure how to interpret these lines. Could I change anything to shut down some cores while I shop for a new system?

 

[tvsjr]

Your processor only supports C1. If it supported other levels, your output would look something like (this is from an E5-2670):

Code:

dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc/bma
dev.cpu.0.cx_usage_counters: 3930646 108054863
dev.cpu.0.cx_usage: 3.50% 96.49% last 203us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/41

This says this core has been in C1 for 3.50% and C2 for 96.49% of the last 203us.

The reality is the Opteron 6128 is *old* and built quite a bit before the big focus on power management that exists today. And, you're running a 32-core system. Tweak all you want, you're not going to make a 32-core system with 12 drives spinning in it a 20-watt box.

Just for comparison, my system (see sig) idles around 325W and will spike to 450-475W if I hit it fairly hard (virus scans, usually). I'm sure I could spike it further if I were sufficiently determined.

 

[keboose]

Okay then, I can deal with leaving it as-is until I get new hardware. If I were to replace the processors with two new(er) xeon processors, or maybe one high end consumer CPU, would I be able to take advantage of all the power saving features available?

 

[tvsjr]

Yep. In fact, power management should be default if you have a CPU that supports it. While you're upgrading, ditch the USB boot devices and install a single, small, SSD!

 

[keboose]

Awesome, thanks for clearing everything up! Do the xeon e-series processors have as many power saving features as a consumer processor? I am considering an i7-7820x, as it's passmark score is (roughly) equivalent to 4x the Opteron's score, and the price isn't too terrible, though I would prefer a xeon processor for ECC memory support. A possible candidate is an E5-2670 v2, assuming it can save power about as well as the i7.

 

[anmnz]

Well, Intel's incentives towards power efficiency come from datacentre designers trying to cool racks stuffed full of Xeons, not from gamers looking at a few bucks on their power bills! So yes, expect Xeons to have all the power saving features.

The Xeon you mention is older than the i7 by a few years but seems to be a popular choice, and new enough to be in the same ballpark power-wise as current CPUs, as I understand it.

 

[keboose]

Thank you for clarifying that. Now I have to actually do the shopping. Average price for these swing wildly all over the place, but it seems to average around $400-500. You think one I found for sale on Amazon for $266 could be fake?

 

[anmnz]

I believe there was a huge price crash on used E5-2670 (v2 specifically? not sure) a couple of years back, presumed due to large-scale datacentre upgrades flooding the market, and they were going for somewhere in the region of $100. So those prices all sound very high to me. I'd look on eBay. (BTW I seem to recall there was a defect with the original E5-2670's, not the v2's, and if you're buying those you want to be sure you're getting the later stepping. Others here will know more than me!)

 

[keboose]

That does clear it up. The 400+ ones might be old listings, who knows. I found some used ones on ebay for about $180, marked "seller refurbished." I don't know if I trust those listings (how do you even refurbish a cpu?) so will probably spring for the more expensive amazon listing advertised as a new CPU.

*Edit: Bought the stuff (including a used CPU on ebay.) I'll add a reply when I install it all.

 

[keboose]

So I got the new processor, motherboard, and RAM. To benchmark the difference, I ran a video encode using the Handbrake CLI software, using the same source file, and the same flags for each test. To compare, I also ran the same encode on my normal PC, which runs an i7-3820 CPU. I encoded a movie from the h.264 codec to h.265. The results are below (to give an idea of the difference, The passmark website ranks a dual Opteron 6128 setup roughly the same as my i7, so four Opterons should encode twice as fast, and the Xeon processor has a passmark score of just about double my i7.)

Code:

---------FOUR AMD OPTERON 6128:--------------
ENCODE TIME:	4hrs 11min 57sec
AVERAGE FPS:	10.409752
FILE SIZE:	  2,095,532,267 bytes
AVG POWER:	  375W
IDLE POWER:	 235W
PROCESS CPU%:   1600% - 2000% (Out of a possible 3200%)


---------ONE INTEL i7-3820:------------------
ENCODE TIME:	3hrs 5min 25sec
AVERAGE FPS:	14.15
FILE SIZE:	  2,094,622,140 bytes (different operating system means different result? Looked the same to me.)
AVG POWER:	  200W (this includes two monitors and misc. peripherals)
IDLE POWER:	 120W (again, monitors and peripherals)
PROCESS CPU%:   90% (Windows CPU use graph is just a percent of entire CPU unit, not individual cores)


---------ONE INTEL XEON E5-2670 V2:---------
ENCODE TIME:	1hrs 34min 41sec
AVERAGE FPS:	27.710720
FILE SIZE:	  2,095,532,267 bytes
AVG POWER:	  125W
IDLE POWER:	 40W
PROCESS CPU%:   1200% - 1500% (Out of a possible 2000%)

The results are night and day! This data really drives home how quickly technology can age, and also how passmark scores shouldn't determine your opinions of CPU models. My desktop CPU beat the AMD processors easily, even though it only had half the passmark score of the four combined AMD's. But that has nothing on the Xeon processor, which DID meet expectations set by passmark, in that it was almost exactly twice as fast as my i7, and almost THREE times faster at encoding than the Opterons.

Not to mention the power savings! the new system idles at 40 watts, which is practically nothing compared to either of the other setups. I'm impressed with how little power the hard drives pull when the system is idle, even though they don't spin down for a long time. With some quick math, I figure the difference in power draw will make up the cost of the parts in about 24 months. That's damn quick considering the total cost (585 USD) for the CPU, board, and RAM. That means for the last two years that I used the original hardware, I was basically wasting $250 every year in electricity. I am very happy I made the decision to upgrade.

 

[danb35]

I believe there was a huge price crash on used E5-2670 (v2 specifically? not sure) a couple of years back, presumed due to large-scale datacentre upgrades flooding the market,

That was the v1 (which weren't marked as v1, since there was no later version at the time) 2670s. I got my pair of them for around $125, not quite two years ago.

 

[SMnasMAN]

hey, do you have any info on how your entire FN system idle / loads now? (ie on your x9 + 12x hdds?) thanks