Save energy by making use of all C-States

Status
Not open for further replies.

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
I'm using a server with an Opteron 3280 Processor. The power consumption seemed too high to me so I searched and found this thread in the FreeBSD-Forums. I thought I'd give it a try, enabled powerd and set the following tunables:

tunables.jpg


Lo and behold, the power consumption sank by about 13 Watts! That's 30$ less per year on the energy bill where I live :)
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Thanks Handle for this tip. To get this implemented in Freenas, did you just create tunables with "dev.cpu.X.cx_lowest=C2" in freenas' GUI or did you have to follow the lengthy thread you referred to ?

Also, do you have to reboot for the changes to take effect?

My freenas server draws an absurd 350W idle!! thats 24/7...

If I can squeeze a few watts out of the CPU's without being detrimental to the performance too much, I will. My workloads are mostly reads so I assume its not gonna be impacting too much my experience.
 
Last edited:

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
I enabled powerd, then opened the shell and ran "sysctl dev.cpu.0.cx_supported" to find out the lowest C-State supported by my CPU. After that I just created the tunables shown in the picture. Your CPU might support lower C-States, however, if you experience any lags when all cores are set to the lowest C-State you should try to set the first core to the 2nd lowest and see if that fixes the problem.

A reboot isn't needed for the changes to take effect.

What is your setup? 350W sounds way too high. How did you measure that?
 
Last edited:

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
I enabled powerd, then opened the shell and ran "sysctl dev.cpu.0.cx_supported" to find out the lowest C-State supported by my CPU. After that I just created the tunables shown in the picture. Your CPU might support lower C-States, however, if you experience any lags when all cores are set to the lowest C-State you should try to set the first core to the 2nd lowest and see if that fixes the problem.

A reboot isn't needed for the changes to take effect.

What is your setup? 350W sounds way too high. How did you measure that?

OK first of all, powerd seems to be enabled on my server (under Settings > advanced) (checkbox ticked). Then, I ran the command suggested by Rand:

Code:
sysctl dev.cpu |grep cx
dev.cpu.0.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_usage: 100.00% 0.00% 0.00% last 416us
dev.cpu.1.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_usage: 100.00% 0.00% 0.00% last 60754us
dev.cpu.2.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.2.cx_lowest: C1
dev.cpu.2.cx_usage: 100.00% 0.00% 0.00% last 6583us
dev.cpu.3.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.3.cx_lowest: C1
dev.cpu.3.cx_usage: 100.00% 0.00% 0.00% last 94571us
dev.cpu.4.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.4.cx_lowest: C1
dev.cpu.4.cx_usage: 100.00% 0.00% 0.00% last 371us
dev.cpu.5.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.5.cx_lowest: C1
dev.cpu.5.cx_usage: 100.00% 0.00% 0.00% last 187us
dev.cpu.6.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.6.cx_lowest: C1
dev.cpu.6.cx_usage: 100.00% 0.00% 0.00% last 26155us
dev.cpu.7.cx_supported: C1/1/1 C2/2/1 C3/3/35
dev.cpu.7.cx_lowest: C1
dev.cpu.7.cx_usage: 100.00% 0.00% 0.00% last 5163us


So If I understand this, the lowest state my CPU's support (pair of Xeon L5420), is C1. Would I see a major energy difference between C1 and C2?? I fear performance would dramatically reduce.... Like I said, my server is used for backups about 3 times a day (small daily rsync runs from Linux servers), and mostly idles or is used for video playback on a Kodi (XBMC) media center... No CPU usage at all, CPU's are 99.9% of the time below 10% usage. I will try C1 for now and see what happens...

As for my actual energy usage, its indeed between 340 & 350W. This is an old Supermicro SC836 16 bay chassis with 8x SATA2 & SATA3 drives, running off of a pair of IBM M1015 HBA controllers. 48GB RAM and a pair of redundant 710W (or 800W, I cant tell without pulling the server off the rack) PSU's. The actual power reading is provided by my APC UPS which the freenas server is connected to.

I know, this is CRAZY. The fans are running at the second lowest speed but the IPMI card being defective (its working fine but for whatever reasons started to throw sensor reading errors a while back), I cannot rely on the readings too much. Nevertheless, ipmitool provides this:

Code:
[root@freenas] ~# ipmitool sensor list all
CPU Temp 1       | na         |            | na    | na        | na        | na        | na        | na        | na       
CPU Temp 2       | na         |            | na    | na        | na        | na        | na        | na        | na       
CPU Temp 3       | 30.000     | degrees C  | ok    | na        | na        | na        | 76.000    | 78.000    | 80.000   
CPU Temp 4       | na         |            | na    | na        | na        | na        | na        | na        | na       
Sys Temp         | na         |            | na    | na        | na        | na        | na        | na        | na       
CPU1 Vcore       | 1.656      | Volts      | nr    | 0.680     | 0.688     | 0.696     | 1.624     | 1.632     | 1.640    
CPU2 Vcore       | 1.656      | Volts      | nr    | 0.680     | 0.688     | 0.696     | 1.624     | 1.632     | 1.640    
3.3V             | 1.984      | Volts      | nr    | 2.912     | 2.928     | 2.944     | 3.648     | 3.664     | 3.680    
5V               | 2.904      | Volts      | nr    | 4.416     | 4.440     | 4.464     | 5.520     | 5.544     | 5.568    
12V              | 8.832      | Volts      | nr    | 10.464    | 10.560    | 10.656    | 13.344    | 13.440    | 13.536   
-12V             | -20.500    | Volts      | nr    | -10.500   | -10.600   | -10.700   | -13.300   | -13.400   | -13.500  
1.5V             | 3.248      | Volts      | nr    | 1.296     | 1.312     | 1.328     | 1.664     | 1.680     | 1.696    
5VSB             | na         |            | na    | na        | na        | na        | na        | na        | na       
VBAT             | na         |            | na    | na        | na        | na        | na        | na        | na       
Fan1             | 0.000      | RPM        | nr    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan2             | 4400.000   | RPM        | ok    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan3             | 3700.000   | RPM        | ok    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan4             | 3600.000   | RPM        | ok    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan5             | 0.000      | RPM        | nr    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan6             | 0.000      | RPM        | nr    | 200.000   | 300.000   | 400.000   | na        | na        | na       
Fan7/CPU1        | na         |            | na    | na        | na        | na        | na        | na        | na       
Fan8/CPU2        | na         |            | na    | na        | na        | na        | na        | na        | na       
Intrusion        | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na       
Power Supply     | na         | discrete   | na    | na        | na        | na        | na        | na        | na       
CPU0 Internal E  | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na       
CPU1 Internal E  | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na       
CPU Overheat     | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na       
Thermal Trip0    | na         | discrete   | na    | na        | na        | na        | na        | na        | na       
Thermal Trip1    | na         | discrete   | na    | na        | na        | na        | na        | na        | na       
BIOS             | na         | discrete   | na    | na        | na        | na        | na        | na        | na  


According to ipmitool, the fans would be running in the 4000RPM range. I am not sure if this is accurate or even realistic, but all I can tell is that they are noticeably quiet, about half of what they sound like when I boot up the server (they run at 100% speed). With the way they are spinning now, the HDD's are all below 32C, and all cores are reported below 60C.
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
First,
you will not loose performance by enabling C-States - the cpu will ramp up on request without significant delay.

Second -
if this is an Supermicro X7 board (which seems to be common on older SC836s) then you might have the rather power hungry FB-DIMMs (5-8W per) in addition to 2x50W worth of CPUs and 16 disks that need some juice too (5-10W per).

Fans and HBAs take their toll as well, so I while your consumption is still on the high side its at least partly explainable...
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
well... after about an hour with C1 setting, I dont see a single improvement on wattage.... All 8 cores are set with C-state C1.. Is that normal?

Also,

2x Xeon CPU = 50x2=100W
12x 4GB FB Dimm = 12x8=96W
8x SATA HDD = 8x10=80W

That would mean 4 fans & 2 HBA's would take around 65W... Is that normal? And thats assuming a worst case scenario with 8w per RAM stick, and two CPU's at full TDP!

Time for this server to retire or be upgraded... Its costing me about 200$ per year of electricity for this extra 200W that it uses when compared to the other more energy efficient servers.....
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Well 4 fans might take 20W easily and two HBAs at 10W each would be another 20. Plus mainboard itself and maybe some other stuff ...

And yes, if you spend $600 on an upgrade you will have more power with the same or less cost. My SC836 runs at 120-180W with 12 disks (depending on the disks) (E3 CPU, v3/v5)

Oh and I believe C1 is the default value.
You might simply try to set C2...
 

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
Regarding C-States: The higher the number the lower the C-State. C3 is the lowest for your CPU, so you can change it to C3 for every core and you should see a difference in power consumption.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Regarding C-States: The higher the number the lower the C-State. C3 is the lowest for your CPU, so you can change it to C3 for every core and you should see a difference in power consumption.
You saw this from the output posted before?

  1. dev.cpu.3.cx_supported: C1/1/1 C2/2/1 C3/3/35
  2. dev.cpu.3.cx_lowest: C1
  3. dev.cpu.3.cx_usage: 100.00% 0.00% 0.00% last 94571us
 

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
Yes, I did. It says that the lowest (and therefore most power saving) C-State supported by your processor is C3 but C1 is currently the lowest allowed. You can create the same tunables you see in my first post but with C3 instead of C2. You should then see your power usage drop a little.
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
C1 is the *****HIGHEST***** power.

And also, if your FreeNAS box is drawing 350W, you are using completely inappropriate, or completely antique hardware.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Or you're using AMD
Sir, I believe if you look at the original post I made, I said, "...you are using completely inappropriate, or completely antique hardware", and thus, covered that base.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
C1 is the *****HIGHEST***** power.

And also, if your FreeNAS box is drawing 350W, you are using completely inappropriate, or completely antique hardware.

All detail is available in post #5. It is antique by today's standards, and certainly not energy efficient, so I am looking for upgrading this hardware. On the other hand, energy cost here is the lowest in north america and probably one of the lowest in the entire world.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
On the other hand, energy cost here is the lowest in north america and probably one of the lowest in the entire world.
A dangerously short-scoped bit of logic sir. I certainly don't advocate spending $50 to save $1 in energy, and you won't catch me at any Greenpeace rallies. But I certainly do what I can to keep negative pricing pressures on energy as a commodity.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
I certainly don't advocate spending $50 to save $1 in energy

The C-State trick did absolutely nothing for me. I'm glad OP had success in reducing its "environmental footprint" by doing something entirely free, but for my server, I dont see a single watt reduction from my UPS readings... Perhaps the setting was somehow active by default? Perhaps these CPU's are so old they dont fully support the parameters? ( I highly doubt).

I don't want to get into a semi political argument, I believe we live in totally different societies and countries, but I fully get your point. At the end of the day, its all about cost VS benefit ratio. So unless this server dies, or I get a surprise income, or I can significantly reduce my energy usage for a rather small expense, I will keep running this thing and keep an eye for upgrade parts... According to my quick calculation, my "ROI" would be between 2 and 3 years, which is what I expect this server to keep running..
 

russnas

Contributor
Joined
May 31, 2013
Messages
113
i didnt expect this to work with newer intel cpu and with Enable powerd (Power Saving Daemon), im sure the first few C states were enabled by default, i put unfortunately put my hdd's standby on to reduce power which will save 50$ yearly, not worth it if it causes more wear, put the system dataset on a 2.5 inch drive with the plugins which is much better for the raid
 

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
If i remember correctly there are bios settings that needs to set properly to allow for this functionality by software.
 
Status
Not open for further replies.
Top