Save energy by making use of all C-States

Status
Not open for further replies.

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
The C-State trick did absolutely nothing for me. I'm glad OP had success in reducing its "environmental footprint" by doing something entirely free, but for my server, I dont see a single watt reduction from my UPS readings... Perhaps the setting was somehow active by default? Perhaps these CPU's are so old they dont fully support the parameters? ( I highly doubt).
No, the setting wasn't active by default, indicated by "dev.cpu.0.cx_lowest: C1". It would have been active if it said "dev.cpu.1.cx_lowest: C3" because the lowest C-State your processor supports is C3 ("dev.cpu.0.cx_supported: C1/1/1 C2/2/1 C3/3/35"). You thought that C1 is the lowest C-State because it has the smallest number, however, that is not the case. The higher the number of the C-State the more power saving it is.

Did you change the tunables to C3 instead of C1? If not then it's no wonder that you see no difference in power consumption.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
No, the setting wasn't active by default, indicated by "dev.cpu.0.cx_lowest: C1". It would have been active if it said "dev.cpu.1.cx_lowest: C3" because the lowest C-State your processor supports is C3 ("dev.cpu.0.cx_supported: C1/1/1 C2/2/1 C3/3/35"). You thought that C1 is the lowest C-State because it has the smallest number, however, that is not the case. The higher the number of the C-State the more power saving it is.

Did you change the tunables to C3 instead of C1? If not then it's no wonder that you see no difference in power consumption.

Of course I did. Same reading on the UPS to which the freenas server is connected. See screenshot of my tunables and a photo showing the reading on the UPS..

And yes, I forgot to change the comment on the 3rd tunable ;)
 

Attachments

  • freenas.png
    freenas.png
    163.1 KB · Views: 949
  • ups.JPG
    ups.JPG
    1 MB · Views: 801

styno

Patron
Joined
Apr 11, 2016
Messages
466
So... X10SL7-F with i3-4160, latest FW & bios. I already did a reset on ipmi & bios config and a complete power down just to make sure that everything is reset.
On the same cpu only core 0 is able to go to state C2. (and does so according to the cx_usage. The power consumption actually drops.)
Has anyone seen this behavior before and maybe a clue how to use C2 on the other cores as well?

(The system is idling around 60Watts now with fans at full speed.)

Code:
# sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C2
dev.cpu.3.cx_usage: 100.00% last 3us
dev.cpu.3.cx_lowest: C2
dev.cpu.3.cx_supported: C1/1/1
dev.cpu.2.cx_usage: 100.00% last 23us
dev.cpu.2.cx_lowest: C2
dev.cpu.2.cx_supported: C1/1/1
dev.cpu.1.cx_usage: 100.00% last 230us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1
dev.cpu.0.cx_usage: 33.02% 66.97% last 2us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/148
 

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
Of course I did. Same reading on the UPS to which the freenas server is connected. See screenshot of my tunables and a photo showing the reading on the UPS..

And yes, I forgot to change the comment on the 3rd tunable ;)
What does "sysctl dev.cpu | grep cx" show now? Just to make sure it's really active.

@styno did you check your bios for any settings regarding C-States?
 

styno

Patron
Joined
Apr 11, 2016
Messages
466
@styno did you check your bios for any settings regarding C-States?
Yes I did, the bios looks like configured correctly by default, all C-States are enabled (1~7)
 

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
Yes I did, the bios looks like configured correctly by default, all C-States are enabled (1~7)
I suspect that the energy saving functions of your chipset aren't fully supported by FreeBSD then, but I'm really no expert on this topic.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Hmm okay , after a complete hardware upgrade with more modern hardware (Supermicro X9SCL-F with Xeon E3-1220V2, Kingston 32GB DDR3-1600 ECC, fans fully configured with PWM, same hard drives, and only one PSU module instead of 2), I idle at 120W. THis is not jaw-dropping by any means but better than the humongous 340W I had before..

The power modules PWR-801L-1R from Supermicro are horrible energy wise. I will replace them soon. Probably will save another 20-25W or so.

Regarding settings within Freenas, I have now the following:

Code:
# sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C1
dev.cpu.3.cx_usage: 100.00% 0.00% last 405us
dev.cpu.3.cx_lowest: C1
dev.cpu.3.cx_supported: C1/1/1 C2/3/80
dev.cpu.2.cx_usage: 100.00% 0.00% last 310us
dev.cpu.2.cx_lowest: C1
dev.cpu.2.cx_supported: C1/1/1 C2/3/80
dev.cpu.1.cx_usage: 100.00% 0.00% last 5889us
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_supported: C1/1/1 C2/3/80
dev.cpu.0.cx_usage: 100.00% 0.00% last 2us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/3/80


I set all tunables to C2 since it is the lowest setting energy wise. Im not sure if its helping but I didnt try with the tunables off.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
I set all tunables to C2 since it is the lowest setting energy wise.
I must not be understanding you. Line 2 is the only tunable you should be setting (I believe the others will take their cue from it), and it appears to be set to C1. Try going to the command line and setting the sysctl directly. Then repeat the grep out, see if it shows C2. If it does, then set a post-init sysctl to make that setting.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Hey DrKK, you are not misunderstanding me, I am misunderstanding how this works.

1. How do I set this from CLI exactly?

Something like "sysctl hw.acpi.cpu.cx_lowest=C2" ??

2. From the WebUI, how would I set the parameters? Variable would be "hw.acpi.cpu.cx_lowest", value would be "C2" and type would be "sysctl"??
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Hey DrKK, you are not misunderstanding me, I am misunderstanding how this works.

1. How do I set this from CLI exactly?

Something like "sysctl hw.acpi.cpu.cx_lowest=C2" ??

2. From the WebUI, how would I set the parameters? Variable would be "hw.acpi.cpu.cx_lowest", value would be "C2" and type would be "sysctl"??
yes, yes.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Then it seems to be working well, I saw an immediate decrease of 10w in average when I issued the sysctl setting in CLI.

Now

Code:
[root@freenas] ~# sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C2
dev.cpu.3.cx_usage: 6.96% 93.03% last 3us
dev.cpu.3.cx_lowest: C2
dev.cpu.3.cx_supported: C1/1/1 C2/3/80
dev.cpu.2.cx_usage: 6.36% 93.63% last 347us
dev.cpu.2.cx_lowest: C2
dev.cpu.2.cx_supported: C1/1/1 C2/3/80
dev.cpu.1.cx_usage: 7.46% 92.53% last 905us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/3/80
dev.cpu.0.cx_usage: 5.78% 94.21% last 2831us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/3/80


If your drives are over 35C, you're doing it wrong.

Looking at the temps, would you recommend I spin up the fans a bit more?

Code:
/dev/da0
34 (Min/Max 17/41)
/dev/da1
37 (Min/Max 34/39)
/dev/da2
37 (Min/Max 36/40)
/dev/da3
39 (Min/Max 35/40)
/dev/da4
36 (Min/Max 34/39)
/dev/da5
35 (Min/Max 18/39)
/dev/da6
34 (Min/Max 18/38)
/dev/da7
39 (Min/Max 23/43)


Code:
[root@freenas] ~# sysctl -a | egrep -E "cpu\.[0-9]+\.temp"
dev.cpu.3.temperature: 40.0C
dev.cpu.2.temperature: 33.0C
dev.cpu.1.temperature: 38.0C
dev.cpu.0.temperature: 39.0C


EDIT: BTW smart is configured from the WebUI with Check interval of 30min, Difference of 5, Informational of 35, Critical of 40. Yet, I never got any emails from my system that the hard drives temps were above 35 or 40... Is that normal? Email is working I just tried the Test Email and got it.
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Yet, I never got any emails from my system that the hard drives temps were above 35 or 40
Under services->SMART in the GUI, you will find that you have to input your email address there. Did you not put it in there? This is separate from the "root" user email address deal.

Then it seems to be working well, I saw an immediate decrease of 10w in average
That sounds right. You should set it as a post-init tunable in the GUI, so that it automatically does this for you every time you start the FreeNAS.

Looking at the temps, would you recommend I spin up the fans a bit more?
Well, that depends on how butt-clenching we're being. Yes, I'd recommend, all things being equal, that you would run your drives cooler than this if possible. But you seem to be stable in the high 30's, which is certainly "OK", and if spinning up your fans would be noisy/inconvenient/the wife would be pissed off, then I don't think having your array in the 34-39C range is that bad. So it's up to you. But you have quite an investment in drives, so to shoot for the ideal temperature range for drive longevity that seems to be the large-enterprise-consensus (so, low 30's) is not unreasonable. But I'll be the first to admit that 39C is still within the green light zone, and I would not lose sleep over it. OK, that's a lie, I would probably lose sleep over it, but that's just DrKK being unreasonable.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Under services->SMART in the GUI, you will find that you have to input your email address there. Did you not put it in there? This is separate from the "root" user email address deal.
Yes I did. In the past I never got any emails about smart business... Just emails from Freenas itself (daily security run, scrub starting, errors or warnings, etc). But never about smart. This is true for my former 9.2.1.6 setup as well as the current 9.10 setup. Basically this never worked.

Is there a manual way to trigger a fake smart alarm and see if email would work?

You should set it as a post-init tunable in the GUI, so that it automatically does this for you every time you start the FreeNAS
Done!

But I'll be the first to admit that 39C is still within the green light zone, and I would not lose sleep over it. OK, that's a lie, I would probably lose sleep over it, but that's just DrKK being unreasonable.
Hahaha that one made me laugh!! But the server is currently somehow quiet enough so I can work a few feet from it. Quite frankly, the power module fan is so much louder than the case fans, spinning them up a bit more would probably go unnoticed. I am to upgrade the power modules to more efficient ones soon (platinum). That will be more quiet, and save some power as well from heat and noise.. Perhaps for the same overall power uage I will be able to spin the chassis fans a bit more?

I will keep an eye on smart temps to see if there is a trend to be above 38 - 39 C often. Thats why if smart email notifications were working that would be great.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
I never got any emails about smart business.
SMART only sends an email if there's an issue.

If you want to trigger a message, then go into services->smart->settings, and set the "critical" temp to be some number BELOW the current drive temperatures. Of course, change it back in a minute or two. :)
 

Handle

Dabbler
Joined
Oct 20, 2015
Messages
30
FreeNAS just seems to be an energy hog. I've switched to Proxmox, added two 4TB harddisks and my power consumption is around 50W in idle state, whereas with FreeNAS it was 55-60W without the harddisks (only two SSDs).
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
FreeNAS (by which, you mean, FreeBSD) is not an energy hog. But it does require the user to configure things correctly.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
OK to cap the smartd issue regarding hard drive temps and email warnings, I opened a bug ticket

https://bugs.freenas.org/issues/16254

Everything is properly configured yet its not working. I just had a A/C failure in my office where Freenas sits, and the hard drive temps went up to 40+ degrees. Never got an email from freenas, yet the syslog is overfilled with smartd messages... Look at the bug report. Something seems to be crashing in the notification framework...

Anyways its not good. Quite a chance I happened to come back from work earlier that day and find my office at 32C instead of the usual 20-21C...
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
And you get regular, FreeNAS emails? When you go to system->email, and "send test email", you *DO* receive that?

My skepticism is simply this: If there was a bug in this, we'd have heard jihad by now, for sure.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Hello DrKK

Yes I do get security runs and standard emails from freenas, and yes when I go to system > email, and click"send test email", I *DO* receive the test email no problem!

No need to be skeptic. The bug in the python framework was already acknowledged and confirmed by the Devs. Actually, someone had already opened a ticket on the bugtracker for this.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
The bug in the python framework was already acknowledged and confirmed by the Devs.
Fascinating, so we're all affected by this? Or just a couple people?
 
Status
Not open for further replies.
Top