C-states "bug"?

[Fran Aquino]

After a couple of years with FreeNAS on an AMD Athlon II X3 415e (3 cores @ 2.5 GHz, 45 W TDP) I decided to swap the CPU for an FX 8320e (8 cores @ 3.2 GHz, 95 W TDP) as this probably will be the last gasp for socket AM3+ and I'm not still ready for a full hardware upgrade. To make sure I didn't screw it up I disconnected all my zpool disks (6x 2.5 Samsung SpinPoint M7) and FreeNAS USB boot drive and installed Windows [ok, shame] to a new 2.5 drive do some stress testing with OCCP, Prime95 etc.

Under Windows my ElCheapoWattMeter showed anything from 45 W idle to 170 W at full load, but after reconnecting the zpool drives and booting FreeNAS again, idle power hovers around 100 W... something didn't quit feel right, so after some googling I got to this thread.

After setting hw.acpi.cpu.cx_lowest=C2, idle power went down to 55 W. That 10 W difference when idle seems consistent with the difference in the number of connected HDDs.

 

[joeschmuck]

On my AMD FX 8320E + AMD 760G chipset + 6x Samsung Spinpoint M7 idle power consumption hovers around 100 W with hw.acpi.cpu.cx_lowest=C1
After setting it to C2, idle power went down to 55 W.

That is a very substantial savings, good for you trying it.

 

[181228]

I just tried setting the lowest state to C2 on an FX-8350 using a 4-drive Z2 pool.

My idle power dropped from 120W to 83W, with the drives still spinning and a powerd daemon that won't start because I have an AMD CPU.

That's a savings of 37W and a decrease in energy cost of 31%.

Wow. The savings can be so significant that I don't understand why the devs continue to turn a blind eye to this.

 

[Holt Andrei Tiberiu]

Last time i tested, each HDD you put in a system is 5W of power ( yeah, and I will have 25, and 3 x 14 disk shelv's ).
I played around with power settings also, on my DL380p G8, the server will draw as low as 58w with both PSU ( in bios i used maximum power savings, this also put my cpu at max 1200MHz and disabled HT, no drives or hba card).
With all settings normal, psu power set to balance/efficiency, the server uses 90-100w.
I used the so mentioned ElCheapoWattMeter i had, and each disk I inserted in the server got me another extra 5W. YEEE
Not mentioning the 15.000 rpm FC drives in the netapp shel's, they consume 7-11 w/disk.

 

[joeschmuck]

Wow. The savings can be so significant that I don't understand why the devs continue to turn a blind eye to this.

I don't think they are turning a blind eye to it. I think it's more of the fact that many people here don't really care and would rather have a fast responsive system, dropping the C state can impact that peppy performance. Additionally what works for one system may not work on a different system and could actually cause issues. Please note that while FreeNAS is looking like a well polished program, it was still originally created with the end user being somewhat tech savvy so playing around with C states is fair game if you want to use them but it's still at your own risk. We are glad that you posted your success as someone else may be able to benefit from this.

 

[Ericloewe]

Power management improvements really should be directed upstream at FreeBSD. That's where the relevant development happens. Or rather, it happens when there's hardware available.

So, if you have access to hardware you wish would have better support for power management, ping someone who's involved in that and offer them access to the hardware. Or offer to send it their way if you don't need it anymore and it's still reasonably recent.

 

[DrKK]

Gentlemen: Your necromancy of a 40+ month old post is commendable.

 

[181228]

I think it was worth it -- I just cut my box's power consumption by 54%. :D

CPU operate at MHz frequencies. Dropping from C1 to C2 shouldn't impact "peppy performance." I've observed no perceptible change in responsiveness of the system, which is to be expected because on a human time scale the C-state transitions are instantaneous (microsecond events). Anyone who is interested in power savings is more likely to create a latency problem by spinning down their disk array. C1/C2 C-states? They're not worth worrying about in terms of system responsiveness.

As far as power management goes, the only problems that need to be directed upstream are the lack of support for AMD-specific power management. that's definitely an upstream problem. From what I recall in examining the FreeNAS scripts, they purposefully set the C-state to stay at C1 during system initialization, which means that never entering C2 is a FreeNAS development decision and is not an upstream support problem.

In keeping with the bringing of good news: Last night I set the 4-disk Z2 pool to spin down after a extended period of non-use and this morning when I checked the power meter my system was idling at 55W.

That's quite an improvement from the 120W baseline at idle. The combination of C2 and spindown changes have decreased idle power consumption to 46% of what it was before. I think the saving of 65W when nobody is using the system is significant.

I hope that this info might help someone else who's having problems with high power consumption on an AMD box. Cheers.

 

[jgreco]

Wow. The savings can be so significant that I don't understand why the devs continue to turn a blind eye to this.

Because trying to muck around with C-states can cause systems to freeze or crash in unexpected ways. BTDT. There are platforms where it just doesn't work correctly.

FreeNAS is developed as a way to beta test TrueNAS, and I suspect that the enterprise market segment that TrueNAS is aimed at isn't particularly concerned about this sort of thing.

Additionally, spinning drives up and down is known to be a significant factor in reduced lifespan of the drives.

 

[mkninc]

Is the C2 C-State setting already used by default in current FreeNAS versions?
I didn't set the any sysctl settings.

FreeNAS-11.2-RELEASE-U1

Code:

root:~ # sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C2
dev.cpu.7.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.7.cx_usage_counters: 170798 5389792
dev.cpu.7.cx_usage: 3.07% 96.92% last 326us
dev.cpu.7.cx_lowest: C2
dev.cpu.7.cx_supported: C1/1/1 C2/2/50
dev.cpu.6.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.6.cx_usage_counters: 152940 4894398
dev.cpu.6.cx_usage: 3.03% 96.96% last 300us
dev.cpu.6.cx_lowest: C2
dev.cpu.6.cx_supported: C1/1/1 C2/2/50
dev.cpu.5.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.5.cx_usage_counters: 245730 5983897
dev.cpu.5.cx_usage: 3.94% 96.05% last 1979us
dev.cpu.5.cx_lowest: C2
dev.cpu.5.cx_supported: C1/1/1 C2/2/50
dev.cpu.4.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.4.cx_usage_counters: 169761 5198103
dev.cpu.4.cx_usage: 3.16% 96.83% last 215us
dev.cpu.4.cx_lowest: C2
dev.cpu.4.cx_supported: C1/1/1 C2/2/50
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.3.cx_usage_counters: 340849 5682081
dev.cpu.3.cx_usage: 5.65% 94.34% last 1383us
dev.cpu.3.cx_lowest: C2
dev.cpu.3.cx_supported: C1/1/1 C2/2/50
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.2.cx_usage_counters: 151415 5268792
dev.cpu.2.cx_usage: 2.79% 97.20% last 62us
dev.cpu.2.cx_lowest: C2
dev.cpu.2.cx_supported: C1/1/1 C2/2/50
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.1.cx_usage_counters: 318357 5638779
dev.cpu.1.cx_usage: 5.34% 94.65% last 160us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/2/50
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.0.cx_usage_counters: 272840 23726813
dev.cpu.0.cx_usage: 1.13% 98.86% last 343us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/50

 

[181228]

FreeNAS is developed as a way to beta test TrueNAS, and I suspect that the enterprise market segment that TrueNAS is aimed at isn't particularly concerned about this sort of thing.

Thanks.