powerd and how it can impact server performance...

[cyberjock]

So I got bored this morning and decided to experiment with powerd along with some latency issues with my server. Recently migrated my server from Server 2008 R2 to FreeNAS and I have been experimenting with my system.. here's my hardware:

CPU - Intel Xeon E5605 2.13Ghz Quad core (80W thermal) without HT support and BIOS version F12
Motherboard - Gigabyte EX58-UD5
RAM - 12GB of Kingston ECC RAM
Video card - EVGA 512-P3-1310-LR (Nvidia GeForce 210 512MB of RAM)
RAID Cont - Areca 1280ML-24 Port with battery backup
NIC - Intel Server Dual Port(Gb)
Hard Drives - WD20EARS x 18 on RAIDZ3 (23.2TiB total available)
PSU - OCZ Z Series Gold OCZZ850M 850W

Note: I have disabled all unused hardware in the BIOS. This includes onboard NICs, SATA & IDE controller, Firewire, etc.

My server issue (mentioning it here because it is related to this thread): When traversing through directories on my server it often takes 5-10 seconds for a directory to display on my desktops(Windows 7). Even if the directory is empty! I can double click on a new folder and it may load instantly. If I go back to the parent folder and click again it may load instantly or lag(hmmmm!). If I do a 'zpool iostat 1' I can see that the zpool reads about 300k per second every second while the server is lagging then goes idle when the directory is displayed. If I'm trying to stream a video file and traverse directories I can expect my video to freeze until the completion if the directory listing(even if its an empty directory!) which is BS for my hardware. I even tried to force the directory into the ARC by doing a cron job executing 'ls -aR /mnt/tank' every 30 mins. I don't have alot of files either.

So now that I have my FreeNAS server up exactly how I want it I decided it was time to see what kind of power I'm sucking down and hopefully fix my latency problem. This can get annoying if I can expect lag every time I traverse a directory! It's already annoying and I've only been using my server for about 2 weeks.

My UPS has a wattmeter built-in and I've verified that with the server off it reads 000 watts. Booted up my server and after booting is finished I found that powerd is enabled. Good to go. (Hint: Wrong.. but we'll get to that later)

Wattage with system idle - 234w

WOW.. that's a bit high. What's my wattage without the hard drives?

Wattage idle with no hard drives - 99w

What if I disable powerd?

Wattage idle with no hard drives and powerd disabled - 108w

At this point I'd like to mention something about power supplies. They typically have peak efficiency around 50% and have relatively poor efficiency <20% and above 80%. I plan to add some 3TB drives in the next 30 days but I probably should have opted for a 500-600w power supply. Oh well. It was all I had and I wasn't about to buy ANOTHER power supply. I have 2 spares already!

So powerd saves me a whopping 9 watts.

I so hoped for more than that(but 8-12w seems to be pretty typical). But wait.. I've just noticed something. The lag when traversing directories is gone!

Time for some detective work. So what speeds are available to my CPU?

# syctl dev.cpu.0.freq_levels

dev.cpu.0.freq_levels: 2128/80000 1995/66000 1862/55000 1729/45000 1596/37000 1463/30000 1330/25000 1197/20000 1047/17500 897/15000 748/12500 598/10000 448/7500 299/5000 149/2500

WOW. My CPU can go down to 149Mhz. Like I expect to see that, right?

If you run this command you can get the frequency(in Mhz) of your CPU. If you run single CPU this works great, dual CPU you'll have 2 parameters to monitor.

# sysctl dev.cpu.0.freq
dev.cpu0.freq: 149

So my CPU is at 149Mhz. Fantastic. If I disable powerd from the GUI:

# sysctl dev.cpu.0.freq
dev.cpu0.freq: 2128

With the server idle and 1 second intervals while enabling powerd you can see my CPU ramp down:

dev.cpu.0.freq: 2128
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 897
dev.cpu.0.freq: 897
dev.cpu.0.freq: 748
dev.cpu.0.freq: 748
dev.cpu.0.freq: 598
dev.cpu.0.freq: 598
dev.cpu.0.freq: 448
dev.cpu.0.freq: 448
dev.cpu.0.freq: 299
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149

So what happens when streaming a blu-ray? I just so happen to have a blu-ray disk on my server...(1 second intervals again)

dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149

So my CPU can handle dishing out a blu-ray with 149-299Mhz. Amazing!

So what about a directory listing where the streaming freezes and I have to wait 5-10 second for the directory listing?

dev.cpu.0.freq: 149 <- requested the directory
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149 <- received the directory

WTH? My CPU is idle when I'm sitting at my desk waiting impatiently for a directory listing? This doesn't seem right.

So I go disable powerd, verified CPU frequency is now 2128Mhz and streaming is flawless along with directory listing. I traversed about 15 directories(usually every other directory or so would freeze) and not one freeze or delay of more than 1 second, even directories with 100s of files.

So can we tweak powerd to make it more friendly? After all, this is unacceptable and since I'm bored and this is a learning opportunity why not see about saving those 9watts. Yes, I know if I replace a single incandescent light bulb with a CFL I can save roughly 60-90w, but this is about learning and not about the watts.

With regards to FreeNAS(as well as most other popular OSes) is that the "default" should work for most people. I tend to stick with Intel CPUs, very common hardware, and my hardware far exceeds the recommended specs for the OS. My philosophy is that if you have all 3 of those(of course, AMD CPUs that meet the recommended speeds are acceptable too) and you are being forced to tweak you probably have done something wrong or overlooked something. If you don't have something wrong then its likely a bug. Being that I'm fairly new to FreeBSD I try to avoid jumping to the conclusion that something is a bug when user error is far more. After all, if you aren't sure if what you are doing is right how are you to know if you are doing something wrong?

So I must have something wrong, right? Well, lets take a look at my BIOS settings since its a little hard to get the powerd settings wrong. After all, its a freaking checkbox!

Sure enough, my BIOS has "C3/C6/C7 State Support" and it is disabled. Oops. Let's enable it!

Lets stream a blu-ray and do a directory listing. Sure enough it doesn't freeze. So what happened to CPU freq this time?

dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 2128 <- opened a directory and received the listing
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330

Here's some oddball stuff I'm adding since I did do a little more testing afterwards. I like to play and learn!

1. I disabled the "Virtualization Technology" in my BIOS and saw idle watts drop by 35w. So if you have a FreeNAS server, you should consider disabling this feature as it really adds zero benefit but costs you quite some watts. My CPU temp even went down by 2C!

So what have we learned from all of this:

1. Consider BIOS settings when you have performance issues that can't be identified by insufficient hardware and/or failing hardware.
2. User error with setting up the system.
3. Disable that pesky CPU virtualization feature. It's worthless for FreeNAS and seems to impact CPU power usage and temp.
4. powerd does work exactly as expected, but the results are minimal. If you are having performance issues you may want to try disabling it. You never know when your system is choking on low CPU speed. I've left powerd enabled now that I've fixed my BIOS setting.
5. powerd seems to take about 10-15 seconds to really ramp down CPU frequency all the way. If you are in a business environment I'd probably disable it completely. More than likely the CPU will never really go completely idle and the potential problems don't outweight the benefit. In a business environment you probably aren't too worried about saving 10w. For home use leaving it enabled is probably the better option if you care about saving some watts.

 

[jgreco]

That's kind of interesting. powerd is not particularly sophisticated and is most useful on laptops these days.

It may help to understand that during the CPU frequency wars of the mid-2000's, designers were happy to burn watts in order to gain a little speed. However, operating systems handled idle states in various ways, and there was little incentive to "be efficient".

We can thank the advent of multiple cores for slowing down growth in watt-burn, and then virtualization did something amazing: it forced operating systems to become more efficient. An idle OS shouldn't be consuming lots of cycles, and OS's that did were not good neighbors when virtualized. That combined with a drive for green computing, and CPU designers started providing features to help lower consumption too. As a result, most mainstream OS's evolved power-saving strategies that went beyond mere HLT and got more aggressive.

But to a large extent, the benefits of twiddling this sort of thing are nowhere near as great as they once were. CPU technology has gotten pretty good at managing power levels - so much, in fact, that I was completely amazed when we first built an E3-1230 in the shop and ended up with a platform that idled at around 60 watts and ran maybe 110 at full tilt, or something like that.

So anyways, I thought you might want to "play further" at least if you have a watt meter of some sort handy.

The fun news is, you don't need powerd to get in the way! Turn off powerd and manage the settings yourself for best fun.

Here's how to see what's going on.

# sysctl -a | grep dev.cpu.0.freq
dev.cpu.0.freq: 1300
dev.cpu.0.freq_levels: 1300/4582 1137/4009 1100/3750 962/3281 825/2812 800/2501 700/2188 600/1875 500/1563 400/1250 300/937 200/625 100/312

That's the current level followed by available candidate levels. 1300 is max on this N36L, and system eats around 61 watts (16GB+4xWD20EARS+BR10i).

# sysctl -w dev.cpu.0.freq=100
dev.cpu.0.freq: 1300 -> 100

So now it starts at 51 watts but then drifts up to 57 because FreeNAS is perpetually running stuff in the background, and at 100, it isn't fast enough for all the stuff to ever get all done. But that's interesting all by itself, because it means that even at a lower CPU speed we know that it eats from around 51 watts to 57 watts. So you can also create a process to eat CPU time and measure at 1300 if you want.

Code:

# python
Python 2.7.3 (default, Oct 24 2012, 08:18:42)
[GCC 4.2.2 20070831 prerelease [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> i=1
>>>
>>> while True:
...     i=i+1
...

which will happily eat up one CPU core until you hit control-C.

You can also see what states are available

Code:

dev.cpu.0.cx_supported: C1/1 C2/1 C3/1

and even set them

Code:

dev.cpu.0.cx_lowest=C2
dev.cpu.1.cx_lowest=C3

Setting one CPU to be kept out of C3. But neither the N36L nor the ESXi box I'm looking at support anything beyond C1. I'm not sure if this is a FreeNAS kernel limitation or a platform limitation, and since I don't really care, I'm just passing these hints on to you if you should happen to wish to experiment further.

 

[cyberjock]

I did do some reading on C-states. I was hoping to add something about them, but I couldn't find any command that actually tells you what C-state your CPU is in at any given moment or anything that tells you historical data on your c-states.

Overall I'm pretty disappointed in the whole efficiency of my system. I expected lower watts without the hard drives. The CPU has been a major disappointment in Windows since day one. I bought the CPU without HT and saved something like $90. But it has always performed poorly until FreeNAS. With Server 2008 R2 I was lucky to get 40MB/sec over the network. My CPU seems to support C-states C1, C2 and C3.

On my system when I set the speed with the sysctl command it seemed to work fine, but for 9 watts it really isn't work messing with. For me it just means my scrubs won't be at 800MB/sec anymore.

 

[Milhouse]

powertop should tell you about C-state usage, but I doubt there is a FreeBSD version.

 

[ProtoSD]

Shouldn't this thread be in the performance section of the forums? It seems like the more logical place, but if you have a good reason for having it in the NOOB section, that's fine too.

 

[cyberjock]

I guess it could be moved. I put it in the noob section since alot of newbies read here first.

 

[ProtoSD]

I think it's a good post and it'll just get buried by the daily garbage if it stays here. Plus its kinda teaching the noobs bad habits ;)


I just moved it and gave it a LONG expiring redirect, so it's like having it in both places for 2 months.