Tale of two servers and power consumption, the daemon and the shark

[Borja Marcos]

Hi :)

I currently have two identical servers. They are HP Microserver Gen8 (the basic configuration) with 10 GB of memory and four WD Red 3 TB drives (different firmware and "NASWare" version, whatever that means).

One of them is running FreeBSD 10.1-STABLE and the other one FreeNAS 9.3. Checking the power consumption of both, which, frankly, is very low, I notice that it's possible to achieve a close to 8% reduction in idle power consumption, which I think is not bad.

Exhibit one: graph (obtained with a Ubiquiti mPower unit) showing the power consumption of both servers. Green line running FreeBSD, blue line running FreeNAS.

And what happened at 8:45? I issued this command on the FreeNAS system, matching what I usually do on FreeBSD.

Code:

[root@freenas] ~# sysctl hw.acpi.cpu.cx_lowest=C3
hw.acpi.cpu.cx_lowest: C1 -> C3

And now we can see that the processor is entering the low power state when idle. (C2 in this case, I must check the BIOS setup and, by the way, I haven't compared the configuration of both machines).

Code:

dev.cpu.0.cx_supported: C1/1/1 C2/2/96
dev.cpu.0.cx_lowest: C3
dev.cpu.0.cx_usage: 2.39% 97.60% last 1us
dev.cpu.1.cx_supported: C1/1/1 C2/2/96
dev.cpu.1.cx_lowest: C3
dev.cpu.1.cx_usage: 2.83% 97.16% last 3us

I know it can depend on the particular machine, processor *and*, more confusingly, power management configuration thanks to the typical peecee chaos, but it's worth having a look I think.

I am wondering of course, will such a configuration have an impact on performance?

 

[cyberjock]

I know it can depend on the particular machine, processor *and*, more confusingly, power management configuration thanks to the typical peecee chaos, but it's worth having a look I think.

I am wondering of course, will such a configuration have an impact on performance?

The short answer is that "yes, it can have an impact on performance".

While I commend you for figuring this out and posting, there's some info to share. Me and a few other people did some tests with C-states last year sometime. Here's things to consider...

1. Not all CPUs support >C1 support. Most do, but some don't. Some AMDs seems to *really* dislike this setting, so making it the default would probably piss some people off. Instability on a file server makes for unhappy sysadmins... so i've heard. Some motherboards don't go deeper than C1 without problems either.
2. Some hardware does not do graceful entry and exist from >C1 states. As a result its not recommended C3 be the default.
3. Something like 80% (don't quote me) of the power savings you will get from C-states will be saved by going to C1 and no further. As C1 is pretty universally supported and generally expected to be used, it not working with given hardware (Intel/AMD or a particular model of motherboard) would be pretty well known in short order.
4. There is a performance penalty when going into and out of states deeper than C1. This can add serious latency for some situations and could be a real killer for workloads that are intermittently creating workload. For home users you probably wouldn't be upset if it took a second for the server to 'wake up'. But in a production environment where you are doing things like iSCSI which is sensitive to latency from end-to-end that could be a real mess.
5. Depending on your PSU's efficiency curve, it is actually possible to decrease the power usage of your motherboard, but see an increase in power usage at the wall because your PSU is less efficient at lower power. So for an unlucky group of people, they would actually see power usage increase!
6. We're talking about 4 watts from your graph. Yes, if you look at 24x7 usage, that comes back to an ideal savings of about 35kwh in a year. Or about $3 for most of us. If you were to swap out a 100w incandescent with a 13w CFL, and you only used that light bulb 2 hours a day (which is overly low), you'd save about 64kwh in a year, or about $6 or so. Is a savings of $3 worth the potential problems from #1, #2, #3, and especially #5? In my opinion, not really. I'd also bet you aren't trying your hardest to keep lights off in your house, using CFLs everywhere it's reasonable, and doing your laundry by hand either. So 3 watts isn't exactly bleeding power either. FreeNAS isn't meant to be heavily tweaked for all hardware (if you want something that is highly optimized that's what TrueNAS is for). Yes, you can do it (and you may find it works just fine for you). I personally leave C1 state and I am plenty happy with that. If C3 makes you happy, feel free to stick with it. :)

While its a fun experiment to do, the potential consequences from an unreliable\unstable server makes it somewhat risky. Let's say that 2 months from now you are having to reboot your server every day and with no errors or anything to go on you can't figure out what the problem is. Let's pretend it's because of C3 state. How much time and effort are you going to spend trying to identify and fix the problem? Is your time and effort worth $2 a year to simply avoid the problem altogether? Personally, if I started having stability problems (which might not start for days or weeks after a setting change) how much time are you going to lose to troubleshooting the problem? Honestly, I don't even know where I'd start to troubleshoot a problem like that.

I have no doubt that some hardcore power savers could easily argue up and down forever that a watt saved is a watt saved. But the larger picture needs to be considered too. Putting hardware to sleep and such has been very difficult to do right. Apple laptops have ultra low power usage compared to Windows because Apple owns all of the hardware and they spend stupendous amounts of time optimizing their OS to work with their exact hardware. It's paid off for them too. FreeNAS uses commodity hardware that has an almost infinite combination of hardware that you could use, so deep level optimizations aren't really possible. One of the great things about sticking with the "crowd" is that if things go wrong you'll probably find 50 people that have used your hardware and can vouch that everything works. Deviate too much and you are left on your own to troubleshoot and correct the problem. Now as a FreeBSD user you probably know quite a bit more than our average user in the forum. But most of our users don't even know where log files are kept, so troubleshooting anything that doesn't give an error that is immediately obvious to the most casual of computer users will probably confuse them to no end.

If it works great for you, feel free to use it and feel better about the wattage saved. I'll stick with my C1 state. ;)

Now if you'll excuse me I have to read up on these Ubiquiti mPower because they look very cool and I think I want one. ;)

 

[Borja Marcos]

1. Not all CPUs support >C1 support. Most do, but some don't. Some AMDs seems to *really* dislike this setting, so making it the default would probably piss some people off. Instability on a file server makes for unhappy sysadmins... so i've heard. Some motherboards don't go deeper than C1 without problems either.
2. Some hardware does not do graceful entry and exist from >C1 states. As a result its not recommended C3 be the default.

Understood. Actually both FreeBSD and FreeNAS don't detect a state further than C2 in this configuration, must be some BIOS oddity.

I didn't know that it can be that risky. Which can be worrysome, because if I recall correctly these HP systems come with default settings that leave to the system to adjust "power" dynamically. Left to themselves they at least tinker with the CPU speed, but I am not sure about C-states.

So, it's not only better to leave that adjustment alone, but to check BIOS power management settings thoroughly in order to avoid nasty surprises.

4. There is a performance penalty when going into and out of states deeper than C1. This can add serious latency for some situations and could be a real killer for workloads that are intermittently creating workload. For home users you probably wouldn't be upset if it took a second for the server to 'wake up'. But in a production environment where you are doing things like iSCSI which is sensitive to latency from end-to-end that could be a real mess.

As much as a second? Wow. Anyway in my case it's just hitting C2, for some reason neither FreeBSD nor FreeNAS are detecting C3 as available.

5. Depending on your PSU's efficiency curve, it is actually possible to decrease the power usage of your motherboard, but see an increase in power usage at the wall because your PSU is less efficient at lower power. So for an unlucky group of people, they would actually see power usage increase!

Aha, that's interesting. The power supplies in these things seem to be rather goood, as the decrease in power consumption shows, although their power factor varies quite notiecably with consumption. Anyway I'm not a commercial customer, so I don't get a reactive power penalty.

6. We're talking about 4 watts from your graph. Yes, if you look at 24x7 usage, that comes back to an ideal savings of about 35kwh in a year. Or about $3 for most of us. If you were to swap out a 100w incandescent with a 13w CFL, and you only used that light bulb 2 hours a day (which is overly low), you'd save about 64kwh in a year, or about $6 or so. Is a savings of $3 worth the potential problems from #1, #2, #3, and especially #5? In my opinion, not really.

You are right, though in Spain it can be around 7 euro. If you run several servers it can be noticeable. Speaking of lamps, I am mostly using LEDs, here we pay for electricity almost double than you (and I hate the IR radiation of tungsten/halogens anyway) ;)

I'd also bet you aren't trying your hardest to keep lights off in your house, using CFLs everywhere it's reasonable, and doing your laundry by hand either. So 3 watts isn't exactly bleeding power either. FreeNAS isn't meant to be heavily tweaked for all hardware (if you want something that is highly optimized that's what TrueNAS is for). Yes, you can do it (and you may find it works just fine for you). I personally leave C1 state and I am plenty happy with that. If C3 makes you happy, feel free to stick with it. :)

Point taken, I stand corrected :)

If it works great for you, feel free to use it and feel better about the wattage saved. I'll stick with my C1 state. ;)

I agree :) Well, at least it was an interesting academic exercise ;)

Now if you'll excuse me I have to read up on these Ubiquiti mPower because they look very cool and I think I want one. ;)

[/quote]
You'll probably find them quite interesting, altough their management software is a pain in the ass with a collage of different software packages like MongoDB. That said, you can have an excellent power monitoring solution for a ridiculous price. And they have enough accuracy to let me know when it's time to clean an aquarium pump because of the variation in power draw.

 

[Borja Marcos]

Thank you and I think that this is good advice as long as people know when and where to use it.

Please note that Cyberjock's points to recommend not to use those power saving settings are more than compelling! I wasn't aware of those potential problems myself.

 

[cyberjock]

I'm all LED here too. I went all LED back in 2008 because I could. They were outrageously expensive (like $25-30 a bulb when on sale) but I love them. Zero burnouts in 6 years. :)

I did put a in my cart, so I'll have a new toy to play with soon enough. The temp sensors like seem totally cool, but they appear to require other hardware I'm not familiar with, so I'll pass on those until I can find more info on them. :P