Does TrueNAS supports any kind of power management/power savings?

[MonkY]

Hello,

I have just "upgraded" about a week ago from a Synology NAS (with 8 + 5 drive bays) to a Supermicro with 22 drive bays, and some of my friends suggested i should go with TrueNAS and zfs. I have created 2 pools (one using 12 SAS Enterprise 7200 RPM drives, and another one using 10 SATA WD Red Drives - i believe they are 5900 RPM or something like that).

The system was build with specially as a NAS, therefore the specs are kind of low (mostly to keep down the noise and the power consumption). It's a Xeon E3-1270 v3 with 32GB RAM. I was expecting a higher power consumption (after all, the drives are about double in number), but I was amazed to see the new NAS consuming more than 3 times compared to my old Synology. Before I was barely touching 100W/hr (with both NAS & Expansion ON), and now I'm using more than 350W/hr. In "idle".

I know, for some of you, power consumption wouldn't matter that much, but for me, that means a lot. And not only because of the price, but also because I have often power failures in here, and my UPS's must keep them as long online as possible. I also only use the NAS a few times a day, therefore it's just not productive to use that much power all day - but it's very important to be accessed as fast as possible - I was thinking about shutting it down and turning it on every time I need the NAS, but it's taking like 7-8 minutes for it to boot completely.

Anyway, the main issue I have (sorry, I'm kinda noob when it comes to Linux of FreeBSD), is that my SAS pool never goes into idle. The SATA drives do, and that's it. Also, i have not found a single way to put TrueNAS into sleep mode, hibernate, standby or anything like that. It's acting more like a SERVER than a NAS, and for me that's really bad.

I have tried in Storage - Disks - Edit Disk to set all my SAS drives as follows:
HDD Standby: 10
Advanced Power Management: Level 1 - Minimum power usage with Standby (spindown)
Force HDD Standby: Enabled
Acoustic Level: Minimum
Enable SMART: Enabled

They are the same for both the SAS and the SATA drives both pools, but only SATA enters in standby. Why? What else can I do?

I have also disabled the SMART service, with no success. I mean, it was disabled, but the disks kept on spinning.
I have moved my Plugins and Jail storage to a different pool, on a SSD array. That didn't helped either.

So, can I get some help over here? Is there a way to make all drives spin down? Is there any chance to put my TrueNAS computer in SLEEP mode, HIBERNATE, STAND-BY, SUSPEND TO RAM, S5 or whatever?! Or is it a lost cause, and TrueNAS actually is dedicated to 24/7 online servers?

Thanks in advance!

 

[Samuel Tai]

Most drives nowadays no longer support APM. See https://www.truenas.com/community/threads/playing-with-disk-settings.86601/post-600158 for how to check.

As for sleeping/hibernating, TrueNAS is based on FreeBSD, which doesn't have great support for this.

 

[NugentS]

"10 SATA WD Red Drives "
Which model(s) specifically? If they are SMR then do not use them

 

[jgreco]

Also, i have not found a single way to put TrueNAS into sleep mode, hibernate, standby or anything like that. It's acting more like a SERVER than a NAS,

Uh, a NAS *is* a server. It isn't clear to me what you think a NAS is that wouldn't be a server.

Is there any chance to put my TrueNAS computer in SLEEP mode, HIBERNATE, STAND-BY, SUSPEND TO RAM, S5 or whatever?! Or is it a lost cause, and TrueNAS actually is dedicated to 24/7 online servers?

Putting a NAS to sleep/etc is very weird. Most NAS's need to be able to respond to people accessing them, so they are powered on 24/7. Many of the QNAP/Synology/etc units happen to be low power ARM CPU's. which idle at low power consumption, but remain powered on to interact with the network. The hard drives may be spun down, but the NAS system itself isn't doing any of the things you describe. People get very annoyed when their fileserver "hangs" because a session is open to it but the thing has gone offline. Therefore the vast majority of NAS units are set to merely power down their drives, if they can.

FreeNAS/TrueNAS is not designed to do any of the power saving mode things you describe either. For better or for worse, ZFS is a CPU-hungry and RAM-hungry beast designed for large scale fileservice, and so the systems that drive FreeNAS are typically more power-hungry than a 5-10 watt ARM NAS platform. So, when FreeNAS is on, expect to be burning somewhat more watts. And just like the ARM NAS units, it needs to be on to be able to interact with network clients.

Additionally, FreeNAS/TrueNAS is really a free variant of iX's commercial TrueNAS offering, which is targeted at large business NAS needs. In these environments, you typically do not ever power down the NAS, both because operations tend to require it for various things, and because it is risky to shut it down. Therefore there is not any particular focus on developing power-saving features. Modern CPU's are already very power efficient when idle.

What does generally work is that you can shut down your NAS when it isn't being used, and use WOL to start it when you'd like it to. This requires an ethernet chipset and mainboard that correctly supports this feature. The main problem here is that randomly shutting off your NAS interferes with the normal maintenance such as scrubs and SMART tests that should be regularly performed to make sure your NAS is healthy.

As for sleeping/hibernating, TrueNAS is based on FreeBSD, which doesn't have great support for this.

Well, really, only Windows has "great" support for this, and that's mainly just because it is the default thing a mainboard manufacturer tests against. However, FreeBSD on a proper server board does indeed generally support some power saving modes, and you can look into whether the zzz command actually works and how to configure it. The problem isn't that it doesn't work, it's that it isn't integrated cleanly and is basically creating a very messy environment.

Only Apple has really developed anything credible in this direction, IMO (thinking about the Bonjour Sleep Proxy).

 

[MonkY]

Most drives nowadays no longer support APM. See https://www.truenas.com/community/threads/playing-with-disk-settings.86601/post-600158 for how to check.

As for sleeping/hibernating, TrueNAS is based on FreeBSD, which doesn't have great support for this.

Unfortunately, executing command camcontrol identify drive-id on each drive returns me these errors:

camcontrol: ATA ATA_IDENTIFY via pass_16 failed
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

 

[MonkY]

"10 SATA WD Red Drives "
Which model(s) specifically? If they are SMR then do not use them

I think 5 of them are WD Red (WD60EFRX) and 5 of them are WD Red Plus (WD60EFZX). I have no ideea if they are SMR or not.

 

[flashdrive]

disclaimer: I haven't read all the comments.

@MonkY

"Or is it a lost cause, and TrueNAS actually is dedicated to 24/7 online servers?"

tldr: yes; forget standby. Have a look for another distro which is supporting this: Unraid (maybe? I don't remember) or https://www.openmediavault.org/


After having the same scenario as you considering: low NAS data usage, save power consumption I am now at the point of giving up this standby topic.

There are many post from me as a reminder to myself, the latest being:

Howto for myself - pool degraded, what to do

as per GUI the pool is degraded theory: 1) HDDs are not capable for long time run NAS 2) HDD spindown is affecting this negativly 3) HDD vibrations are too much for the system to stay always online Howto: Check all physical connections (power and data) unplug and replug check BIOS / EFI if...

www.truenas.com

Today is the 2nd time my pool got degraded after last Sundays event.

My suspicion is about the HDDs going into standby mode and not recovering in time (non NAS HDDs). So they get kicked out by the system, not even shown in EFI anymore.

The time and maintenance I have to spend to:
a) physically open the host
b) reconnect the ejected drives
c) resilvering and checking for the health of the pool
d) downtime

is not worth the additional expense in terms of energy cost to me.

Given in my example the difference is 20 Watts...

What works fine is to shutdown and restart the host during the night by the system itself.

 

[flashdrive]

@MonkY

Get rid of any SMR drives asap:

On WD Red NAS Drives - Western Digital Corporate Blog

On WD Red NAS Drives - Data - Western Digital Corporate Blog

blog.westerndigital.com

 

[MonkY]

Uh, a NAS *is* a server. It isn't clear to me what you think a NAS is that wouldn't be a server.

NAS = Network Attached Storage. It doesn't automatically imply (for me at least), that it's a dedicated server required to be up & running 24/7, even if not in use. Specially for home use, like I do. I don't see the point of consuming so much resources when it's not beeing used.

Putting a NAS to sleep/etc is very weird. Most NAS's need to be able to respond to people accessing them, so they are powered on 24/7. Many of the QNAP/Synology/etc units happen to be low power ARM CPU's. which idle at low power consumption, but remain powered on to interact with the network. The hard drives may be spun down, but the NAS system itself isn't doing any of the things you describe. People get very annoyed when their fileserver "hangs" because a session is open to it but the thing has gone offline. Therefore the vast majority of NAS units are set to merely power down their drives, if they can.

It might be weird for you, but everyone has different needs. And I don't agree with your "Synology units happen to be low power ARM CPU's", as I have at work a Synology NAS powered by Xeon, which goes into "hibernate" after 15 minutes of not being used. The drives are set to spin down after 10 minutes in idle. At home, i've been using the same scenario for over 8 years, and I haven't had a single drive failed on me. Bonus: everything is silent, consumption is almost zero. When not in use, of course.

Regarding "people get very annoyed when their fileserver "hangs" because a session is open", i disagree again. The NAS wouldn't go into sleep mode if there is any SMB, FTP or NFS session opened or files being in use. So there is nothing to hang.

FreeNAS/TrueNAS is not designed to do any of the power saving mode things you describe either. For better or for worse, ZFS is a CPU-hungry and RAM-hungry beast designed for large scale fileservice, and so the systems that drive FreeNAS are typically more power-hungry than a 5-10 watt ARM NAS platform. So, when FreeNAS is on, expect to be burning somewhat more watts. And just like the ARM NAS units, it needs to be on to be able to interact with network clients.

Unfortunately, I found out the hard way. I've been reading the changelogs of TrueNAS before I have installed it and then saved over 50TB of data... There are a lot of entries in the changelogs regarding power save features, specially on the drives. Also, why would TrueNAS actually have settings for disks like "HDD Standby", Advanced Power Management and Force HDD Standby if it would be as you said "not designed to do any of the power saving"?

What does generally work is that you can shut down your NAS when it isn't being used, and use WOL to start it when you'd like it to. This requires an ethernet chipset and mainboard that correctly supports this feature. The main problem here is that randomly shutting off your NAS interferes with the normal maintenance such as scrubs and SMART tests that should be regularly performed to make sure your NAS is healthy.

I would consider it shut down and wake it with WOL if the Supermicro board i am using wouldn't be such a pain regarding the boot times. Sometimes it even takes up to 10 minutes to completely boot. And that's way too slow for my needs.

Well, really, only Windows has "great" support for this, and that's mainly just because it is the default thing a mainboard manufacturer tests against. However, FreeBSD on a proper server board does indeed generally support some power saving modes, and you can look into whether the zzz command actually works and how to configure it. The problem isn't that it doesn't work, it's that it isn't integrated cleanly and is basically creating a very messy environment.

If Windows would've had zfs support, I would quit using TrueNAS in a second, and replace it with Windows Server or something. I am actually sorry I haven't tried openmediavault, which supports all kinds of power management, WoL, etc. But for now, it's really complicated, as both of my pools are almost full, and would be a real pain to move it on some other drives. Therefore, I'm still having hopes for TrueNAS...

 

[MonkY]

@MonkY

Get rid of any SMR drives asap:

On WD Red NAS Drives - Western Digital Corporate Blog

On WD Red NAS Drives - Data - Western Digital Corporate Blog

blog.westerndigital.com

Thank you for the link. But I haven't experienced any issues with them. And some of them I use for more than 3-4 years. I don't even think they are under warranty...

 

[MonkY]

disclaimer: I haven't read all the comments.

@MonkY

"Or is it a lost cause, and TrueNAS actually is dedicated to 24/7 online servers?"

tldr: yes; forget standby. Have a look for another distro which is supporting this: Unraid (maybe? I don't remember) or https://www.openmediavault.org/

Thank you. I am really sorry now I haven't tried openmediavault first. But as it doesn't support zfs, it would be almost impossible for me to migrate to it...

After having the same scenario as you considering: low NAS data usage, save power consumption I am now at the point of giving up this standby topic.

There are many post from me as a reminder to myself, the latest being:

Howto for myself - pool degraded, what to do

as per GUI the pool is degraded theory: 1) HDDs are not capable for long time run NAS 2) HDD spindown is affecting this negativly 3) HDD vibrations are too much for the system to stay always online Howto: Check all physical connections (power and data) unplug and replug check BIOS / EFI if...

www.truenas.com

Today is the 2nd time my pool got degraded after last Sundays event.

My suspicion is about the HDDs going into standby mode and not recovering in time (non NAS HDDs). So they get kicked out by the system, not even shown in EFI anymore.

The time and maintenance I have to spend to:
a) physically open the host
b) reconnect the ejected drives
c) resilvering and checking for the health of the pool
d) downtime

is not worth the additional expense in terms of energy cost to me.

Given in my example the difference is 20 Watts...

What works fine is to shutdown and restart the host during the night by the system itself.

For more than 8 years I've being using Synology NAS's with tons of disks. The disks would spin down, the NAS would go into some hibernate state... I have never had a failed disk. I have some clients that did, but I was fortunate. Also, the power usage savings are huge. If a fully running NAS would use around 150-200Watts, they hardly use 20-30Watts with drives in standby. And almost no Watts while in hibernate (I'm guessing it's under 5W or so). But I had to gave up on Synology because of two reasons: 1. The costs (every upgrade it's killing me). 2. The support (on my last ticket they told me they don't have any oficial service in Europe - wow!).

 

[NugentS]

I think 5 of them are WD Red (WD60EFRX) and 5 of them are WD Red Plus (WD60EFZX). I have no ideea if they are SMR or not.

All of those are CMR (according to nascompares.com) - so all good

 

[Borja Marcos]

Depending on the motherboard, firmware and processor you can achieve some power savings (especially with hungry CPUs) using the processor C-states and powerd.

However, as I was told in this forum years ago, do it at your own risk. In my particular case, a Microserver Gen8 with a Xeon E3-1260L, it works and the only cost is slightly higher jitter for NTP and ping responses, for example, and nothing else.

But again consequences can be disastrous if you are using hardware that is incompatible for whatever reason. And the power saving won't be dramatic either,

This is my original thread. Again, the warning was clear!

Tale of two servers and power consumption, the daemon and the shark

Hi :) I currently have two identical servers. They are HP Microserver Gen8 (the basic configuration) with 10 GB of memory and four WD Red 3 TB drives (different firmware and "NASWare" version, whatever that means). One of them is running FreeBSD 10.1-STABLE and the other one FreeNAS 9.3...

www.truenas.com

 

[flashdrive]

@MonkY

giyf OMV + ZFS

I have not tried, but have a look for yourself, e.g.

Reddit - Dive into anything

www.reddit.com

When I was evaluating distros for my diy NAS build I was using a virtual machine to have a look into all of those.

Simulating different scenarios like removing a disk and see..

Also in retrospective:

The amount of time I have invested in the diy build is in no relation to the outcome

It would have been cheaper for me to buy a 4figure NAS of the shelf.

 

[MonkY]

Depending on the motherboard, firmware and processor you can achieve some power savings (especially with hungry CPUs) using the processor C-states and powerd.

However, as I was told in this forum years ago, do it at your own risk. In my particular case, a Microserver Gen8 with a Xeon E3-1260L, it works and the only cost is slightly higher jitter for NTP and ping responses, for example, and nothing else.

But again consequences can be disastrous if you are using hardware that is incompatible for whatever reason. And the power saving won't be dramatic either,

This is my original thread. Again, the warning was clear!

Tale of two servers and power consumption, the daemon and the shark

Hi :) I currently have two identical servers. They are HP Microserver Gen8 (the basic configuration) with 10 GB of memory and four WD Red 3 TB drives (different firmware and "NASWare" version, whatever that means). One of them is running FreeBSD 10.1-STABLE and the other one FreeNAS 9.3...

www.truenas.com

After running the command in there (sysctl hw.acpi.cpu.cx_lowest=C3), i got this result:
hw.acpi.cpu.cx_lowest: C8 -> C3
But then, nothing happened. The power consumption remains the same. Is there anything more I can do or check for?
Do I need to run this after each reboot?

 

[flashdrive]

@MonkY

"But then, nothing happened. The power consumption remains the same. Is there anything more I can do or check for?"

Again I advise against trying to "force" TrueNAS to do something which is not going to work from my experience.

I would be glad if you could figure out a working scenario. If you are willing to spend the time investigate: that's your decision.

I have stopped putting in more time for my own host.


The other idea would be: what is consuming this much power in your host in the first place? Do you really need such a system?

I did purchase a 6bay dell 2U rack server once only to figure out after the purchase that the power cost of running that thing was prohibitive high, apart from the noise impact.

edit:

have a look into this ongoing discussion:

Low Power DIY TrueNAS Build

Hello together, I registered here as a new TrueNAS user, at home I currently have a Synology NAS, but it is deprecated and will not get any (security)-updates any more, so thats why I plan to migrate to a DIY low power TrueNAS :-) The main use cases are to share movies/pictures on TV, and...

www.truenas.com

 

[Borja Marcos]

I have been doing some tests and the most effective way to reduce consumption is to reduce clock speed.

Storage is I/O bound rather than CPU bound unless you need a lot of throughput.

You can reduce CPU frequency using the powerd daemon flags. Also, remember that there are important differences between CPUs. My MicroServers use low power Xeons, for instance.

You can try this:

First, set this tunable (loader.conf) to 1: hint.p4tcc.0.disabled

And reboot. I read somewhere long ago that it limits the available clock frequencies to the "native" frequencies supported by the CPU.

Once rebooted, open a command line and use this command:

Code:

% sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 2401/45000 2400/45000 2300/42000 2200/40000 2100/38000 2000/35000 1900/33000 1800/31000 1700/29000 1600/27000

The result is a list of supported frequencies together with the power TDP associated with them. For example in my case "2401" is associated to 45 W of TDP.

Why 2401 when there is a 2400? Because 2401 is the "turbo boost" frequency. You must look up the turbo boost frequency value for your particular processor. Mine is a Xeon E3-1260L and the turbo boost frequency is 3.30 GHz.

If you want a reduction in CPU power consumption you can change the maxium clock frequency to 1.6 GHz.

How? Set up a powerd_options tunable. The -M parameter is the frequency limit you want to configure. In my case, 1.6 GHz.

You can see the effect on the attached power consumption graph. Between 10:20 and 10:30 am I had my usual configuration (letting Turbo Boost work). From 12:30 or so, after limiting the maximum clock frequency, I have started the same services (I run quite a lot of stuff including an Elasticsearch node). You can see the difference, those high power bursts have almost disappeared.

The success and whether it is worth the hassle or not will depend on your hardware.

And anyway do not expect higher power reductions than maybe 10% or so. As you can see on the indexing latency graph, I have reduced performance.

 

[MonkY]

Thanks Borja Marcos, but I am pretty sure my main power consumption comes from the 24 drives (actually I have around 28 if i count the internal ones). It's not the CPU as it sits most of the time in idle frequencies. Also, even if i fully load the CPU, the difference is minimal (like 10-20W or so). But the drives... when I manage to put my SATA drives in idle, i save around 80W. The issue is with the SAS drives, who don't want to go into sleep mode, no matter what I do. I am guess they are using more than 100W. So, just by making the drives sleep, I would save 180W. But not just that, I would keep the temperature and the noise down - which is perfect, as my purpose is not to use this unit 24/7, but only a few times a day.

So, I'm still looking for WHY IN THE WORLD MY SAS DRIVES WOULDN'T JUST GO TO SLEEP?!? :)

 

[Samuel Tai]

So, I'm still looking for WHY IN THE WORLD MY SAS DRIVES WOULDN'T JUST GO TO SLEEP?!? :)

Because SAS is a data center storage technology, and that use case has no requirement for sleep, as that would vastly worsen disk latency and wear for little to no economic benefit.

 

[MonkY]

Because SAS is a data center storage technology, and that use case has no requirement for sleep, as that would vastly worsen disk latency and wear for little to no economic benefit.

If that would be so, then why the same drives would sleep in my Windows Server environment, but not in TrueNAS? In Windows Server it's not "data center storage technology" anymore? Also, as I've said before, why would TrueNAS permit settings for SAS like "HDD Standby", "Advanced Power Management" and "Force HDD Standby", if that's not working at all? I honestly don't think it's the drives fault... there is something happening in TrueNAS, preventing them from going into standby. And that's something I want to fix with my NAS, nothing more...