PSU for 12 drives

[raid40000]

So, I have this board X11SSH-CTF with a (Skylake) Xeon E3 and I am planning to buy this PSU 500W seasonic, SS-500L2U.

I have 12 WD NAS RED (3TB) and maxed out RAM. Will 500W suffice ?

CPU is: 80 W TDP.
Drives are: 4W (let's say 10W)
Board: can be probably ignored
RAM: can be probably ignored
4 Fans: 12v, no idea amperage so say 3.6 to 5W each. 20W

So 4*12+80 = 128W + 20W + 50% safety margin 222W, is that accurate enough?

Note: yes, I can somehow check sensors or connect to the APC and get more precise numbers but... I do not have a proper PSU to test at the moment.

If the above is correct, a PSU gold rated, 500W should be enough but I would like to confirm.

The board supports up to 16 drives.

 

[Nick2253]

Did you review the PSU sizing sticky?

https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

 

[raid40000]

Did you review the PSU sizing sticky?

https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

Apologies I did not. I read now and I believe the getting the SeaSonic G-750 maybe a bit overkill and it would be a bit expensive compared. So if the 500W suffices, I think I am still good or did I made a mistake ?

It is a 2U server PSU what I need, not a desktop one.

 

[Nick2253]

You're underestimating the power needs of your drives by a factor of 2 at least. Spin-up will push you to at minimum 12W/drive (1A @ 12V), but more typically at least 20W (1.7A @ 12V). I would recommend at least 600W, but if you want to be safe (why else are you looking at FreeNAS?) you should get a 700+W PSU. If you can't find the G-750, there are other quality PSUs available.

 

[raid40000]

You're underestimating the power needs of your drives by a factor of 2 at least. Spin-up will push you to at minimum 12W/drive (1A @ 12V), but more typically at least 20W (1.7A @ 12V). I would recommend at least 600W, but if you want to be safe (why else are you looking at FreeNAS?) you should get a 700+W PSU. If you can't find the G-750, there are other quality PSUs available.

you may well be right, however, I checked the specs on Amazon it says 4W.

Also:
http://www.tomshardware.com/reviews/red-wd20efrx-wd30efrx-nas,3248-8.html
http://www.storagereview.com/western_digital_red_nas_hard_drive_review_wd30efrx says:

Power

• 2TB/3TB: read/write 4.4W, idle 4.1W, standy/sleep .6W
• 1TB: read/write 3.7W, idle 3.2W, standby/sleep .6W

So assuming it is a fair 12W that would be 144W + 80W CPU + 25W FANS + 48W RAM (way exaggerated) and motherboard we can say put the rest ? Seems feasible around 350-450W total with a single CPU considering it is Skylake, WD RED 3TB drives and DDR4.

I am not 100% sure to be honest. The desktop version I have for testing is 650W and is just like... not even noticing I have something plugged in.

 

[Nick2253]

Again, though, that's not spin-up power draw. Spin-up is significantly more demanding than idle or normal read/write operations.

Spin-up for most harddrives is in the 1.5A to 2A range on the 12V rail. Put another way, that's 18-24W. Per drive! Someone around here, I think it was @Bidule0hm did the actual measurements on WD drives, and they are in the 2A ballpark on spin-up.

If you want to underpower your system, it's your funeral, not mine. But keep in mind that the recommendations made here are not done trivially. There is room to maneuver if you really want to min/max your system, but think about the cost savings of buying a slightly oversized PSU vs buying a second PSU when your first one can't handle the load and blows out exactly one month outside of warranty.

 

[Bidule0hm]

Yep, the thread is here: https://forums.freenas.org/index.php?threads/how-to-measure-the-drive-spin-up-peak-current.38885/

And the 5 + 12 V currents are from 2.5 A to a bit more than 3 A peak so even if you take 0.5 A for the electronics (and that's very generous actually) that's still 2-2.5 A on the 12 V rail so about 30 W.

The thing is a desktop is not a server and a server is not a desktop machine. The server needs to be rock stable and reliable, all that while running 24/7 for years. Well, it's what I want from a server, maybe I'm wrong but I don't want a server die on me when I most need it so...

 

[Stux]

According to the specs, peak spinup is about 1.75A.

So at least 25W.

Per drive.


Ram is not an inconsequential amount of power, but everything pales in comparison to disk spinup.

 

[raid40000]

Bear in mind this server setup is a server "in budget", which means I am not aiming for a redundant PSU, just a 2U that can handle the load and will not deliver unstable voltages while in use or break randomly because of defects. Any 1tier or 2tier PSUs are good and the 500W Season is expected to be good.

@Stux and the others: Spin-up is a good point as noticed by @Nick2253 and I did not really considered when I posted so glad I did. Now, the reality is that 25W*12 is still 300W with a fair margin to the max. 500W rated PSU, isn't it?. Even with fans and CPU at max which is not the case during boot up. > 90% of the time the system will not be using 25W per drive.

Either way, the solution for spikes during boot would be staggered spin-up. Rather than getting 700W, and If I did I would go for a redundant PSU.

The 2U PSU I found in the 100€ to 150€ price range are just 500W to 750W but I would rather go for 450W Seasonic for 100€ than 1200W random unknown non-reviewed manufacturer for 100€. Plus, the UPS backing up the system is rated 600W.

Am I wrong in assuming that other than in boot the system will be using <400W most of the time ? I understand the concerns and the recommendations of getting 700W (or more) instead of the 500W, and I will reconsider it if you are telling me that the PSU will suddenly explode and destroy the entire array and the motherboard because it is totally under-powered sooner or later.

So even assuming 36W, 6 more than the calculated by Bidule0hm the 500W would handle it, even without staggered spin-up.

Finally, just wanted to link this because is exactly the number of disks I am using.

 

[Nick2253]

Am I wrong in assuming that other than in boot the system will be using <400W most of the time ? I understand the concerns and the recommendations of getting 700W (or more) instead of the 500W, and I will reconsider it if you are telling me that the PSU will suddenly explode and destroy the entire array and the motherboard because it is totally under-powered sooner or later.

You are correct that the system would be using less than 400W most of the time. But you are wrong about the rest of the system using less power at boot up.

The single most demanding load you put on a PSU is during bootup. The CPU will at least briefly run at full power until the BIOS/UEFI can check everything and put it into low power mode. The fans will draw their max current as they too spinup.

There's a reason @jgreco laid out the calculation you need in his sticky. The point is that you need to consider the worst case. So that means you cannot just pick and choose what you want to consider when planning your power needs.

As far as failure modes go, it's unlikely that your PSU would suddenly explode. While most well-made PSUs can handle a small overdraw for a brief period of time, that ability is greatly diminished as the components wear and age. What this means is that, as the PSU gets older, what worked once may not work. When a PSU gets overdrawn beyond its ability to handle that draw, the voltage on the lines will droop. If everything is as it's supposed to be, the PSU will catch the fault, and shut off to protect itself and the things connected to it. At best, that's a hard stop for your system (which is not a good thing). If they droop, then they could create a whole mess of problems with your system, up to and including frying stuff.

 

[raid40000]

@Nick2253 you are right, however, in this particular case I would say I am ok "risking" with 500W, 750W is for sure great but IMHO overkill for just 12 drives in a Skylake with an embedded LSI.

But the spin-up is totally a concern and it would not be wise to ignore it, fortunately seems I am covered on that regard via the Utility Setup for the LSI, am I right ?

http://imgur.com/a/tD0Ni (inline images not working for that link)

 

[Nick2253]

I'm not going to keep arguing with you about this, because it's your money, not mine.

If you want to trust that you have figured out how to min/max your system in a way that fits your risk profile and your budget, then more power to you. But if you want any of us to OK your decision, you probably won't get much support here, because the consensus is the recommendations in the sticky. And that consensus comes from years of collective experience on these forums.

Staggered spinup is great when it works, but that's only if it works. There are a number of different ways that vendors implement it, and not all are cross compatible. The most common is either PUIS (power up in standby) or using pin 11 signaling in a backplane. However, I couldn't figure out one way or another how your HBA does it. So until you test that it works, don't depend on it working.

Furthermore, you have to keep it in mind when you do any kind of BIOS/firmware updates, or anything that could knock out your existing setup. If you depend on staggered spinnup to protect your PSU, then you have the added burden of checking that it's working. It might only take one bootup without it to damage your PSU or something worse, especially if it happens when your PSU is 5+ years old.

 

[raid40000]

That will be fine, and I totally posted because of that consensus. It is not good to fully rely on staggering, agreed.

But considering the situation (and that I could not find a proper provider as for yet with the budget I have for the PSU) it will have to hold one way or another unless it just does not work. However, I will follow your advice and I will be gathering another proper PSU and eventually the 500W will be a backup one.

May be a bit silly not to do that now... agreed. But the cost of a SSN-6522G is more than 300$. So again, when I have some more budget for it I will improve it but I do not want to go for a random cheap PSU just because it is 750W as I know it will blow up, eventually.

thanks for the recommendations and the note on the spin-up, hopefully the staggering will work fine or I can make it work and would reduce any spike even if I had one with 750W.

 

[adrianwi]

For what it's worth, I'm running 9 drives with a 450W power supply and whilst it's probably drawing more than the 80% accepted level at startup, the power meter I've got it plugged into is generally drawing around 110-120W in normal operation. It's only ever shutdown/restarted for updates or upgrades which is very infrequently on the 9.3.1 train, and even then I don't think it's hitting 100%.

I did panic a little when I read @jgreco's thread, and whilst the advice I was given when I built my system with 6-7 drives was probably right, I suspect now it would be at least 100W higher.

I did try and do the math based on the component data sheets, and also found this which validated pretty much what I'd calculated (give or take 10%) >
http://outervision.com/b/vdYOfL

If I add any more drives I'll probably upgrade the PSU at the same time, but for the time being I'm fairly happy.

 

[raid40000]

I put the PSU 500W, I put a power meter on boot... it did not go past 300W, more like it increased up to 240W max. or so for some seconds. Then dropped, to the regular usage which is less than 200W by reading the PSU LCD considering there are other things plugged in. However, I do not think you can measure the actual spike on wattage without an oscilloscope as Bidule0hm did.

I am not sure the staggering is actually working because I never put the jumpers for the PUIS on the WD disks. I would say the spike on each disk is not 36W per disk though, I would say it is what the spec say which leaves a good margin for the power supply.

Most of the time is running below 50%, the fans on the PSU do not even turn on which considering it is meant to be a good PSU is better than having a random brand 750W PSU that normally are more like up to 750W.

 

[Spearfoot]

I put the PSU 500W, I put a power meter on boot... it did not go past 300W, more like it increased up to 240W max. or so for some seconds. Then dropped, to the regular usage which is less than 200W by reading the PSU LCD considering there are other things plugged in. However, I do not think you can measure the actual spike on wattage without an oscilloscope as Bidule0hm did.

I am not sure the staggering is actually working because I never put the jumpers for the PUIS on the WD disks. I would say the spike on each disk is not 36W per disk though, I would say it is what the spec say which leaves a good margin for the power supply.

Most of the time is running below 50%, the fans on the PSU do not even turn on which considering it is meant to be a good PSU is better than having a random brand 750W PSU that normally are more like up to 750W.

Your meter reading doesn't measure the transient startup current drawn by your system. Unless you're using an oscilloscope, any other type of measuring device will tend to dampen the measured initial spike in current at system start, which occurs in a very short time interval -- on the order of milliseconds (thousandths of a second). That's why you're only seeing ~300W. In reality, each time you boot your system it is drawing much more than 300W and you're stressing the PSU. It will work fine... for a while, until it doesn't. When will it fail? Tomorrow; next week; next year; five years from now; possibly never... there's no way to tell. How do I know all this stuff? I was educated as an electrical engineer before being seduced to the 'Dark Side' of computer programming.

You're determined to use a 500W PSU and that's fine; it's your money, your system, your data, and your decision. Just don't do so under the false assumption that you've proven us wrong with your power measurements and that we don't know what we're talking about. :)

 

[Bidule0hm]

Plus doing the measure at the input of the PSU is pretty much useless because the PSU have big capacitors who act as an energy storage (mainly to keep output rails stable when power is switched from mains to battery by an UPS for example) and by doing so they smooth the peaks.

NB: @Spearfoot actually it's more dozens of ms scale but I agree and power meters are usually doing averaging over hundred of ms anyway.

 

[raid40000]

@Bidule0hm @Spearfoot That would be correct and I already indicated that on my comment. I have to say that on the power meter takes up to 5 seconds to reach around 230W on the display then it goes down, 300W would be inaccurate.

I disagree though on the general advice that the "post it" is giving, let me explain myself:

Whereas the more wattage you put in the better and following the guidance, more than 2000W should be required to power up 32HDs in a case for less than 1s. Getting 750W on a bad PSU can be way worse than getting a 500W PSU that is properly rated and in Tier-1.

IMHO, the actual advice should be more like:

- Get a decent PSU, allow for some W margin but most important make sure you pick a Tier-1 PSU from a reputable make. Else throwing watts on it will not cut it and you can experience a failure over the long run either way.

- And always get hardware that supports for staggered spin-up, so even if you have 4 drives and more than enough wattage there is no need to stress the PSU and cause spikes in power.

- If you have serious availability requirements: get a redundant PSU.

Rather than suggesting pick a 750W PSU when 90% of the time you will only need less than 250W when the server is in use.

In my particular case the 500W, being honest, I do not think are surpassed during spin-up and as per the basic maths... the time will tell if it will 'explode' or not, but I feel confident than a random PSU 750W with the same cost as the one I got would eventually 'blow up' equally.

If you need 750W for 12 Drives, I wonder what people with 36 disks or more is actually using... I must assume they do staggered spin-up or some other solution.

This is meant to be a constructive comment, not trying to be harsh or saying the general recommendation is not correct regardless.

 

[MatthewSteinhoff]

The difference between theory and practice is greater in practice than in theory.

A brand new, freshly-lubricated, perfectly temperature balanced hard drive on the test bench draws fewer watts than an older hard drive with dried-up lubricant that has been repeatedly temperature-cycled.

A drive that spins up just fine on day one at 1.5 amps may not spin up with three amps after a couple years.

You came here for expert advice and the experts say 750 watts will give you the greatest chance of success. That said, you're not going to hurt my feelings if you want to use a 400-watt power supply.

Show : Tangential war story from an old IT guy...

We had an IBM RS/6000 server at the newspaper that kept a text copy of every story that ever ran in the newspaper. That server had an uptime of over four years and hadn't been fully powered down in the seven years I worked there. It held five 4.5 GB (yes, GB not TB) drives.

One weekend, we had to replace our aging UPS system which required everything in the data center to be powered off for five hours.

Those hard drives got cold for, likely, the first time since the system was installed. The lubrication turned rock-hard. We blew out one of the two power supplies when we turned it on -- too much power draw.

After swapping in a spare power supply, the drives still wouldn't spin up. We brought them up to temperature (85 degrees versus the 55 degrees the data center reached with nothing running) and three of the five spun up.

Called IBM support. Normal troubleshooting. Finally, the guy said...

"Look, this is going to sound crazy but... take the three stuck drives out, set them on the floor, bounce them on the raised floor a couple times - half an inch at most - then spin the whole drive like a top flat on the floor. Then quickly put it back in the machine. One at a time do that then reboot the machine."

Sure enough, it worked. Warming, thumping and spinning the drives loosened the platters enough that they would spin up.

Vocabulary word of the day: stiction.

Cheers,
Matt

 

[raid40000]

The difference between theory and practice is greater in practice than in theory.
...
A drive that spins up just fine on day one at 1.5 amps may not spin up with three amps after a couple years.
...
Cheers,
Matt

It should. You purchased the wrong PSU otherwise, even if you got way more wattage it was still a far worst purchase.

A good PSU may stop working, sure. However, a tier1 "platinum" rated PSU will simply *NOT* do that. Ever.

If disks start doing weird things... replace them, I suppose. The current level of electronics will not simply move from requiring 1.72A to suddenly 3A or else a modern graphic card would burn whole the computer after 5 years. If the drive is faulty is faulty.

Saying "go for 750W instead of 500W" is something I can understand but it is not that simple and IMHO it is better to go for a tier 1 500W than a tier 2 or tier 3 random PSU on 1000W instead. Over the years the 1000W will fail, the other one will not. (A PSU that will be running 90% of the time under 50% of the max wattage will not fail because two or three random spikes in power unless you truly under-size the PSU.)

And for the record, I got a PSU on my first attempt - brand new - it was faulty (turning on the system but the SAS controller randomly not being powered up) with no disks attached. Then I got the Seasonic there, supposed to be better and no problems.

The guys mentioned me about the spin-up and they were totally right, now on the general advice I think is just too general and simplistic. In my case 500W covers the requirement for the spin-up... sure 750W would cover more, and so on. But I have low budget, thus, I wanted to make sure it is not mandatory to get a way over-sized PSU.

I believe recommending hardware that supports staggered spin-up is probably a better general advice, regardless of the PSU that you have.