Dual PSU or bigger PSU?

[Stilez]

Show : See signature or this collapsed section, for system details + specs

Supermicro X10 + E5-1620 v4
RAM: 128GB (6 module) 2400 ECC
HBA: 2 x LSI 9211
NIC: Chelsio T420 dual 10G (foreseeable max: 2)
Fans: 9 120mm AP121 (foreseeable max: 12)
PSU: EVGA Supernova G2 750W (underpowered, to be upgraded)
Storage:
16 x 7200 enterprise, mixed SAS/SATA (foreseeable max: 22-24)
Usage: 3 way mirrors (3x4TB+3x6TB+3x6TB+3x8TB) + test pool/spares, main pool capacity 22TB @ 55-60% usage, dedup saving 3.9x
ZIL: Intel P3700 NVMe
L2ARC: Samsung 250gb NVMe
Boot: 2 x Intel 320 SSD mirror
Power draw/temp: staggered start enabled, vertical fanned, disk temps 28-35C

I'm unhappy about the PSU in my current build and plan to upgrade that area. It's anEVGA Supernova G2 - single rail, accolades for build quality, stability and quality, like many of EVGA/Superflower's current lineup, but only 750W and I'm running way more disks than I originally planned, to the point that I'm not sure how it hasn't fallen over yet. I don't feel I want to chance my luck further so some kind of upgrade is going to happen.

I've enabled staggered startup on all but boot drives, but the HDDs are mostly rated as drawing up to 1.5A on 5v and 3.08A on 12v as a worst (peak/transient) case. There are currently 16 HDDs and potentially I can see a future need for up to 22-24 disks, although perhaps in 2 pools and not all active at the same time. They are all directly powered, there's no separate backplane. There are also significant power-drawing components powered from the baseboard - 10G cards, high performance SSDs, ECC, CPU, and fans.

Being fair, 90% of the time the NAS isn't doing heavy work, but even ordinary work can do a lot - major file copying/moving, scrubs, resilvers, multitasking, and writing data from clients. I've had Samba working at 1GByte/sec writes, and the pool has potentially got up to 18 HDDs in it + ZIL + L2ARC now (and could have more than 20 in future), so if they all get writing, there's a fair load.

Working out my current needs based on absolute peak current draw, I'm beyond the 1200W level and looking at a 1600W - 2000W PSU when one allows for overhead and reduced performance over its lifetime (1300-1400W on 12v and potentially up to 36A transient draw on 5v if every one of 24 disks needed its peak current simultaneously, although that's unlikely). But as it's already been handling 16 HDDs on a 750W PSU for months, asserting that 1200W isn't enough seems excessive. I'm not sure what to think.

I'm also especially wondering if it's safe to use two 750W as dual PSUs, and split the HDDs between them, rather than a monolithic 1600W. I can't get the kind of dual-PSU case one might get if doing this in a business, so it's either big or dual. These are the pros and cons I can think of:

• Big PSU (1200W or 1600W, platinum): "Usual solution" hence known to work. But expensive for 1200W and very expensive indeed for 1600W (£320-£400 for EVGA and £380-480 for Superflower), here in the UK. 2000W is plain unaffordable at those rates :) There's a second problem which is that these large PSUs are mainly designed for driving multiple graphics cards, so the 5v rail only goes up to 24-30A - by contrast my load is disproportionately heavy on 5v since I'll have up to 20-24 HDDs with specs giving up to 1.5A @ 5v each for peak draw, plus anything else needing it. So even a very large monolithic PSU might struggle with the 5v draw in some extreme cases, even if perhaps unlikely. Also the larger EVGA models only have 5 cable ports for drives so I'd have to load every port fully which isn't ideal for cable heating and maximum current per cable limits.
  
• Dual PSU (750W + 750/850W): Ideal in every way except for my big unresolved concern - I'm apprehensive whether it's as safe as a monolithic PSU, insofar as I've only ever used a single PSU. Then again the electrics seem sensible - make the power connection common to both (for on/off function), crosswire the ground planes (common ground), and use one for baseboard + 1/3 of the disks and the other one for the other 2/3 of the disks. I'd buy another of the same brand/line as I already have. If buying a dual PSU cable also need to check the pins are done right, but there are resources for this online, and with bitcoin mining rigs this isn't as freakish/uncommon as it once was. It also gets me 8-10 cable ports rather than 5 to split my HDDs across, it immediately gets me up to 48A on 5v as well as sufficient 12v, and perhaps lower fan noise (heat split between 2 units and 2 less-loaded fans). Since the PSUs only have 0v in common and one only powers the baseboard+ some HDDs while the other only powers the remaining HDDs, there isn't apparently scope for one to take all the load leaving the other idle, or for current loops, but I don't have the knowledge to exclude the risk of other interactions such as ground plane, or supply-side earthing, or the data cables coming from a board supplied by PSU1 and connected to a HDD powered by 5v/12v (from the same 0v plane) from PSU2. That sort of thing.

Regarding dual PSUs - I'm confident of my ability to check the wiring and wire it up, but what are the running implications of two high quality PSUs paralleled this way, both providing 5v and 12v to different components on the same system? There shouldn't be current loops but electrically I'm not sure what the implications would be. Could there be much/any risk to my hardware and drives? Or is it pretty much totally safe given some technical knowhow and solid high quality cross-connections? Is it basically the same as any other redundant PSU system and really 100% straightforward/normal?

Otherwise if it's monolithic, what wattage is likely to be sufficient?

 

[ethereal]

you'll need to put your system specs in the post so everybody can read it

 

[Stilez]

you'll need to put your system specs in the post so everybody can read it

They're already in the sig and also pointed to at the start of the 2nd para, but I collapsed them to not clutter up everyone's screen on every post I do. Thanks though, I'm hoping that is enough to be clear.

 

[SweetAndLow]

They're already in the sig and also pointed to at the start of the 2nd para, but I collapsed them to not clutter up everyone's screen on every post I do. Thanks though, I'm hoping that is enough to be clear.

The point is no one can see them. Signatures are not displayed on some platforms.

Have you tried monitoring your power usage? I have 24 drives and my peak usage is 338W. Check ipmi for that info. I don't think with 24 drives your 750w power supply will be overloaded.

 

[Stilez]

The point is no one can see them. Signatures are not displayed on some platforms.

Ahh.. thank you. I've added it inline at the top of the post as well - it now makes sense.

Have you tried monitoring your power usage? I have 24 drives and my peak usage is 338W. Check ipmi for that info. I don't think with 24 drives your 750w power supply will be overloaded.

No, and although that would be ideal, I'm unfortunately sceptical whether any usual power meter or IMPI would be adequate either. 3 obstacles:

1. Needs a suitable pickup to measure current draw on the HDD side of PSU, and equipment with millisecond-scale capture-and-hold/peak record capability - To measure at the HDD side not the supply side of the PSU, you either need a capable lab pickup (to read current draw on the required resolution on a contactless basis within multiple insulated 5v/12v cables simultaneously) or modded PSU cables with breaks (for inserting multiple ammeters) or something.
   Also to measure any momentary 5v/12v peak draws that exist, you'd need a pretty good recorder/analyser/oscilloscope to record and analyse the data, or some other decent lab equipment (and knowhow) capable of capturing transient current draw peaks at the HDD side of a PSU that's supplying 20-odd 4-pin/SATA drives across 4-5 cables. A transient could be a fraction of a second.
   
2. Knowledge of scenarios for testing - Assuming one can technically capture and record peak current data, you need to know the kinds of scenario that might tend to produce it. From the datasheets all we know is that in some scenarios, the HDD's motor and logic circuits can momentarily draw 3x the current on both 5v and 12v that it draws at maximum on an ongoing basis under heavy load. They could all do the same at once, in very unfortunate circumstances. But there's no further data on likelihood, duration, circumstances or anything else. It might be several times a week, thousands of times an hour but only with some kinds of load/collisions/RW timings, or once a decade. We just don't know how to test it in any meaningful way to get a useful figure.
   
3. Data not captured/provided in digital form AFAIK (IPMI etc) - I don't think IPMI captures it. The HDD current draw just doesn't go anywhere near the baseboard for IPMI to get it directly, so only the PSU and HDD can detect it. The PSU doesn't speak to IPMI, and data captured by the drive doesn't capture transient 5v + 12v current draw peaks. If you know of a way that drives provide maximum millisecond-based peak amperage draws to the baseboard controller, or a way that IPMI logs maximum aggregated transient current draw for all drives simultaneously on a millisecond basis, then I'd love to know of it. It seems so unlikely that I had excluded it from my thoughts.

My baseboard does have IPMI but the SAS drives don't provide SMART attribute info, much less current info, and I don't have any electronic measurement devices beyond a normal hobbyist voltmeter for simple needs, which can't detect or hold peaks of any kind, let alone millisecond/centisecond, nor do I know how to test it or what R/W scenarios tend to trigger the current peaks referenced in the datasheets.

I think it's a good idea in principle, and blogs like 45drives have managed to graph it empirically; certainly a manufacturer would do it, but I think it's well beyond an ordinary user like me :( Therefore it looks like my only practical solution is to take the datasheet figures on faith, and ensure I meet the drives' power needs, hence my question..... :)

 

[SweetAndLow]

Ahh.. thank you. I've added it inline at the top of the post as well - it now makes sense.

No, and although that would be ideal, I'm unfortunately sceptical whether any usual power meter or IMPI would be adequate either. 3 obstacles:

1. Needs a suitable pickup to measure current draw on the HDD side of PSU, and equipment with millisecond-scale capture-and-hold/peak record capability - To measure at the HDD side not the supply side of the PSU, you either need a capable lab pickup (to read current draw on the required resolution on a contactless basis within multiple insulated 5v/12v cables simultaneously) or modded PSU cables with breaks (for inserting multiple ammeters) or something.
   Also to measure any momentary 5v/12v peak draws that exist, you'd need a pretty good recorder/analyser/oscilloscope to record and analyse the data, or some other decent lab equipment (and knowhow) capable of capturing transient current draw peaks at the HDD side of a PSU that's supplying 20-odd 4-pin/SATA drives across 4-5 cables. A transient could be a fraction of a second.
   
2. Knowledge of scenarios for testing - Assuming one can technically capture and record peak current data, you need to know the kinds of scenario that might tend to produce it. From the datasheets all we know is that in some scenarios, the HDD's motor and logic circuits can momentarily draw 3x the current on both 5v and 12v that it draws at maximum on an ongoing basis under heavy load. They could all do the same at once, in very unfortunate circumstances. But there's no further data on likelihood, duration, circumstances or anything else. It might be several times a week, thousands of times an hour but only with some kinds of load/collisions/RW timings, or once a decade. We just don't know how to test it in any meaningful way to get a useful figure.
   
3. Data not captured/provided in digital form AFAIK (IPMI etc) - I don't think IPMI captures it. The HDD current draw just doesn't go anywhere near the baseboard for IPMI to get it directly, so only the PSU and HDD can detect it. The PSU doesn't speak to IPMI, and data captured by the drive doesn't capture transient 5v + 12v current draw peaks. If you know of a way that drives provide maximum millisecond-based peak amperage draws to the baseboard controller, or a way that IPMI logs maximum aggregated transient current draw for all drives simultaneously on a millisecond basis, then I'd love to know of it. It seems so unlikely that I had excluded it from my thoughts.

My baseboard does have IPMI but the SAS drives don't provide SMART attribute info, much less current info, and I don't have any electronic measurement devices beyond a normal hobbyist voltmeter for simple needs, which can't detect or hold peaks of any kind, let alone millisecond/centisecond, nor do I know how to test it or what R/W scenarios tend to trigger the current peaks referenced in the datasheets.

I think it's a good idea in principle, and blogs like 45drives have managed to graph it empirically; certainly a manufacturer would do it, but I think it's well beyond an ordinary user like me :( Therefore it looks like my only practical solution is to take the datasheet figures on faith, and ensure I meet the drives' power needs, hence my question..... :)

Dude you are way over thinking this. Just buy a kill-a-watt for $20 and you will know your estimated power usage. Either you are close to your limits or you're not. I'm pretty sure you're not but since you added your hardware specs I see your using 7200rpm drives and with 24 of those you might want to get a 1200w psu.

 

[Stilez]

Dude you are way over thinking this. Just buy a kill-a-watt for $20 and you will know your estimated power usage. Either you are close to your limits or you're not. I'm pretty sure you're not but since you added your hardware specs I see your using 7200rpm drives and with 24 of those you might want to get a 1200w psu.

I'm not sure that referencing the manufacturer's own datasheet for power draw, and multiplying by number of HDDs, is "over thinking" it. It seems naïve, as if you're saying "it's hard to measure transients, so you should assume they don't happen and just measure the average operational draw instead (which is semi-useless to know in the context, since we already know it's 1/3 of the size of transient draw). Also ignore that pretty much all motors and digital devices draw peaks and that the manufacturer themselves say they happen".

Mainly, I'm saying that you just can't do what you're thinking, and measure current draw that way. It takes good lab equipment and specialist knowledge, otherwise it's simply meaningless and invalid for PSU sizing purposes. Especially, HDD array current draw can not be measured at the wall or with IPMI as you suggest - that's flat out impossible. This is why @jgreco 's PSU sizing guide suggests "reserving about 35W [3A]" for each hard drive, even though average current draw is closer to 0.8 - 1.0A. He's spot on - peak 12v draw is about 3 amps (see table 2 p.10) before taking account of lack of exact synchronicity across an array. If anything, it's underthinking to blatantly ignore good information, and the laws of physics won't cut us slack if we do.

Hence my question's focus.

 

[danb35]

Dude you are way over thinking this. Just buy a kill-a-watt for $20 and you will know your estimated power usage.

...which will tell OP nothing about peak power usage, which is kind of his point (and @jgreco's point in his PSU sizing resource). Staggered spinup will help peak usage, but it's far from a perfect solution.

 

[danb35]

It takes good lab equipment and specialist knowledge, otherwise it's simply meaningless and invalid for PSU sizing purposes.

Neither DC current clamps nor oscilloscopes are all that expensive--you can get a nice 4-channel, 50 MHz DSO for $349 last I looked, and I bought a DC current clamp probe for around $50 a year or two back. But that will, at best, give you information for a couple of drives at a time. Unless you really hack on the cables...

 

[SweetAndLow]

...which will tell OP nothing about peak power usage, which is kind of his point (and @jgreco's point in his PSU sizing resource). Staggered spinup will help peak usage, but it's far from a perfect solution.

So you are saying if you run your system for a couple years that the power usage numbers you get are not good enough to to help size a power supply?

 

[danb35]

So you are saying if you run your system for a couple years that the power usage numbers you get are not good enough to to help size a power supply?

I'm saying that the prevailing opinion among those whose opinions I trust is that PSUs should be sized for peak demand, not average demand. And for determining peak demand, no, those power usage numbers are not useful.

 

[Stilez]

Staggered spinup will help peak usage, but it's far from a perfect solution.

Far from perfect, indeed. Staggered spinup can reduce the startup load, but it won't tell me what the reduced load is, because it's very dependent on firmware and baseboard designer, it (possibly) may not control spinup after periods of drive idle but only at boot - and it definitely does nothing at all for transient in-use peaks which are what concern me in use (I don't expect to reboot often!). Also while discussing staggered spinup, the 12v line is primarily about the motors (and perhaps head) - but 12v is only half of the picture anyway. The 5v line powering HDD logic circuitry averages 0.5-0.7A per drive, but can peak at 1.5A in use, and you definitely don't want brownouts on your DSPs and digital side of a drive during peak moments. That's up to ~36A peak draw for the disk array if they synced... but since big PSUs are always tuned for graphics and PCIe, and PCIe is 12v only, even a 1600W-2000W PSU isn't usually specced to deliver over 24A or perhaps 30A on 5v, and it's not clear how well 5v transient peaks can be met from its capacitors.

By contrast even a dual 750W PSU can easily supply around 40A-48A on 5v as well as 1500W on 12v, across twice the number of HDD cable ports (cutting down heating), and for 1/3 to 1/4 of the price of a 1600 PSU of similar quality - if and only if it's safe to dual-rig them.....

Neither DC current clamps nor oscilloscopes are all that expensive--you can get a nice 4-channel, 50 MHz DSO for $349 last I looked, and I bought a DC current clamp probe for around $50 a year or two back. But that will, at best, give you information for a couple of drives at a time. Unless you really hack on the cables...

An oscilloscope would be the right tool, you could measure 4 or 5 HDDs at a time by clamping the multi-plug cable at its base, or even clamp all 5 cables together, if it's capable of measuring these kinds of DC transients via clamp without directly sitting in the electrical pathway. But for a once-off it's excessive, it's the same price as a 1200W - 1600W PSU itself.

So it's nice, but really impractical. My options come down to monolithic vs dual, which in turn distills down to "dual is better all around, for me, but I have concerns: are the concerns valid or okay?"

 

[LIGISTX]

Regardless of the other issues being discussed here, as far as multi vs single PSU’s go, I saw a Linus tech tips video a few days ago with a new and pretty cool PSU option. Literally designed for GPU mining, but would be applicable here. I am just not sure if the OEM is high quaint enough to get the “let it freenas” backing of your high tire Corsair/evga/seasonic units. I have not looked into who is actually making these puppies. But it is a very cool and elegant way to split up the load onto multiple PSU’s, and it supports failover so in the right situation (not sure a freenas box would be that situation since you will likely be using as many power headers as possible to help level the load) your box won’t go down if one PSU dies.

https://youtu.be/Ap1JZrMt9bs


Sent from my iPhone using Tapatalk

 

[danb35]

But for a once-off it's excessive,

Sure, but if you have some other reason to need (or want) the scope...

Alternatively, there are pocket scopes that are much cheaper. I wouldn't trust them for precision work, but they should do for this sort of thing. Something like this. Or a USB scope.

 

[Chris Moore]

Here is the ultimate solution to this debate. Get one of these and move your existing system board and drives into it. It comes with all the power supplies you will ever need.
https://www.ebay.com/itm/263410230195

 

[Chris Moore]

So it's nice, but really impractical. My options come down to monolithic vs dual, which in turn distills down to "dual is better all around, for me, but I have concerns: are the concerns valid or okay?"

By the way, I run a triple redundant supply in two of my FreeNAS servers and a dual redundant supply in the third. If you use an actual server chassis with an actual server power supply, you don't have these questions.

by contrast my load is disproportionately heavy on 5v

Hard drives use mostly 12v for the motor and only draw a small amount from the 5v for the logic.

 

[Chris Moore]

If you really must buy two regular "consumer" power supply and combine the power output to run your system, you can get one of these jewels:
http://phanteks.com/PH-PWCOB.html

It will give you fail-over protection as long as one supply is actually powerful enough to drive the load.
An important thing to keep in mind though, the most consumption is during startup.
Most servers that are designed to handle 24 drives will have 1200watt power supplies. It might be overkill but isn't that complicated.

 

[LIGISTX]

If you really must buy two regular "consumer" power supply and combine the power output to run your system, you can get one of these jewels:
http://phanteks.com/PH-PWCOB.html

It will give you fail-over protection as long as one supply is actually powerful enough to drive the load.
An important thing to keep in mind though, the most consumption is during startup.
Most servers that are designed to handle 24 drives will have 1200watt power supplies. It might be overkill but isn't that complicated.

The YouTube video I showed previously has that built into the PSU, which is pretty cool. Just not sure what the OEM of them is thus not sure the quality.


Sent from my iPhone using Tapatalk

 

[Stilez]

Here is the ultimate solution to this debate. Get one of these and move your existing system board and drives into it.

By the way, I run a triple redundant supply in two of my FreeNAS servers and a dual redundant supply in the third. If you use an actual server chassis with an actual server power supply, you don't have these questions.

The thing is, it isn't ideal for my use-case... even if I agree it's a good one for other users. I've already built a custom HDD rack because I didn't find one I liked the build of, enough, on the market. Disks vertically racked in open space with wide (20mm) spacing and a full 15mm of antivibration mounts on every drive, good quality high static pressure fans below (so air flows up past the disks into the room), not to the side within a case, plenty of room above and below the fans so airflow isn't impeded, and working with natural convection not despite it. I get operational HDD temps of 27-30C, and perhaps when really busy as much as 35C, there's almost certainly no inter-drive vibration or undamped resonance worth speaking of, and the fans are almost silent which is a nice feature for a corner of my work room. I wouldn't swap it for a server room chassis if I could. This cost me £15 (USD 20) + fans to build. Where could I get a chassis like it for sane money? Performance is too good!

Then there's the PSUs. I could migrate my racks themselves into a chassis, or break one down just to get the PSU.... but then again it's SOHO use and idles for days in between very busy times, so >=Platinum rated is probably what I need, as they're often 88-90% efficient down to 10% or 20% load (low loss) and 10 year warranty. I can also afford them new rather than with several years on the clock.

Some reasons why I can see the point but not sure it's the right solution for me.

Hard drives use mostly 12v for the motor and only draw a small amount from the 5v for the logic.

I'm not sure you've crunched the figures, although I'm sure in practice you're right, the datasheet is a bit more suggestive that it's not always so. The "small amount" is a peak of 1.5A per HDD (Table 2 p.10). Multiply by a pool of 20-24 HDDs, add 20% for headroom and another 25% or so for degradation over the years and it's not small. Not that many PSUs can supply 40-50A peak at 5v, which in principle is the kind of figure we're looking at for peak supply. But two PSUs with a common ground (even a modest high quality 2 x 750W) could do it with ease, because 5V is often supplied around 24A for PSUs in the 750 - 1600W range (the emphasis and headline wattage being on the 12v rail supply) .... hence the question in the OP.

If you really must buy two regular "consumer" power supply and combine the power output to run your system, you can get one of these jewels: http://phanteks.com/PH-PWCOB.html

I like it! But alas, it's purely for combining ATX to baseboard. No other ports. Will it help any more than a glorified cross-connection? It leaves the basic question still as unanswered. And, as LIGISTX says, who knows about quality? The PSUs are top notch and have been torn down by experts. The ccrossover shown here, probably hasn't...?

 

[Chris Moore]

You have all the answers. Go for it.