PSU for 40 HDD solution

[sniper29a]

I have a server with 35 HDDs (so far) with possibility to expand up to 120 HDD.

I use for years Seasonic Prime-TX series for their exceptional features, quality, and 90-95% efficiency

Of course, classical ATX PSU usually have 5-6 peripheral ports and official spec says max. 4 drives per cable/rail or 20-24 per PSU.

Seasonic Prime-TX 650W, 5 rails = 20 HDD (if lucky and no need to connect something via peripheral rail such as Intel Expanders with Molex power supply). 20 HDDs working at their usual 6W cap is just 120W + server HW barely 200W. 30% PSU load. Not bad about 92% PSU efficiency, but 50% load gives sweet spot of 95-97% efficiency (hard to say from graphs)

Seasonic Prime-TX 1000W, 6 rails = 24HDD = 144W + server HW = 220W or 22% load and PSU efficiency drop.


KLUDGE

I asked Seasonic if there is a possibility to hook up 6 drives per rail. While not recommended, it is possible if you carefully check max. current draw per drive. Most modern HDDs have 0.92A/5V + 0.75A/12V = max. 12.5W

Spinning up HDD motor seems to use mostly 5V rail, but I wouldn't bet on it too much. I used max. specified current draw.

6 x 12.5 = 75W, which is official peripheral limit of 6 pin peripheral cable/rail. Of course, from crypto mining we know there is HUUUGEEE margin. With official Seasonic 18AWG modular cables, your limit is connectors itself. ATX standard says SATA power connector 3 x 5V/1.5A

For a few seconds, rail may be overloaded and Seasonic overload protection suppose to kick in. It never happened and I was running 35 HDDs from 1000W Seasonic Prime-TX. It was working like 2 months.

Of course, there is always issues with quality of cabling. It is called "pfuschen" in German...you never know with those high quality Chinese third party cables.

I had finally decent PSU load and almost max. efficiency at 26% load but this week SMARTD send me a message "unable to open device /dev/sdX"...damn, I thought my first HDD failure in 30+ years in IT. After further investigation, it was WD Gold 14TB drive barely 1 year old...weird...I tried "hot start" because HBA/Expanders unofficial romance is Russian Roulette...HDD still dead. After reboot, drive is back...one hour later 4 drives "died" on same cable...I swapped cable...it worked for a day...same drive "died" again.

Running expanders powered by risers (intended for GPU mining) was most likely issue...somewhere dropped voltage/current, one hundred various adapters on the way to HDD. Once, HDD block device disappeared from kernel...HBA/Expander found it again and registered it as a new device /dev/sdam. Unfortunately, device inaccessible. Interestingly enough, LVM JBOD group still worked with missing drive.

Meanwhile, I have contacted a company designing PCBs, electronic hackers...request was simple "are you able to design break-out board capable to utilize PCIe 8pin rails and split 12V into 5V/12V for powering many HDDs." No reply so far, IMPOSSIBLE? I do not like this word, in my world exists only two words ANYTHING POSSIBLE or INCOMPETENT - where is will there is a way.

Guy I purchased HBAs, Expanders told me his bad experience with "backplanes". It killed a few of his HDDs. I do not know what he did, but it works in IT world, so it might be classical Kb99Ch syndrome. Of course, backplanes are not for puny mortals...SAS2 24HDD Supermicro backplane costs like 300€, and you still need Expander.

Supermicro offers 40HDD JBOD servers...it is not for puny mortals like me...I do not earn thousands per month to afford it, profit is tight.

I have rewired whole server last night...2 x PSU, 4 HDDs per cable. So far it works, but I have been there...it worked before as well...only time will tell...of course, PSU load is around 10% per PSU and efficiency dropped to 90%. APC UPS seems to show same wattage numbers. Only time will tell.

Of course, it still doesn't resolve challenge to expand server to 120 HDDs. I can connect "only" 40 HDDs so far.

Any sugestion guys, please?

 

[AlexGG]

Multiple PSUs? I run two PSUs, mainboard, CPUs and OS SSD powered from one PSU, and the hard drives powered from the other PSU, synchronized by the device similar to this - https://aliexpress.com/item/32815317603.html (it is not really an endorsement of this specific unit, just an illustration)

 

[sniper29a]

BackBlaze seems to have the solution, but normal PSU cannot handle more than 20-25A on 5V rail. With spin up staggering, it may be possible to run something like 30-40 HDDs per PSU. Connected with harnesses and adaptors hooked up to ATX 24pin connector and so on.

 

[jgreco]

Just a note from someone who does this professionally that there are a lot of risks in undersizing PSU's in this manner.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can find. DO NOT DO THIS. Your NAS lives or dies by...

www.truenas.com

 

[sniper29a]

I have seen it years ago, and I agree with him.

I wanna see how you overload 750 or 1000W PSU ;-)

Seasonic Prime-TX 1kW PSU has got 25A limit on 5V rail. You can safely start 25 drives at once with this PSU (all my Seagate, WD, Toshiba have max. 0.5-0.95A on 5V power supply)

If you manage spin up stagger, each drive operates barely 0.5A on 5V rail. Theoretically 50 drives on 1kW PSU ;-)

Backblaze Pod 6.0 has got 60 drives on two EVGA 750W PSUs, and it works for years with thousands of pods they built.

 

[jgreco]

I wanna see how you overload 750 or 1000W PSU

In a word? Easily.

25A limit on 5V rail. You can safely start 25 drives at once with this PSU (all my Seagate, WD, Toshiba have max. 0.5-0.95A on 5V power supply)

Drives draw from the 12V rail for spinup. If you were to bother to look at the thread I quoted, we literally have graphs. If you were to bother to think about why it is that vendors like Supermicro outfit their 90-drive SSG-640SP-E1CR90 with dual 2600W PSU's. It's not for the crappy 5V rail. It's for the massive potential draw on the 12V.

Backblaze Pod 6.0 has got 60 drives on two EVGA 750W PSUs, and it works for years with thousands of pods they built.

Yes, but you have to guarantee staggered spin-up and carefully engineer things so that you do not do bad things. This includes not spinning drives down. It's actually discussed in the article I linked to above.

 

[sniper29a]

HDD servo allegedly uses only 5V, thus spin up takes more of 5V But it is manufacturer secret and nobody really knows. I use spec written on disk, that is for me theoretical max + 20% margin for ecoTerrorists claiming what is not there ;-)

1kW Seasonic PSU has got limit 83A on 12V rail.

WD Gold 14TB = 5V @ 0.44A, 12V @ 0.52
Seagate Exos X16 16TB = 5V @ 0.9, 12V 0.72A

you will need lots of drives to overload 1kW high quality PSU ;-)

You can have theoretically 100 WD Gold drives - 2.2 + 6.24 = 8.44W combined for both rails x 100 = 844W

Or at least 70 Exos X16 16TB drives 4.5 + 8.64 =13.14W combined.

Of course, it is impossible because 5V rail is the weakest point on every ATX PSU. All 650-850W PSUs have only 20A per 5V rail, only 1000W has got 25A.

My server has got 35 drives now and spin up takes 550W. That is like max 450W for HDDs, rest is CPU/MB/HBA/Expanders power draw. Or max.
12.5W per drive average. I bet it is even less because CPU+MB draws usually max. power at start before it switches of to lower states.

If you bothered to read spec for each HDD, you won't be whining about your theories of theoretical overload Also, every server uses dual PSU for redundancy, not for spinning up HDDs ROFL. Every server is designed for max. efficiency which is around 50% load. Server PSUs rarely have 90-97% efficiency as ATX. They are usually somewhere below 90%.

Any enterprise level HW uses SAS drives that draw more current than SATA, especially the fast ones. Not to mention, they use professional solutions like backplanes. They design for general purpose and various upgrade options.

If you use 90 x Exos X16, You need 5V @ 81A = 405W and 12V @ 64.8A = 777.6W or 1182.6W for spin up of HDDs. Add expanders, HBAs, CPU, MB and you get theoretical max. 1400W...thus 2600W PSU for best efficiency at 50% load ;-) It is general purpose server, they simply add more power for upgrades and safety from crazy users without clue...if CPU + VRM on MB draws 200W, you must have cap for it. It is very simple.

I am talking about simple storage server that most of the time idles. You are comparing Windows with Apple ;-)

 

[jgreco]

allegedly

I'm not really interested in your ridiculous allegations.

nobody really knows

Well, see, in the real world, we use these things called meters and scopes to measure the reality, and there's a lot of that posted in the PSU sizing thread that shows you to be ... what's the word ... wrong. Additionally, most HDD spec sheets actually specify power requirements for spinup as a separate line item.

Perhaps YOU don't really know. That doesn't mean nobody knows. There are those of us who are paid to know, and in fact do know.

If you bothered to read spec for each HDD, you won't be whining about your theories of theoretical overload

Actually, I do read the specs. I do this professionally. When you're doing this professionally, you actually learn how to do this stuff correctly. I have a background in electrical engineering. My shop builds servers for Internet Service Providers and other Internet businesses, and has been doing this for a long time.

Every server is designed for max. efficiency which is around 50% load. Server PSUs rarely have 90-97% efficiency as ATX. They are usually somewhere below 90%.

You clearly have no clue what you are talking about.

You are comparing Windows with Apple ;-)

No, but my best guess here is that I'm comparing professional with amateur.

 

[Ericloewe]

To prove Mr. Grinch's point, here's the sort of PSU that Supermicro considers reasonable for its 44-bay chassis (the JBOD variant of the 36-bay server chassis):

Supermicro 1200W 1U Redundant Power Supply (PWS-1K23A-1R)

PMBus 1.2 Support for Remote Management Flexible Redundancy Control 80 Plus Titanium Supermicro Certified

store.supermicro.com

1.2 kW Titanium rated. Curiously enough, it's awfully close to what you'd have if you took the 30 W figure we like to throw around and multiply it by 44.

 

[jgreco]

Mr. Grinch's point, here's the sort of PSU that Supermicro

You did see above where I quoted the PSU for the 90-bay unit? ZOMG:

vendors like Supermicro outfit their 90-drive SSG-640SP-E1CR90 with dual 2600W PSU's.

That's gotta be 208/240V-only. 20 amps at 120V is only 1.92kW (remember your derating). Hard to run one of these at home!

The density being offered at some data centers today is amazing. It becomes easy to see how a rack could require 33kW A+B power when you start stacking up these units. That's 900 drives in a 42U rack, and with 20TB drives around the corner, that means you could cram 18 petabytes in a single rack.

By way of comparison, we were happy putting 120V@15A A+B in racks back at the beginning of the commercial Internet. This gave you 1.44kW of usable power.

Fun stuff to geek out about.

 

[decibel83]

I have a server with 35 HDDs (so far) with possibility to expand up to 120 HDD.

Hi,
a little bit OT, sorry.

I’m about to plan an high density NAS with TrueNAS: could you share your hardware specs, please?

 

[jgreco]

Hi,
a little bit OT, sorry.

I’m about to plan an high density NAS with TrueNAS: could you share your hardware specs, please?

You really need to go with a dedicated storage chassis for high drive count solutions. Check out the Storinator (from 45drives) for stuff on the smaller side, or the 90-drive SSG Supermicro chassis outlined above. These things need to be able to crank out thousands of watts at startup. You can learn why in this article:

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can find. DO NOT DO THIS. Your NAS lives or dies by...

www.truenas.com

 

[Ericloewe]

Check out the Storinator (from 45drives) for stuff on the smaller side, o

There's a guy with a similar thing that's having all sorts of weird disk issues. Since Supermicro's stuff seems to be cheaper than 45drives' machines of similar size, I'd look there first. I think they also have 60-drive units now.