sniper29a
Cadet
- Joined
- Jan 13, 2022
- Messages
- 5
I have a server with 35 HDDs (so far) with possibility to expand up to 120 HDD.
I use for years Seasonic Prime-TX series for their exceptional features, quality, and 90-95% efficiency
Of course, classical ATX PSU usually have 5-6 peripheral ports and official spec says max. 4 drives per cable/rail or 20-24 per PSU.
Seasonic Prime-TX 650W, 5 rails = 20 HDD (if lucky and no need to connect something via peripheral rail such as Intel Expanders with Molex power supply). 20 HDDs working at their usual 6W cap is just 120W + server HW barely 200W. 30% PSU load. Not bad about 92% PSU efficiency, but 50% load gives sweet spot of 95-97% efficiency (hard to say from graphs)
Seasonic Prime-TX 1000W, 6 rails = 24HDD = 144W + server HW = 220W or 22% load and PSU efficiency drop.
KLUDGE
I asked Seasonic if there is a possibility to hook up 6 drives per rail. While not recommended, it is possible if you carefully check max. current draw per drive. Most modern HDDs have 0.92A/5V + 0.75A/12V = max. 12.5W
Spinning up HDD motor seems to use mostly 5V rail, but I wouldn't bet on it too much. I used max. specified current draw.
6 x 12.5 = 75W, which is official peripheral limit of 6 pin peripheral cable/rail. Of course, from crypto mining we know there is HUUUGEEE margin. With official Seasonic 18AWG modular cables, your limit is connectors itself. ATX standard says SATA power connector 3 x 5V/1.5A
For a few seconds, rail may be overloaded and Seasonic overload protection suppose to kick in. It never happened and I was running 35 HDDs from 1000W Seasonic Prime-TX. It was working like 2 months.
Of course, there is always issues with quality of cabling. It is called "pfuschen" in German...you never know with those high quality Chinese third party cables.
I had finally decent PSU load and almost max. efficiency at 26% load but this week SMARTD send me a message "unable to open device /dev/sdX"...damn, I thought my first HDD failure in 30+ years in IT. After further investigation, it was WD Gold 14TB drive barely 1 year old...weird...I tried "hot start" because HBA/Expanders unofficial romance is Russian Roulette...HDD still dead. After reboot, drive is back...one hour later 4 drives "died" on same cable...I swapped cable...it worked for a day...same drive "died" again.
Running expanders powered by risers (intended for GPU mining) was most likely issue...somewhere dropped voltage/current, one hundred various adapters on the way to HDD. Once, HDD block device disappeared from kernel...HBA/Expander found it again and registered it as a new device /dev/sdam. Unfortunately, device inaccessible. Interestingly enough, LVM JBOD group still worked with missing drive.
Meanwhile, I have contacted a company designing PCBs, electronic hackers...request was simple "are you able to design break-out board capable to utilize PCIe 8pin rails and split 12V into 5V/12V for powering many HDDs." No reply so far, IMPOSSIBLE? I do not like this word, in my world exists only two words ANYTHING POSSIBLE or INCOMPETENT - where is will there is a way.
Guy I purchased HBAs, Expanders told me his bad experience with "backplanes". It killed a few of his HDDs. I do not know what he did, but it works in IT world, so it might be classical Kb99Ch syndrome. Of course, backplanes are not for puny mortals...SAS2 24HDD Supermicro backplane costs like 300€, and you still need Expander.
Supermicro offers 40HDD JBOD servers...it is not for puny mortals like me...I do not earn thousands per month to afford it, profit is tight.
I have rewired whole server last night...2 x PSU, 4 HDDs per cable. So far it works, but I have been there...it worked before as well...only time will tell...of course, PSU load is around 10% per PSU and efficiency dropped to 90%. APC UPS seems to show same wattage numbers. Only time will tell.
Of course, it still doesn't resolve challenge to expand server to 120 HDDs. I can connect "only" 40 HDDs so far.
Any sugestion guys, please?
I use for years Seasonic Prime-TX series for their exceptional features, quality, and 90-95% efficiency
Of course, classical ATX PSU usually have 5-6 peripheral ports and official spec says max. 4 drives per cable/rail or 20-24 per PSU.
Seasonic Prime-TX 650W, 5 rails = 20 HDD (if lucky and no need to connect something via peripheral rail such as Intel Expanders with Molex power supply). 20 HDDs working at their usual 6W cap is just 120W + server HW barely 200W. 30% PSU load. Not bad about 92% PSU efficiency, but 50% load gives sweet spot of 95-97% efficiency (hard to say from graphs)
Seasonic Prime-TX 1000W, 6 rails = 24HDD = 144W + server HW = 220W or 22% load and PSU efficiency drop.
KLUDGE
I asked Seasonic if there is a possibility to hook up 6 drives per rail. While not recommended, it is possible if you carefully check max. current draw per drive. Most modern HDDs have 0.92A/5V + 0.75A/12V = max. 12.5W
Spinning up HDD motor seems to use mostly 5V rail, but I wouldn't bet on it too much. I used max. specified current draw.
6 x 12.5 = 75W, which is official peripheral limit of 6 pin peripheral cable/rail. Of course, from crypto mining we know there is HUUUGEEE margin. With official Seasonic 18AWG modular cables, your limit is connectors itself. ATX standard says SATA power connector 3 x 5V/1.5A
For a few seconds, rail may be overloaded and Seasonic overload protection suppose to kick in. It never happened and I was running 35 HDDs from 1000W Seasonic Prime-TX. It was working like 2 months.
Of course, there is always issues with quality of cabling. It is called "pfuschen" in German...you never know with those high quality Chinese third party cables.
I had finally decent PSU load and almost max. efficiency at 26% load but this week SMARTD send me a message "unable to open device /dev/sdX"...damn, I thought my first HDD failure in 30+ years in IT. After further investigation, it was WD Gold 14TB drive barely 1 year old...weird...I tried "hot start" because HBA/Expanders unofficial romance is Russian Roulette...HDD still dead. After reboot, drive is back...one hour later 4 drives "died" on same cable...I swapped cable...it worked for a day...same drive "died" again.
Running expanders powered by risers (intended for GPU mining) was most likely issue...somewhere dropped voltage/current, one hundred various adapters on the way to HDD. Once, HDD block device disappeared from kernel...HBA/Expander found it again and registered it as a new device /dev/sdam. Unfortunately, device inaccessible. Interestingly enough, LVM JBOD group still worked with missing drive.
Meanwhile, I have contacted a company designing PCBs, electronic hackers...request was simple "are you able to design break-out board capable to utilize PCIe 8pin rails and split 12V into 5V/12V for powering many HDDs." No reply so far, IMPOSSIBLE? I do not like this word, in my world exists only two words ANYTHING POSSIBLE or INCOMPETENT - where is will there is a way.
Guy I purchased HBAs, Expanders told me his bad experience with "backplanes". It killed a few of his HDDs. I do not know what he did, but it works in IT world, so it might be classical Kb99Ch syndrome. Of course, backplanes are not for puny mortals...SAS2 24HDD Supermicro backplane costs like 300€, and you still need Expander.
Supermicro offers 40HDD JBOD servers...it is not for puny mortals like me...I do not earn thousands per month to afford it, profit is tight.
I have rewired whole server last night...2 x PSU, 4 HDDs per cable. So far it works, but I have been there...it worked before as well...only time will tell...of course, PSU load is around 10% per PSU and efficiency dropped to 90%. APC UPS seems to show same wattage numbers. Only time will tell.
Of course, it still doesn't resolve challenge to expand server to 120 HDDs. I can connect "only" 40 HDDs so far.
Any sugestion guys, please?