135TB POD works, now some modifications for #2

[RichR]

Hello,

Variations needed on Backblaze storage POD. Pertinent links below (if you want a lot of background)

Their version 2 http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
Their version 1 http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
extra reading http://extrememediaservers.blogspot.com/ (26 comments after the picture of the crimping tool/set - 3/4 way down page)

We bought a mostly assembled POD (version 2 - minus the hard drives), installed 45 3TB Hitachi drives, installed FreeNAS, put it in a datacenter, and all is "mostly good." There is a bug in the SuperMicro motherboard that does not allow a soft reboot that reads the USB stick on the MB even though it is listed first in the startup disk order. It's been reported to them back and forth - but we'll see what happens. Hard reboots work fine (but it's in a datacenter).

I am building #2 from scratch mostly, and have 2 main issues.

#1 - Power.
The original has 2 power supplies, one for 25 drives + 3 fans, and the other for 20 drives + MB, fans and the rest. We want to cut this down to one power supply. If you look near the very bottom of the blogspot linked page (in the comments), you'll see startup is the issue (which makes total sense), and the pictures of the regular usage is what would be expected (in terms of wattage). Our 3TB drives are rated as 1.2A(+5V) and 2.0A(+12V). Do the math and it's easy to see that you'll never find a PSU that is rated for a minimum of 54A on the 5 volt rail. The easy solution seems to be to use staggered spin up on the BIOS of a controller card (a SATA "option" according to the SATA spec). Easy for a smaller number of drives, more difficult because of the port multipliers (9) feeding all the drives (45). Not all RAID/SATA controllers are port multiplier compatible, especially when you are talking a larger number of drives.

So, 1) any suggestions on cards (we can use 3 PCI)? The Syba cards (from the original build) do not have staggered spin up as an option, it seems highpoint aren't very compatible with FreeNAS (according to a bunch of posts) but some do have the option but are not port multiplier friendly.

Or 2) and out of the box ideas on how to get the drives spun without killing/damaging something? Btw, I'm assuming that trying to start 45 drives at once would damage something.... what would the something be????

btw, the Seasonic Platinum 860 get incredible reviews..... and even though there is a 1000w version, once the drives are spinning that would be way overkill.

#2 - Motherboard
I don't like not being able to soft reboot the system and not have it come up correctly. Any suggestions on a MicroATX MB that will allow at least 32 GB memory, 3 PCI, etc? The Supermicro we're using now is great (X8SIL-F) but getting memory for it has been a challenge (to get 32GB), and the i3 processor is fine.

Thanks for any responses,

Rich

 

[jgreco]

Or 2) and out of the box ideas on how to get the drives spun without killing/damaging something? Btw, I'm assuming that trying to start 45 drives at once would damage something.... what would the something be????

btw, the Seasonic Platinum 860 get incredible reviews..... and even though there is a 1000w version, once the drives are spinning that would be way overkill.

There are basically a few things that can go wrong trying to do that. If the power supply is able to withstand the momentary overload resulting from all those drives spinning, you're stressing all the components in the power supply, which typically shortens the lifetime. Depending on what goes wrong when one of them eventually fails, your unit may go dead (likely best case) or lethal-to-equipment voltages might be passed within your unit (unlikely worst case). It is kind of just waiting to see if there's a spectacular failure. Otherwise, if the power supply isn't able to withstand the momentary overload, your voltages sag, and you suddenly have 45 hard drives that aren't getting sufficient voltages at the time they need it most. This is bad for their mechanicals, and shortens their lifetime, or kills them immediately, depending.

This is why they invented staggered spinup. Figure that out and use it if you can. They used to put jumpers on drives for it... lord knows how many of those I've set over the years. These days it seems to be a configuration setting, power-up-in-standby. One has to be careful with this because it's theoretically possible to get a bunch of things to spin up simultaneously... but if you do not ever cause the drives to spin DOWN during server operation, then you're really only looking at boot-time spin-up, and I bet that'd be a viable possibility.

Quite frankly I'm skeptical about the claim of 1.2A at 5V. It could be, I suppose. That's the 7K3000? I have a suggestion for you: CHECK. That's not necessarily a number you should assume is being hit. It's easy to check, too. Take a drive power extension of some sort. Cut the red wire. Hook the cut ends up to any halfway decent digital DVM that has an ammeter capability. Get your settings right, then connect it to some power, and *watch*. Repeat the test with more drives if you feel uncomfortable with the accuracy of that 1.2A number. You might find out that the numbers you have to hit are actually more reasonable. But be prepared to be disappointed, too.

Anyways, without doing the math on the Seasonic, let me say that I've been incredibly impressed with another 80PLUS Platinum supply, the Kingwin Stryker 500. I spent a lot of time looking at options for powering some new virtualization hosts, was having quite a bit of frustration trying to find appropriately-sized supplies for a system that eats between 40 and 120 watts. There just aren't easily available platinum supplies available less than 500 watts. So I grumbled and looked at the Kingwin unit, which seemed to be a good match because it was fanless (I wanted the system boards to run variable speed case fans) and it fit the other basic requirements. The efficiency tests don't show results below 20% (100 watts) and so far I've been too lazy to rig up a jig to test watts on the output of the supply (it's a lot of leads to test) but I have a feeling that even at 10% load it's still really close to 90% efficiency. Measuring the AC inlet watts showed a 15 watt drop going from an average supply to the Platinum supply when the system was idle. That's quite a drop.

So my comment for you regarding the Seasonic would be to try the larger unit and not worry too much if it seems oversized. I'm guessing that it's very doable if you're careful and clever about it all.

 

[louisk]

#2 - Motherboard
I don't like not being able to soft reboot the system and not have it come up correctly. Any suggestions on a MicroATX MB that will allow at least 32 GB memory, 3 PCI, etc? The Supermicro we're using now is great (X8SIL-F) but getting memory for it has been a challenge (to get 32GB), and the i3 processor is fine.

Rich

Many Supermicro boards have either a mini-pci slot or (now) built-in IP-KVM capability which supports IPMI and SSH. If yours supports this, I would go this route and get the card.

 

[RichR]

I agree with just about everything you stated.

Regarding staggered spin up, that IS the goal, but finding a compatible controller, RAID or not, that has the ability to control staggered spin up AND is port-multiplier/backplane compatible is the issue (from within the controller BIOS)...."any suggestions on cards (we can use 3 PCI)? The Syba cards (from the original build) do not have staggered spin up as an option, it seems highpoint aren't very compatible with FreeNAS (according to a bunch of posts) but some do have the option but are not port multiplier friendly." Also, I've heard of, but have not been able to find a motherboard that has this in the BIOS.

Also, it's the Deskstar 5K3000 (not 7K3000) - the Startup (max. A) are taken directly from Hitachi's spec sheet (1.2A(+5V) & 2.0A(+12V))...

 

[RichR]

I'll clarify regarding the Supermicro board (X8SIL-F) that we currently use.... The board does not see the thumb drive on a soft reboot (such as sending the command from the FreeNAS GUI, or via ssh). This board does have IPMI (and the interface is pretty slick IMHO) via a dedicated LAN port, and we're testing.... regardless of whether the IPMI reboot command works or not, I don't like the bug. If I'm ssh'd in or in the FreeNAS GUI, I should be able to reboot....

 

[jgreco]

Let me just get to the heart of it. You have *45* of these drives. Go get three of them, figure out how to get them configured for "power up in standby" mode, then test them against the controllers of your choice. I have not done this, so I don't know, but even if there's only a 10% chance that it works is absolutely worth it. It's entirely possible that your controller of choice will do something reasonable with this.

I see three possibilities.

1) The drives will all start all at once. This breaks down into two subcategories:

1a) They literally all start at once and you're toast. So no go.

1b) All the drives attached to one controller start up at once. You might have a manageable situation, do the math.

2) The drives start as each port is probed. Voila. You have staggered spinup and have your problem solved.

3) The drives do not start as probed. This breaks down into:

3a) The drives are not detected or are otherwise rendered unusable by the controller because they're in standby. I find this unlikely.

3b) The drives are probed fine, but you may be left needing a script to start them in UNIX. There you can stagger them yourself.

There might be some other edge cases, like BIOS probes them fine but FreeBSD doesn't, but I'm guessing that FreeBSD won't be the problem with any of this.

And I would be very bothered too about the USB bug. I'd stamp it not suitable for production use unless you really *can* tolerate an outage.

 

[RichR]

Good info

in reverse order - the USB thumb drive bug will be resolved by not using a USB thumb drive - we'll get a small SSD instead.

A S.M.A.R.T. category is "Spin Up Time" in which manually tested results are around .5 second (+/- a bit) querying 6 drives. Pushing the envelope, a .75 delay for 1/3 the drives would probably be enough, although dividing the drives into more groups with a minimum 1 second delay would probably make it happier. Going back to the specs of the drives, 54 amps on the 5V, and 90 amps on the 12V rails divided by at least 3 groups would probably be fine. I agree with 1/2/3, and all this should happen before FreeBSD even gets into the mix....

 

[jgreco]

Building big storage is always full of unanticipated adventure.

 

[jgreco]

Go run over and look at http://www.hitachigst.com/tech/techlib.nsf/techdocs/FFDF1FA949853DBD8825785A005CBC55/$file/DS5K3000_US5K3000_OEMSpecRev1.1.pdf

9.10 Power-up in Standby feature set
Power-Up In Standby feature set allows devices to be powered-up into the Standby power management state to
minimize inrush current at power-up and to allow the host to sequence the spin-up of devices.
This feature set will be enabled/disabled via the SET FEATURES command or use of jumper. When enabled by
a jumper, the feature set shall not be disabled via the SET FEATURES command. The enabling of this feature
set shall be persistent after power cycle.
A device needs a SET FEATURES subcommand to spin-up to active state when the device has powered-up into
Standby. The device remains in Standby until the SET FEATURES subcommand is received.

Looks like you can use this to get your desired results.