Power supply & hot-plugging SATA

[jstrom]

Hi,

this is not really FreeNAS related per se, as it is only hardware.. But there is a lot of good knowledge here so I hope someone might be able to give some input anyway :)

I've got some new hardware which I'm playing around a bit with, before migrating my existing installation to it. Today I've been doing a bit of experimenting regarding hot-plugging SATA devices. From a SATA standpoint, it seems to work more or less flawlessly.. The system handles the connects/disconnects without any problems. However, the PSU does not seem to handle it that good..

Having one single drive attached to one PSU SATA power connector (directly, no drive bays or similar), I want to bring a second drive online. So, on the same cable, second SATA connector, I now plug in the second drive (no data connection at this point).
When I do this, the first drive seems to get a brown-out or something, and momentarily drops out of the system (kernel detecting this and saying it's gone, then re-attached). It goes back online directly, but this could of course still be fatal.
If I attach the drive to the second PSU SATA cable instead of the same, I do not notice any problems.. But hey, I want more than 1 disk per cable.. :)

Some details:
I've tried with two brand new Supermicro PSUs, a PWS-502-PQ (500W) and a PWS-303-PQ (300W). The only load is the motherboard, a X10SL7-F, and this one drive. A basic wattmeter says ~60W on the input side, thus not very high load...
I've tried with three different sets of drives. Although not the newest models, they are new enough. Both 500Gb and some 1T Seagates, with specified startup current around 2.6-2.8A (similar to "modern" ones).

Most of the tests have been made with the 500W PSU, but I tested with the 300W with the same results.

Another test was to connect 11 random old SATA drives on to the PSU; no problems at all neither booting or keeping it running with heavy IO.

In my current system (powered with an Corsair 650W from 2007), I've had no problems at all with this, the few times I've been changing disks... So I'm having a hard time to find this reasonable..

So, some questions..
a) Have I missed something regarding SATA power? Pre-charging to mitigate all of this should be standard on the drive side, if I understand correctly.. Or is the pre-charging circuitry expected to be inside the cable/hot-plug cabinet? All the drives have power pins with varying length (as per the link). Every single SATA power cable I own seems to have same lenght on all pins though.. But should be enough on one side?

b) Am I expecting too much from these PSUs? I cannot believe that the surge from a single drive with this low load would be a problem...

c) Anyone with similar experiences, with these PSUs or other?

Thank you for your input!
Regards
Johan

 

[Ericloewe]

It's a regular ATX PSU, right? Any chance you can borrow a known-good PSU to test this?

Edit: I just remembered something:

All backplanes I've used when hot swapping drives have a few capacitors on them. It could be that these are a requirement to avoid spikes when plugging drives in. They're not present in your setup.

 

[DannyKlenz]

What size of are the caps? They could be easily soldered onto the power cable. You could buy two molex to sata pigtails and solder them in there and see how it works.

 

[cyberjock]

1. Precharging is designed into the connector. One pin sticks out slightly more than others. That's your "precharging". If all of yours are the same length that's not necessarily ideal.
2. Actually, low loading a PSU does make this problem worse because of how voltage regulators work. Increasing load by a large % is the killer. If you only have 50w of load and you add 30w that's an increase of 60%. But if you have 150w of load and you add 30w that's an increase of just 13%. There are also different regulator designs that either can't react fast enough or have other inherent problems with starting surges.
3. Many PSUs can't handle the starting current for hotswapping and such (your PSU may not be rated for hotswapping). Many SATA cables have problems (such as precharging, lower current rating than you want, higher resistance in the connector, etc.) can cause these problems.
4. Sorry but doing your own cap-addition is just not a good idea. Those circuits have more than "just capacitors" on them. Not to mention they are going to be in parallel with *all* of your nice expensive equipment.

What I would do if I were you is I'd put more drives on the system. Get that 500w and put 3-4 drives on the PSU and see how it behaves with taking drives on and off. If that can't handle that you are probably going to have to shop for a better PSU. :(

 

[jstrom]

Thanks for your replies!

It's a regular ATX PSU, right? Any chance you can borrow a known-good PSU to test this?

Yeah, regular ATX PSUs. They can be compared on http://www.supermicro.com/support/resources/pws/pws.aspx (select Mid-tower in the first drop-down). I don't have any spares around at home at least, not these new ones with extra 8-pin motherboard connector.

All backplanes I've used when hot swapping drives have a few capacitors on them. It could be that these are a requirement to avoid spikes when plugging drives in. They're not present in your setup.

What size of are the caps? They could be easily soldered onto the power cable. You could buy two molex to sata pigtails and solder them in there and see how it works.

1. Precharging is designed into the connector. One pin sticks out slightly more than others. That's your "precharging". If all of yours are the same length that's not necessarily ideal.

Yeah, extra caps near the drives would probably help, but I don't really feel like that should be necessary? Also, not something I'm very keen on doing with my new hardware..
Regarding precharging, yes a few of the pins stick out a bit more on the drive side, on all my drives. But not on a single one of the SATA power cables I've got around, either loose adapters or those directly on the PSUs.
I haven't seen any SATA power cables specificly for hot-plugging either, with non-aligned pins, are there such things?

2. Actually, low loading a PSU does make this problem worse because of how voltage regulators work. Increasing load by a large % is the killer. If you only have 50w of load and you add 30w that's an increase of 60%. But if you have 150w of load and you add 30w that's an increase of just 13%. There are also different regulator designs that either can't react fast enough or have other inherent problems with starting surges.

Makes sense. Will have to try and see how it reacts with a little more load on, I'll get back on that.

3. Many PSUs can't handle the starting current for hotswapping and such (your PSU may not be rated for hotswapping). Many SATA cables have problems (such as precharging, lower current rating than you want, higher resistance in the connector, etc.) can cause these problems.

Hm, while I cannot find any detailed specs on this PSU, there is at least one Supermicro chassi which claims to support hotswap, using this PSU. Of course, that is with hotswap drive-cages..

4. Sorry but doing your own cap-addition is just not a good idea. Those circuits have more than "just capacitors" on them. Not to mention they are going to be in parallel with *all* of your nice expensive equipment.

What I would do if I were you is I'd put more drives on the system. Get that 500w and put 3-4 drives on the PSU and see how it behaves with taking drives on and off. If that can't handle that you are probably going to have to shop for a better PSU. :(

I agree, messing with extra caps is not something I'd like to try...

Will get back with some test results with a more loaded PSU!

 

[cyberjock]

I can't remember which connector is "precharged" but if it's on the hard drives then it's good. I know that without the precharge most hard drives will be destroyed when plugging them in due to the grounding and such.

 

[jstrom]

I can't remember which connector is "precharged" but if it's on the hard drives then it's good. I know that without the precharge most hard drives will be destroyed when plugging them in due to the grounding and such.

Not sure about actual pre-charging, but the drives do have what the wikipedia page (http://en.wikipedia.org/wiki/Serial_ATA#Power_connectors) calls "two-stage mating" (but the sequence actually describes 3 stages?), which means GND connects first to avoid running on only 5+12V and no GND, then one pin of 5V+12V should connect and allow pre-charging through some resistance (avoiding spike), and finally in the third stage the rest of the power pins should connect.

Hm.. the article says "Ground pins 4 and 12 in a hot-swap cable are the longest", so I guess there are special hot-swap cables (or at least this would be the case in hot-swap bays?).
Looking at my drives here (an old Seagate barracuda 500GB in this case, and some others, including a 1Tb), I have two "stages" for the power-pins, pin 3-8, 11, 13,14 are connected initially, then the rest. Comparing to the scheme on the above article, it means stage 1/2 is identical on the disk-side, and actually pre-connecting GND would require special cable.. However, pin 3+7+13 are still connected at first, allowing for precharging..

Anyone with a hot-swap cabinet who would be willing to take a look, does it have longer pins for 3+7+13?

From the manual of this particular drive 500GB drive:

The comments on the mating sequence apply to the case of backplane blindmate connector only. In this
case, the mating sequences are:
• the ground pins P4 and P12.
• the pre-charge power pins and the other ground pins.
• the signal pins and the rest of the power pins.
3. There are three power pins for each voltage. One pin from each voltage is used for pre-charge when
installed in a blind-mate backplane configuration.

So... pre-charge is only used when using backplane connector? Still doesn't really say if pre-charge circuits are expected on the connector side or the drive side..

 

[cyberjock]

I'm not sure which side the connector is required to be on.. I believe it is drive-only. It doesn't matter though and you are focusing on pre-charge too much. It's not the solution to your problem and it doesn't fix what you have that is broken. Precharge is *always* used with hard drives.. without it you *will* damage a hard drive the first time you power it on. ;)

 

[jstrom]

I'm not sure which side the connector is required to be on.. I believe it is drive-only. It doesn't matter though and you are focusing on pre-charge too much. It's not the solution to your problem and it doesn't fix what you have that is broken. Precharge is *always* used with hard drives.. without it you *will* damage a hard drive the first time you power it on. ;)

Yeah might be the case. I'm just trying to understand what's going on and getting all facts straight :)

So, did some more testing. I currently have 8 drives connected in total, 4+2+2 per PSU feeds. Then I attached another drive onto one of the feeds with 2 drives, with the result that one drive droppde out (the one farthest away from the PSU on that line).
However, connecting an additional 10th drive on the feed which already had 4 drives, did NOT bring any of those drives offline. But, then I disconnected one of the drives on this line, and when reconnecting the power, a third one detached.. So that wasn't a safe bet either.

With 9 drives + motherboard, the watt-meter says 114W on my 500W PSU.

Anyone with experience on these PSUs?

 

[cyberjock]

One thing I'll say because I don't think I have anything else to add to this topic anyway....

Hotswapping, while supported, isn't recommended IMO. The reason is several fold, not the least of which is that even if you solve this problem, are you ever going to be 100% sure that this problem doesn't crop up someday again? Also we've had too many problems with people pulling the wrong drive from the FreeNAS box and stuff. Much safer to shutdown the server and make 100% sure you want *that* drive out and *this* drive in than to do it with the box on. Seen a few pools go *bang* because people pulled the wrong drive.... sometimes they'd pull several bad drives and astonishingly wouldn't get it through their heads that they should shutdown the server.

So while this is a problem, I think the problem is easily solved with a little common sense.. just don't do hot-swap and hot-plugging!

 

[jstrom]

One thing I'll say because I don't think I have anything else to add to this topic anyway....

Hotswapping, while supported, isn't recommended IMO. The reason is several fold, not the least of which is that even if you solve this problem, are you ever going to be 100% sure that this problem doesn't crop up someday again? Also we've had too many problems with people pulling the wrong drive from the FreeNAS box and stuff. Much safer to shutdown the server and make 100% sure you want *that* drive out and *this* drive in than to do it with the box on. Seen a few pools go *bang* because people pulled the wrong drive.... sometimes they'd pull several bad drives and astonishingly wouldn't get it through their heads that they should shutdown the server.

So while this is a problem, I think the problem is easily solved with a little common sense.. just don't do hot-swap and hot-plugging!

Very true... I could just power everything down (and I just might end up doing that), but I'd still like to know why this won't work. For example, if I buy hotswap chassis, how can I be sure that it would work? What I'd like is some technical evidences explaining why a hot-swap chassis would work, and this won't..

That said, I appreciate all your input, let's hope someone else has might enlighten this situation with more technical details :)
Again, if anyone has a hotswap chassis easily accessible, some pictures of the connectors/circuit boards might reveal some details!

 

[Ericloewe]

I just checked my desktop's hot swap bays. Indeed, the connectors have a few pins longer than the rest.

In any case, I wouldn't trust those PSUs with hot plugging, given your experience.

 

[jstrom]

I just checked my desktop's hot swap bays. Indeed, the connectors have a few pins longer than the rest.

In any case, I wouldn't trust those PSUs with hot plugging, given your experience.

This is the pins on the backplane, in which the disk will plug directly? Or is there any per-drive trays with connectors?

I've sent a mail to my supplier regarding the reliability for hotplugging, we'll see what they say.

 

[Ericloewe]

This is the pins on the backplane, in which the disk will plug directly? Or is there any per-drive trays with connectors?

I've sent a mail to my supplier regarding the reliability for hotplugging, we'll see what they say.

Regular backplane, the tray is literally a heavy piece of painted SECC folded into shape to hold the drive.

 

[jstrom]

Well, to remove all doubt:

SATA 3.0 spec, http://www.knowledgetek.com/datastorage/courses/SATA_3.0-8.14.09(CD).pdf

Page 82, lists all pins, and explicitly states:
"Although the mate order is shown, hot plugging is not supported when using the cable connector receptacle. "

Chapter 6.8, Precharge and Devices Presences Detection (page 179), also clearly specifies:
"The Serial ATA device connector has been specifically designed to accommodate a robust hot- plug capability. One feature of the device connector is the ability for receptacles to limit the instantaneous inrush current through the use of a precharge scheme. This scheme relies on one power delivery contact for each voltage being longer than the remaining contacts in order to allow power to be delivered through this longer contact through a current limiting device. .... All burden for limiting the inrush current for a newly inserted device is borne by the receptacle/backplane "

For "informal purposes", the spec also suggest "Depending on the details of the actual enclosure subsystem design, typical precharge resistor values for the illustrative example scenario may therefore be in the range of 10-20 Ohms. "

Conclusion:
Based on the above details in the SATA spec, hot-plug/hot-swap via a regular SATA power cable is NOT supported. A backplane (or other power connector) with an inrush current limitation circuitry is REQUIRED.
Thus, it's only pure luck that it has worked on my earlier machines!

To verify that my drive does not actually have any circuitry for this in place, I measured the resistance between pre-charge and regular pins (i.e. between P2 and P3) on my drives, 0 ohms.

I wonder if there is any SATA power cables out there, which features
a) 3-stage mating order, and
b) pre-charge resistor

In my current situation I got plenty of space to carefully remove a drive while the system is running, thus a cable like that could actually be usable. But I guess that is not normally the case, and the demand for such cables are quite low.. If it wasn't for the 3-stage mating order, I could build one myself. But modifying a regular SATA power receptor to get the proper mating order probably isn't doable.

If anyone has contradicting information, please, contradict me! :) While looking around for information regarding this, I've never seen anyone explicitly claims regarding support/no support, but I don't want to spread any miss-information.

 

[DannyKlenz]

I recently purchased a molex to sata cable and it has 3 resistors on the orange wire. Was not sure of there purpose, i suppose its possible they are pre-charge resistors.

Edit: Measured 18.3 megohm across the orange wire. I opened it up and its actually two IN4002 diodes then an IN5819 diode.

 

[Robert Smith]

I think they use the diodes for their forward voltage drop characteristics to make 3.3 Volts out of 5 Volts available on the Molex connector.

 

[Ericloewe]

It's an odd solution. Most adapters (and some Backplanes, I imagine) don't give a damn and leave the 3.3V floating.

Of course, that means that nobody ever got around to using 3.3V...

 

[cyberjock]

3.3v is not required for SATA. In fact, the spec makes it effectively optional since disk drives don't use it. But some cables still provide it in case you might need it. That cable scares the daylights out of me. I don't know what is inside the shrink-tubing, but I would never trust a device with that cable. LOL.

 

[DannyKlenz]

Three diodes. Its perfectly safe and not uncommon for things like this to be soldered in line.