Two UPSs supporting one server?

[scurrier]

So, I am shopping for a UPS and discovering that finding the right one for me is not as straight-forward as it would initially appear. Compatibility seems to be a question mark (based on forum posts) even if the UPS appears in the FreeNAS UPS dialog dropdown. Not only that, but so many threads on the internet have someone chiming in about how X brand UPS did something scary to them. Over-charged, bulging batteries, batteries that don't charge, UPS failing even when power didn't, etc.

So I got a crazy idea. My Supermicro 933T chassis has triple redundant power supplies. Why not hook up two, different, inexpensive UPSs? There is a web site that offers refurb UPSs with new batteries and a short warranty.

Potential benefits:

1. A single section of the triple-redundant power supply can handle my server's load (as measured by a Kill-a-watt)
2. If one UPS fails while mains power is still good, I would be protected from that. (Could also achieve this without two UPSs by plugging one PSU into the wall directly)
3. If one UPS fails during a mains outage, I would be protected from that.
4. If one of the UPS has communication compatibility issues with FreeNAS, I could try using the other one instead.

Potential drawbacks:

1. Two opportunities to burn my house down instead of one.
2. I'd be subject to the lesser of the two UPSs' surge protection. (In my opinion, surge protection is kind of a gamble anyway, so I don't think this would keep me up at night.) (This could potentially be overcome with a good surge protector which powers both UPSs)

Regardless of what I choose, I'm thinking it will be a good idea to run an extension cord from the server to the UPS, which would be kept in a safer location. So if the UPS goes all "self-immolation" on me it would be in the middle of a concrete floor or something.

Thoughts? Good/bad?

 

[cyberjock]

I think this is a bad idea. Normally, what happens with redundant PSUs is one carries most of the load while the others are basically "idle". And there's no way to know which PSU is providing the load unless you have a wattmeter hooked up to all of them to monitor them. To boot, sometimes load will shift between PSUs.

So, with the fact that loading is potentially going to be moving randomly, and you have 2 UPSes. So you can see that you are adding a bit of complexity. I think you're creating a situation that is more complex than it needs to be, and you are probably going to end up with a system that is less reliable than what you currently have.

 

[scurrier]

What is your source for the claim that redundant PSUs will shift the load between them like this? Or that one carries all of the load? It's against my electrical intuition. I suppose I could find a way to measure this. Like look at the total combined loading at idle and then measure each PSU independently over a period of time.

 

[cyberjock]

I used to deal with voltage regulators for a living. Part of my job description was that I had to be able to draw one to certain specs... from memory. ;)

There's basically do 3 general designs for redundant PSUs.

1. One is deliberately overvolted slightly(but within spec), while the other is undervolted slightly(but within spec). This causes one to always carry all of the load while the other does not. When you buy a replacement PSU they'll ask if its the master/slave of higher/lower, or whatever terminology they use. Usually a sticker on the PSU will designate which is which.
2. The two PSUs are basically identical. But, when you slide the PSU into the case, some cases will have a small peg that will stick out. It'll trip a switch on the PSU when fully installed. The tripped switch will cause that one PSU to behave at a different set point.
3. Two PSUs are basically identical with no real distinction between them. But, this causes problems when high reliability plays a part because they may be identical components internally, but their behaviour is unique among each other(kind of like how we all have different fingerprints). This causes problems because the voltage droop of the PSUs won't be the same, so they'll somewhat compete with each other. As they fight for contention as loading changes it can actually make a server unreliable, which is why you usually find a situation like #1. There are ways to redesign the voltage regulation so that they don't fight and cause problems, but it hurts efficiency and can add significant cost to the design. For situations like this, the temperature of the PSU components will actually cause voltage droop to shift, causing both PSUs to heat up, lose to the other PSU, then cool down until it wins the contention again.

I've actually thought about a setup like what you described in your original post. It sounds great on the surface, but the devil is hiding in the details(just like so many things related to servers, FreeNAS, and ZFS).

 

[scurrier]

Hmm ok. Thanks for the good explanation. I can see how that would happen.

Still... why would that mean that I shouldn't hook PSUs up to different UPSs? Or at least hook one PSU up to the wall directly and the others on the UPS? I agree at some point a system becomes too complex to understand all the failure modes of. But I don't think this idea has reached that point, yet.

 

[scurrier]

I'll add: My fear of the single UPS is basically driven from my distrust of these UPS brands. It seems like they don't have a reputation for taking their designs or their quality control seriously at the consumer price point. There's a thread in these forums where a UPS turns itself off after a power outage even if power is restored and they don't adequately document or warn the user about this. Seems like they don't really value my data like I do (surprise) and so it's my job to educate myself and protect myself from their poor designs.

 

[cyberjock]

Hmm ok. Thanks for the good explanation. I can see how that would happen.

Still... why would that mean that I shouldn't hook PSUs up to different UPSs? Or at least hook one PSU up to the wall directly and the others on the UPS? I agree at some point a system becomes too complex to understand all the failure modes of. But I don't think this idea has reached that point, yet.

PSUs do not like having voltage provided by a UPS. They have major problems, even with the allegedly cleaner UPSes. Companies often call this technology "power factor corrected" or "voltage regulated". Whatever the name, it's meant to help lower the stress on your PSU so the PSU doesn't actually trip offline.

Check out videos like View: http://www.youtube.com/watch?v=3dEJIzwa_3E
and View: http://www.youtube.com/watch?v=bMqW8eqVBaA


Power factor corrected(which all 80%+ rated PSUs are, do even worse. Many PSUs will actually trip offline on a fault condition when on a UPS unless you have a UPS that provides a better quality signal than the cheaper ones.

My system, which I shopped for VERY carefully, has a Corsair 1200w PSU. Granted, my system draw is typically only about 200w, but I needed the extra power since I game on a dual processor system.

When my UPS picks up my PSU actually makes a different noise than it does on line voltage. They just REALLY don't like it.

Go check out videos of people with oscilloscopes and UPSes and PSUs. Some people show in real-time PSUs that trip offline because of the UPS and the crappy output the UPS provides when on the battery.

There's a thread in these forums where a UPS turns itself off after a power outage even if power is restored and they don't adequately document or warn the user about this.

I know.. I said that. My friend was the victim.

Seems like they don't really value my data like I do (surprise) and so it's my job to educate myself and protect myself from their poor designs.

They value your data.. they also value the almighty dollar more than your data and are happy to force you to buy the server grade UPS to avoid this problem.

In my opinion there's just too many variables to accurately determine how "good" or "bad" a situation is. Believe it or not, but the circuit that cross connects the PSUs can have it's own problems when PSUs aren't behaving. There's a chance that one PSU could trip on a fault and that could cause a power loss of the system of sufficient time to cause the system to power off. If you read up on many of the servers with redundant power supplies they will usually tell you to only hook up 1 PSU to an UPS, or to only hook up all of your PSUs to the same UPS. Still others tell you to only buy server-grade full sine wave PSUs. The definition of a "full sine wave" varies from manufacturer to manufacturer too, which only adds more suckage to picking a UPS.

In short, if you're going to go to this kind of complexity to avoid the problems, why not just buy one of those server-grade UPSes that are $500+ dollars and be done with it. They cost more, but they are far better quality than the $150(and less) UPSes. There's just no substitute for using the right tool for the right job, even if the price is higher. Anything less is almost always a compromise. If it wasn't a compromise, someone else would probably be selling said item at the lower price already.

 

[Henry Miller]

Tripple redundant power supplies are generally designed so that you need 2 to run. It may seem like total power draw is low enough that you can get by with just one, but in reality at least two must be there. The reason for the triple redundancy is instead of buying two power supplies at twice the price you buy three that are a little cheaper because they are smaller. You are still protected from the loss of anyone, but only one. Of course the power supplies must be designed to work together and not fight.

I have seen several system where it was the case that one power supply was enough to handle the load. However the hardware was designed so that if you only had one power supply functioning it would shutdown anyway. (We had a cheater power supply in the lab that told the comptuer it was working, normally, but it wasn't actually pluged in. We never told customers about this: they thought at least 2 were required to run and 3 for redundancy). I would be concerned that the plan of 2 UPSs would not actually give him the protection he thinks he gets if one of his UPS turns out to not work well (for any of the reasons listed by others)

 

[scurrier]

Watched the videos. Yeah, that is pretty ugly, it's about as bad as you could approximate a sine wave. But still, the issue that power factor correction PSUs have with UPS devices is independent of the question of whether or not to plug into different ones.

One thing I just read on the net that really gave me some confidence is that these are AC to DC power supplies, so they typically utilize transformers to do the voltage conversion. And those transformers will isolate the outputs. So, (hand waving) they would not be susceptible to minor differences in input voltage. (hand waving) (hand waving) They could even be completely out-of-sync phases! This really resonates with my previous electrical knowledge and so I have some level of confidence in it.

I don't want to buy a single server-grade UPS because, similar to reasons why we are all using consumer-grade HDDs in RAID instead of a single server-grade HDD, I believe that two decent UPSs are better than a single server-grade UPS. Assuming, of course, that there is no legit electrical reason why not to do it.

I did some further searching and there actually seems to be some consensus on the web that using two UPS or a direct/UPS split is a good way to go. The major no-no is to not plug anything directly into wall power, naked of surge protection. You always want surge protection. Another thing to be wary of is keeping the inrush current low enough that it does not cause a "cascading failure" when it switches over from one power source to another.

Here's one such thread with two vendors chiming in and OK'ing the double UPS scheme. Maybe they want to sell two UPSs instead of one. :) With this, I think I have seen enough and will plan to double up on two reasonably-priced UPSs.
http://community.spiceworks.com/topic/154992-cisco-4506-on-ups

I will do some testing before I trust this system with my data.

My chassis manual says I need a regulated UPS. So I will be sure to do that.

 

[scurrier]

Something told me to hit refresh before I posted, but I didn't. Good additional points I hadn't considered, Henry. My chassis definitely will run off of a single supply. Not that that's making me feel much better about it. But, I still feel better about it than I do trusting a single UPS.

 

[scurrier]

Dug around in my chassis manual again, driven by Henry's points. Good news... my manual includes the following information:

This chassis includes triple redundant power supplies which allow the server to remain running while one or two power supplies are removed.

I love having good documentation. Thanks Supermicro.