My Dream System (I think)

[Grogdor]

Deep-discharging UPS batteries significantly shortens their lifespan; don't use more than half of available runtime.

Ideally, something like apcupsd or nut running on your low-power router should be telling your boxes to shut down long before the load eats any significant chunk of battery, then wake-on-LAN'ing them when power returns.

 

[joeschmuck]

Many of the cheap UPS units have a hardwired power On/Off switch vice a soft power switch which when you press it the opposite action occurs from the state it is in. The soft switch UPS units typically will shut down when the battery becomes low wheras the hard switch will just continue to drain the UPS unit. I will admit that the hard switch UPS units "should" still stop using the battery as some point but I'm pretty certain it will still draw power regardless. My APC UPS is a soft switch type and I'm fairly certain it will not power back on by itself. Additionally you don't want an UPS powering back on unless it has charged up a fair bit because if you have several power interruptions, well you could be booting up your FreeNAS system and then another power drop and the UPS doesn't have enough charge to perform a clean shutdown. I run my UPS settings with FreeNAS so it will shutdown after a set period of time to keep it running during a few brownouts or short power losses.

 

[ChriZ]

I agree with your logic, however this will pose a problem only if you set your server to return to previous state and don't set it to perform a clean shutdown after x time on battery.
I mean, if the UPS stays off, and the power is restored for a while, how would you be able to turn the server back on, assuming that not even the BMC has power?
Or am I missing something here?

 

[jgreco]

That's why users with that sort of need would typically get a smart UPS that has those sorts of management features. You set the UPS to power back on after a certain amount of run capacity has been restored.

 

[ChriZ]

Oh, OK then...
Thanks for the information!
I don't own one and didn't know they have such setting.
It is a good reason to get one, I suppose...:)

 

[jgreco]

It's how they differentiate their UPS's these days. Now that every device can have a competent CPU inserted into it for less than a dollar, it no longer makes sense to define "low end UPS" as one containing discrete electronics and maybe a microcontroller when designing a UPS these days. Firmware features basically cost almost nothing to implement but differentiate the product line. If they put all the features on the low end UPS, no one would buy the expensive gear which is where they actually make a profit.

 

[gpsguy]

And on a lot of those expensive UPS's one generally has a service contract to generate even more profit.


Sent from my iPhone using Tapatalk

 

[joeschmuck]

But most home users will not require a fancy UPS unit. Just set the FreeNAS to shutdown after either xx minutes after a power loss or when the UPS hits low power and ensure your FreeNAS motherboard is set to remain powered off. The home user can just power up the system manually because the average home user doesn't NEED the system to turn back on. Just my opinion of course. If I had a ton of money, yea I would like the better UPS but I'm not a lottery winner.

 

[joeschmuck]

Got the new RAM installed and tested overnight, no issues. I'll of course test it more but time to play. I currently have 16GB of RAM allocated to FreeNAS and I don't see any reason to change this. Now to get Sophos all configured and then the UPS configured and working. Once I have all that, it goes back into the basement. My power draw idle is ~57 watts so this is a move in the right direction, not that I saved any money by purchasing a new system but as someone stated earlier in a different thread, it's the "fun factor" which is why we do a lot of things. To me combining everything and learning ESXi is a fun thing, even if it's aggravating at times.

Once the system is in the basement I will start playing with other VMs to add into this system.

 

[joeschmuck]

So I have the UPS shutdown now automated and although it seams simple now, it was a real bear getting an APC UPS with a USB connection to do this job properly which means having it shutdown all the VMs before the battery dies. I will make some changes because I hate waiting for the 10% mark on the UPS before things start shutting down, 20% would be more reasonable to me, which is about 70 minutes of runtime before the shutdown. But the good thing is I can place the system into the basement this week, well now if I wasn't so damn lazy. But I think I'll work on the battery.low threshold value first.

EDIT: Well that was fast and easy to fix, changed the low threshold to 20% and now I'm a happy camper. I'll document this stuff for others to follow and so I'll remember later on.

 

[Grogdor]

Draining your UPS battery past half will significantly shorten its useful lifespan. What's the point of pillaging your batteries for 70 minutes before clean shutdown? Do you live in a rural area with questionable power delivery? Will your internet connection and client devices even still be running, to care that the VMs are up? What kind of chance will the battery have to recharge before being tasked again? Are you a datacenter with backup diesel generators?

Seems most power outages are either brownouts in the "seconds" range, or transformer switchover lasting a minute or three, or otherwise major infrastructure damage from flooding, ice storms etc. which can take hours of extended/repeated shutdowns to repair.

 

[joeschmuck]

Power reliability is hit and miss in my area. Typically there are frequent minor brown-outs and are mostly dictated by weather, and sometimes we have the less frequent short power loss (5 minutes or less), and maybe twice a year during the winter or severe electrical storms we may have a power outage lasting longer than 4 hours. I'm not a datacenter for if I were, I would have very nice UPS systems what have IP support and then I could use the APC appliance to control my ESXi server much easier.

My system is set to shutdown on the battery.low signal which means that I can have multiple power interruptions without having to shutdown the system but once the trigger occurs, even if the power comes back on 1 second later, the system will shut down.

Why do I want to test it, well because I want to know for certain the system will actually shutdown all of the ESXi VMs and ESXi itself and I can't just roll with faith, I need to know. A battery.low level of 10 is cutting it too close and I know that now which is why I'm now set at a value of 30. I'm not set on this value yet because I don't have all my other loads on the UPS right now which will of course change things if I draw more current.

As for damaging the batteries, cycling the batteries a few times will not cause as much harm as you are indicating, especially with the low discharge rate I'm using. These sealed lead-acid (PbA) batteries are deep cycle and are designed for this type of use.

I'm sorry if you don't see the need to test something like this out, I'd just hate to see my UPS die half way through a controlled shutdown.

 

[Grogdor]

As for damaging the batteries, cycling the batteries a few times will not cause as much harm as you are indicating, especially with the low discharge rate I'm using. These sealed lead-acid (PbA) batteries are deep cycle and are designed for this type of use.

The lower you drag them, the more capacity you lose. Test all you want, it's your money after all.

 

[jgreco]

The purpose of a UPS in this role is to protect the host computer from damage. Shutting off a virtualization host without proper shutdown of the hosted VM's can cause lots of damage.

The lower you drag them, the more capacity you lose. Test all you want, it's your money after all.

So I assume you avoid driving your car and keep it locked up in the garage because driving the car puts miles on it...?

@joeschmuck is doing the wise thing here. Not having done this a hundred times before, he doesn't have the cockiness to just set it up and assume it'll do the thing that he purchased it to do. So he's actually testing. Because if you don't test and the thing you spent a lot of money on fails to work as intended, then that's a lot of money pissed away on a FAIL.

 

[joeschmuck]

The lower you drag them, the more capacity you lose. Test all you want, it's your money after all.

This is completely acceptable to me and if by chance my batteries needed to be replaced after all my testing, new batteries still cost less than taking my family out to dinner. It's all about how you justify it.

@jgreco Thanks for tossing up that justification, it's spot on.

And I am considering using a different method now but I have one last test and this time it will be setting the battery.low value to 80% to verify things start shutting down properly. I was able to get the system to shutdown at 20% only one time, never able to replicate it no matter what I've done so far. now I'll test it at 80% from a full battery to see if there was some issue with the UPS not clearing some flag. If this test fails then I'm off to use a different method, not preferred but it all just needs to work properly.

But hey, the FreeNAS has been running solid for the last month, not a single issue. maybe I should encrypt the drives now and see if I can toss in a heap of trouble :D

 

[Jailer]

The lower you drag them, the more capacity you lose. Test all you want, it's your money after all.

While this is true how frequently it happens plays a major role in it. The impact of running them down to 20% infrequently will be minimal.

 

[jgreco]

While this is true how frequently it happens plays a major role in it. The impact of running them down to 20% infrequently will be minimal.

Strongly agree. The batteries are consumable, as long as they're consumed for some good reason, it is just part of the cost of owning a UPS. Certainly you don't want to be cycling them unnecessarily, leave that for the Tesla Powerwall ;-) ;-)

 

[Grogdor]

The purpose of a UPS in this role is to protect the host computer from damage. Shutting off a virtualization host without proper shutdown of the hosted VM's can cause lots of damage.

@joeschmuck is doing the wise thing here. Not having done this a hundred times before, he doesn't have the cockiness to just set it up and assume it'll do the thing that he purchased it to do. So he's actually testing. Because if you don't test and the thing you spent a lot of money on fails to work as intended, then that's a lot of money pissed away on a FAIL.

It seems that you misunderstood, as nowhere did I advocate blind faith in the UPS setup nor improper shutdowns, just tried to spare his batteries unnecessary wear from a low shutdown threshold which will result in diminished capacity.

So I assume you avoid driving your car and keep it locked up in the garage because driving the car puts miles on it...?

Nah, but I only do burnouts on a tire I'm about to replace. Even clean then wax the chain, the occasional oil change, maybe even a filter too.

 

[jgreco]

Still not particularly worrying. Here at home, we're on the far end of a utility run that basically goes through Tarzan's jungle, about two miles through backyards with fifty or sixty year old trees. We lose power once or twice a year, the A rail UPS (~10 min runtime) always discharge fully before transferring onto the B rail UPS's which are generator-backed. Since replacing them about 4 years ago they've probably been through at least 8 full cycles and they're still showing as very healthy.

I'm actually quite pleased with the new APC SMX units, though I don't know how "FreeNAS-compatible" they might be. Unlike the old APC Smart-UPS units, I haven't caught any of these doing any battery-cooking, and in fact we managed to hit the five year mark before we finally got any sort of battery warning. I had purchased a desktop SMT 750 to experiment with back in ~2010, and it finally gave it up a few months ago and needed a new battery pack. We had one other SMT unit arrive DOA but I think that may be the total extent of the bad things I can say about the new models. I was always griping about replacing batteries in the older Smart-UPS's... with several dozen units there was always one needing replacing, and usually the batteries had cooked and bulged without any alarm or warning.

 

[joeschmuck]

The Dream System is Complete! (Tagline Somewhat Updated)

I finally moved it back to the basement which means that FreeNAS itself is rock solid and operates perfectly. I am in the process of writing up steps I took to create this machine and trying to be detailed but it's actually very time consuming do this, however I need to do this so I don't forget something I did just in case it breaks, at least I can try to fix it. If anyone has a specific question, I'll try to answer it.

I am running my original Sophos machine and my VM Sophos machine on two different networks and will migrate over once I have a few hours to mess around with it and ensure everything is good. Unfortunately restoring a backup configuration from my bare metal machine and to my VM machine didn't go well so I had to recreate all the rules by hand. I'm sure there is a better way to do this, I just need to figure it out as the bare metal machine took me months to get right.

Running other VMs is working great as well although it isn't much different that running VMWare Workstation.