Configuring UPS support for single or multiple FreeNAS servers

[elvisimprsntr]

@Basil Hendroff
1. Depending on make/model, UPS batteries have an expected life of 3 to 5 years. Set up a schedule to proactively replace them.
2. Some UPS models perform an automatic battery self test and will report UPS.STATUS: OL REPLACEBATT.

I had proactively ordered a replacement battery when 2 days later my APC UPS reported my battery needed to be replaced. Hot swapped the battery when the replacement arrived. The NUT driver did not allow me to update the battery date, so I had to temporarily install the APCUPS driver on my pfsense firewall to change the date.

Code:

[2.5.2-RELEASE]/root: upsc apcups
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2021/07/30
battery.runtime: 2373
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 26.0
battery.voltage.nominal: 24.0
device.mfr: American Power Conversion
device.model: Back-UPS BX1500G
device.serial: 3B1026X48797
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 139
input.transfer.low: 88
input.transfer.reason: input voltage out of range
input.voltage: 121.0
input.voltage.nominal: 120
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.firmware: 866.L4 .D
ups.firmware.aux: L4
ups.load: 16
ups.mfr: American Power Conversion
ups.mfr.date: 2010/06/27
ups.model: Back-UPS BX1500G
ups.productid: 0002
ups.realpower.nominal: 865
ups.serial: 3B1026X48797
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

3. Some UPS models you can initiate a UPS self test using the UPSCMD command.

Code:

ups.test.result: Done and passed

On my UPS, I have to issue the stop command between such successive deep or quick tests. I think that allows you to poll UPS.TEST.RESULT before stopping or initiating a new test.

Code:

[2.5.2-RELEASE]/root: upscmd -l apcups@localhost
Instant commands supported on UPS [apcups]:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.stop - Stop a shutdown in progress
test.battery.start.deep - Start a deep battery test
test.battery.start.quick - Start a quick battery test
test.battery.stop - Stop the battery test
test.panel.start - Start testing the UPS panel
test.panel.stop - Stop a UPS panel test

4. If your UPS does not report battery health, consider getting a UPS that does or otherwise take steps to mitigate the risk.

 

[Spearfoot]

During a recent power outage, I noticed a pair of backup CORE servers dropped like a lead balloon rather than shutting down gracefully when the UPS they were connected to (in a master-slave arrangement) reached a low battery condition. I realised then that the UPS batteries were no longer holding their charge. I replaced the batteries this w/e and all is well again.

The question I have is 'Short of periodically testing the UPS, is there a way in software to determine if UPS batteries are nearing EoL and should be replaced'. This is the output for the upsc command on the PowerShield Defender1200VA UPS the CORE servers are connected to.

Code:

root@truenas-b2:~ # upsc ups
battery.charge: 100
battery.voltage: 26.10
battery.voltage.high: 26.00
battery.voltage.low: 20.80
battery.voltage.nominal: 24.0
device.type: ups
driver.name: blazer_usb
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.internal: 0.12
input.current.nominal: 5.0
input.frequency: 50.1
input.frequency.nominal: 50
input.voltage: 233.6
input.voltage.fault: 233.6
input.voltage.nominal: 240
output.voltage: 233.6
ups.beeper.status: enabled
ups.delay.shutdown: 30
ups.delay.start: 180
ups.load: 17
ups.productid: 5161
ups.status: OL
ups.type: offline / line interactive
ups.vendorid: 0665

That's a good question and I wish I knew an answer other than: test the UPS regularly.
This upscmd command shows the following list of available commands supported by one of my UPS systems: upscmd -l ups

Code:

root@bilbo:~ # upscmd -l ups
Instant commands supported on UPS [ups]:

beeper.disable - Description unavailable
beeper.enable - Description unavailable
beeper.mute - Description unavailable
beeper.off - Description unavailable
beeper.on - Description unavailable
load.off - Description unavailable
load.off.delay - Description unavailable
shutdown.reboot - Description unavailable
shutdown.stop - Description unavailable
test.battery.start.deep - Description unavailable
test.battery.start.quick - Description unavailable
test.battery.stop - Description unavailable
test.panel.start - Description unavailable
test.panel.stop - Description unavailable

But when I tried to run test.battery.start.quick, I got this error:

Code:

root@bilbo:~ # upscmd -u root -p root_password ups test.battery.start.quick
Unexpected response from upsd: ERR ACCESS-DENIED

Seems odd that root can't start a quick test!

I researched this a little and it seems I need to add another user to the NUT UPS settings with escalated privileges -- but that's as far as I've gotten as my attention was drawn away by other things.

I still believe that regular testing by means of the upscmd is probably going to be the best way to discover failing batteries.

 

[elvisimprsntr]

I researched this a little and it seems I need to add another user to the NUT UPS settings with escalated privileges -- but that's as far as I've gotten as my attention was drawn away by other things.

 

[Spearfoot]

View attachment 49843

This works; I added an extra user with instcmds=all privileges, and now I can run the battery tests.

With this in place, it should be fairly easy to come up with a cron task to regularly test the UPS and report the results.

 

[elvisimprsntr]

This works; I added an extra user with instcmds=all privileges, and now I can run the battery tests.

With this in place, it should be fairly easy to come up with a cron task to regularly test the UPS and report the results.

Assuming you have UPS notifications set up and your UPS reports battery health, I think NUT will automatically sent a message if the UPS reports the battery needs to be replaced. My pfsense firewall running NUT did.

It does not appear there is a notify event for UPS self test status. Theoretically, it should result in a REPLBATT notification.

UPSMON(8)

Code:

root@NAS-1[/usr/local/etc/nut]# cat upssched.conf
CMDSCRIPT   "/usr/local/bin/sudo /usr/local/bin/custom-upssched-cmd"
PIPEFN      /var/db/nut/upssched.pipe
LOCKFN      /var/db/nut/upssched.lock

AT NOCOMM   * EXECUTE NOTIFY-NOCOMM
AT COMMBAD  * START-TIMER NOTIFY-COMMBAD 10
AT COMMOK   * CANCEL-TIMER NOTIFY-COMMBAD COMMOK
AT FSD      * EXECUTE NOTIFY-FSD
AT LOWBATT  * EXECUTE NOTIFY-LOWBATT
AT ONBATT   * EXECUTE NOTIFY-ONBATT
AT ONLINE   * EXECUTE NOTIFY-ONLINE
AT REPLBATT * EXECUTE NOTIFY-REPLBATT
AT SHUTDOWN * EXECUTE NOTIFY-SHUTDOWN
AT ONBATT   * START-TIMER SHUTDOWN 300
AT ONLINE   * CANCEL-TIMER SHUTDOWN

 

[elvisimprsntr]

Simple BASH script

Code:

#!/bin/sh

/usr/local/bin/upscmd -u {user} -p {pw} apcups@localhost test.battery.stop "1"

sleep 1

/usr/local/bin/upscmd -u {user} -p {pw} apcups@localhost test.battery.start.deep "1"

sleep 1

Added pfsense CRON. For TrueNAS put the script on a data pool and add execute permissions (chmod a+x)

 

[Spearfoot]

Simple BASH script

Code:

#!/bin/sh

/usr/local/bin/upscmd -u {user} -p {pw} apcups@localhost test.battery.stop "1"

sleep 1

/usr/local/bin/upscmd -u {user} -p {pw} apcups@localhost test.battery.start.deep "1"

sleep 1

Added pfsense CRON. For TrueNAS put the script on a data pool and add execute permissions (chmod a+x)

View attachment 49847

Did you intend to run test.battery.stop before test.battery.start.deep? I thought it would need to be the other way 'round, and probably with a longer sleep interval between starting the test and stopping it.

The 'quick' test seems to run for a pre-determined interval and if so may not need an explicit 'stop' command. But I haven't established this for certain.

I'm thinking the 'deep' test will probably need to be stopped after a reasonable period of time -- no idea what 'reasonable' is at this point.

My 'fancy' rack-mount SmartUPS 2200 w/ network card is configurable via a web interface; I set it up long ago to run a weekly self-test. It seems the home/non-enterprise units are trickier to set up.

 

[ChrisRJ]

The question I have is 'Short of periodically testing the UPS, is there a way in software to determine if UPS batteries are nearing EoL and should be replaced'.

Unfortunately not. The only way to test the effective capacity of a battery is to perform at least one charge/discharge cycle. The poor man's approach is to unplug and measure the time. That gives you the current runtime, but that result does not necessarily say much about the next such cycle. To be really sure, you need a battery tester. And that is not something you can buy for USD/EUR 100-200. These devices are really expensive and also need to be calibrated for the battery under test. E.g. a car starter battery has very different characteristics from a UPS battery and is built in a fundamentally different way.

Long story short: For what we have at home it's not worth the hazzle, unless you know someone who has access to such equipment.

 

[Basil Hendroff]

@elvisimprsntr @Spearfoot @ChrisRJ Thanks for the feedback!

If your UPS does not report battery health, consider getting a UPS that does or otherwise take steps to mitigate the risk.

I'm caught between a rock and a hard place with this. The PowerShield Defender 1200VA is the only Australian made UPS on the NUT h/w compatibility list. There's not a lot of h/w technology that's Australian built these days so I feel it's my patriotic duty to support it. Plus it's locally made in my home city

I can run the battery tests.

The test.battery start, stop and deep commands either fail or are not supported on the UPS. Curiously, while the UPS responds to test.battery.start.quick, ups.test.result isn't a supported variable for the blazer_usb driver it uses, so I don't appear to be able to view the result of the test. It looks like I have no choice, but to periodically connect a laptop to the com port of each UPS and run a battery test using the NetGuard s/w specifically designed to operate with the UPS. Clumsy, but doable.

 

[elvisimprsntr]

Did you intend to run test.battery.stop before test.battery.start.deep? I thought it would need to be the other way 'round, and probably with a longer sleep interval between starting the test and stopping it.

The 'quick' test seems to run for a pre-determined interval and if so may not need an explicit 'stop' command. But I haven't established this for certain.

I'm thinking the 'deep' test will probably need to be stopped after a reasonable period of time -- no idea what 'reasonable' is at this point.

My 'fancy' rack-mount SmartUPS 2200 w/ network card is configurable via a web interface; I set it up long ago to run a weekly self-test. It seems the home/non-enterprise units are trickier to set up.

When you issue the stop command it clears the previous test result, which prevents you from polling the test status afterwards

 

[Basil Hendroff]

This caught my eye under CORE UI System/Alert Settings...

I've seen all the other UPS alerts under various conditions, but I've never seen the battery replacement alert. I wonder what the trigger is for this alert, and whether it's a function of the UPS model and NUT driver?

Some UPS models perform an automatic battery self test and will report UPS.STATUS: OL REPLACEBATT.

I wonder if this is the trigger? If so, then it is dependent on the battery model and NUT driver.

 

[Basil Hendroff]

OK, I may be on to something here.

Curiously, while the UPS responds to test.battery.start.quick, ups.test.result isn't a supported variable for the blazer_usb driver it uses, so I don't appear to be able to view the result of the test.

So, I've determined I can run the quick test, but I can't view the result. I'm still able to view variables though while the test is in progress. For example, prior to a test, this is what I might see...

Code:

admin@truenas-b2:~ % upsc ups
battery.charge: 100
battery.voltage: 26.90
battery.voltage.high: 26.00
battery.voltage.low: 20.80
battery.voltage.nominal: 24.0
device.type: ups
driver.name: blazer_usb
driver.parameter.pollinterval: 2
driver.parameter.port: /dev/ugen0.2
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.internal: 0.12
input.current.nominal: 5.0
input.frequency: 50.1
input.frequency.nominal: 50
input.voltage: 239.6
input.voltage.fault: 239.6
input.voltage.nominal: 240
output.voltage: 239.6
ups.beeper.status: enabled
ups.delay.shutdown: 30
ups.delay.start: 180
ups.load: 18
ups.productid: 5161
ups.status: OL
ups.type: offline / line interactive
ups.vendorid: 0665

During the quick test, I might see this...

Code:

battery.charge: 96
battery.voltage: 25.80
battery.voltage.high: 26.00
battery.voltage.low: 20.80
battery.voltage.nominal: 24.0
device.type: ups
driver.name: blazer_usb
driver.parameter.pollinterval: 2
driver.parameter.port: /dev/ugen0.2
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.internal: 0.12
input.current.nominal: 5.0
input.frequency: 50.1
input.frequency.nominal: 50
input.voltage: 241.6
input.voltage.fault: 241.6
input.voltage.nominal: 240
output.voltage: 237.6
ups.beeper.status: enabled
ups.delay.shutdown: 30
ups.delay.start: 180
ups.load: 20
ups.productid: 5161
ups.status: OL BYPASS CAL
ups.type: offline / line interactive
ups.vendorid: 0665

Note the change in the variable ups.status. Assuming there's enough information in these variables, I don't think it will be too difficult to script something to test for battery capacity degradation. The question then is 'On what basis will I decide that battery replacement (or UPS upsizing) is required?' Thoughts anyone?

 

[elvisimprsntr]

I believe UPS.STATUS will read this for replace battery

Code:

ups.status: OL REPLBATT

Which should automatically trigger a notification event and send you an email if you have it set up.

Code:

AT REPLBATT * EXECUTE NOTIFY-REPLBATT

 

[Basil Hendroff]

I believe UPS.STATUS will read this for replace battery

Code:

ups.status: OL REPLBATT

Which should automatically trigger a notification event and send you an email if you have it set up.

Code:

AT REPLBATT * EXECUTE NOTIFY-REPLBATT

If it's this simple, then all I need to do is set up a periodic cron job with a line something like...

Code:

upscmd -u user-p password ups test.battery.start.quick

What I think I'll do is restore the dead batteries in the UPS and run a test to confirm a notification event is triggered.

 

[Basil Hendroff]

What I think I'll do is restore the dead batteries in the UPS and run a test to confirm a notification event is triggered

Well, that failed miserably. Having dead batteries installed is akin to having no batteries installed at all. Doing a quick battery test killed my backup servers immediately. I'm not able to easily verify that a quick battery test will trigger a notification event if the batteries need replacing.

@Spearfoot @elvisimprsntr Assuming a notification event is triggered as batteries age, then the cron single-line command below run periodically should eliminate the need for manual UPS testing.

Code:

upscmd -u user -p password ups test.battery.start.quick

The main issue I see with this is; if the batteries have developed an internal fault, running the command will likely crash any connected servers. This might be difficult to troubleshoot as it's easy to forget a cron job has been set up. Scripting a wrapper around the command with some checks may be a better way to go.

 

[Basil Hendroff]

A couple of follow-up questions...

1. Shutdown Command field

Any idea what the purpose of the Shutdown Command field is? Even without a command defined, the server will gracefully shut down when the UPS reaches low battery.

2. Ambiguous statement

This extract from the NUT driver BLAZER_USB(8)...

runtimecal = value,value,value,value
Parameter used in the (optional) runtime estimation. This takes two runtimes at different loads. Typically, this uses the runtime at full load and the runtime at half load. For instance, if your UPS has a rated runtime of 240 seconds at full load and 720 seconds at half load, you would enter

runtimecal = 240,100,720,50
The first load should always be higher than the second. If you have values available for loads other than 100 and 50 % respectively, you can use those too, but keep them spaced apart as far as reasonably possible. Just don’t get too close to no load (prediction of runtime depends more on idle load for the battery then).

Is the last paragraph a convoluted way of saying fewer values are better? For example, if I had this product run chart...

%VA	10	20	30	40	50	60	70	80	90	100
runtime (seconds)	6720	3060	1800	1200	960	780	600	480	360	300

... then this...

Code:

runtimecal = 300,100,960,50

...is more useful than this?

Code:

runtimecal = 300,100,360,90...6720,10

 

[Spearfoot]

Well, that failed miserably. Having dead batteries installed is akin to having no batteries installed at all. Doing a quick battery test killed my backup servers immediately. I'm not able to easily verify that a quick battery test will trigger a notification event if the batteries need replacing.

@Spearfoot @elvisimprsntr Assuming a notification event is triggered as batteries age, then the cron single-line command below run periodically should eliminate the need for manual UPS testing.

Code:

upscmd -u user -p password ups test.battery.start.quick

The main issue I see with this is; if the batteries have developed an internal fault, running the command will likely crash any connected servers. This might be difficult to troubleshoot as it's easy to forget a cron job has been set up. Scripting a wrapper around the command with some checks may be a better way to go.

Yes, monitoring these gizmos is a conundrum... My 2U rackmount APC SU2200 supports automatic testing, which I have configured to run once a week. Plus it will light up a warning LED when appropriate.

Home-use UPS systems usually have a warning light of some kind, too, but as you've pointed out it's harder to monitor them effectively.

 

[Spearfoot]

A couple of follow-up questions...

1. Shutdown Command field

View attachment 49893

Any idea what the purpose of the Shutdown Command field is? Even without a command defined, the server will gracefully shut down when the UPS reaches low battery.

2. Ambiguous statement

This extract from the NUT driver BLAZER_USB(8)...



Is the last paragraph a convoluted way of saying fewer values are better? For example, if I had this product run chart...

%VA	10	20	30	40	50	60	70	80	90	100
runtime (seconds)	6720	3060	1800	1200	960	780	600	480	360	300

... then this...

Code:

runtimecal = 300,100,960,50

...is more useful than this?

Code:

runtimecal = 300,100,360,90...6720,10

Shutdown command? NUT may have a default it uses if you leave this blank, but here's what I use:

On FreeNAS/TrueNAS virtual machines: ssh root@esxiserver poweroff
This tells the FreeNAS VM's host ESXi server to shut itself down, which -- if you've configured it as described in this resource -- will gracefully shut down all virtual machines (including FreeNAS).

On stand-alone FreeNAS/TrueNAS servers: /sbin/shutdown -p now
Just a standard FreeBSD 'power off' command. (Make sure the path is correct, as shutdown might be located in a different directory depending on the FreeBSD version.)

Don't have any advice to offer on the VA derating curve. I take the other approach, shutting down after a fixed interval rather than whenever the batteries have discharged. I want to go easy on them as much as possible, to preserve their service life.

 

[Basil Hendroff]

I take the other approach, shutting down after a fixed interval rather than whenever the batteries have discharged. I want to go easy on them as much as possible, to preserve their service life.

I wasn't aware of this! What's the theory behind this?

 

[Spearfoot]

I wasn't aware of this! What's the theory behind this?

The physics of batteries... Each discharge/recharge cycle slightly reduces the capacity of the battery; the deeper the discharge, the more the reduction in capacity.

UPS batteries are usually rated for a few hundred such cycles, so you're not likely to suffer enough power outages to suffer too much from this effect, but I like to play it safe.

Also, waiting for the batteries to discharge to a certain level before shutting down is a gamble; what if you underestimate how long it takes your systems to gracefully shut down and find that your battteries don't have enough capacity to last that long? Ooops!

So, again, I play it safe. If the power's been off for 2 or 3 minutes, I shut everything down.