Network UPS Support information/changes

[Milhouse]

(Continuation of this thread which I hijacked)

FreeNAS 8.0.1-RELEASE currently supports locally attached UPS, but not network UPS. The aim of this thread is to assist developers in moving towards supporting both local and network UPS.

To get FreeNAS 8.0.1-R (r8081) to monitor a network UPS (called ups, on host 192.168.0.2), the following changes are necessary to hard-coded localhost variables:

1. Modifying rc.conf.local in _nut_config() as follows:

Code:

#        echo "nut_upslog_ups=\"ups@localhost\""
        echo "nut_upslog_ups=\"ups@192.168.0.2\""

In the above, both the name of the UPS ("ups") and it's host ("localhost") are hardcoded, when both of these should be variables set by the GUI.

2. Changing /usr/local/etc/nut/upsmon.conf:

Code:

#MONITOR ups 1 upsmon fixmepass master
MONITOR ups@192.168.0.2 1 upsmon fixmepass slave

Again, "ups" is the name of the remote UPS, and 192.168.0.2 the host IP address.

Notice that the monitor type has changed to slave, from master.

The user and password no longer seem to be important when in slave mode, although this could (and indeed most likely is) just my setup lacking authentication (the user on my UPS host is certainly not upsmon, nor is the password fixmepass).

3. In /etc/local/rc.d/nut_upsmon, the following line is erroneous:

Code:

nut_upsmon_flags=${nut_upsmon_flags-"localhost"}

The above line causes /usr/local/sbin/upsmon to run with the argument "localhost", which while not a problem is not a valid argument so it is ignored - the setting of nut_upsmon_flags isn't required in nut_upsmon and can be removed.



With the above changes, /etc/local/rc.d/nut_upsmon and /etc/local/rc.d/nut_upslog should be able to start, although unfortunately the config files are overwritten with each service start, which makes testing tricky!

But at least with changes #1 & #2, nut_upslog and nut_upsmon can be started manually, and will log and monitor appropriately. Change #3 is just a nit.

So in summary, the UPS service settings GUI needs to allow the user to choose whether to monitor a local or remote (network) UPS.

When monitoring a network UPS, the UPS name - not necessarily the same as the current Identifier field, though it could be re-used as such, I guess - and the IP/hostname need to be specified, and then used in place of localhost when writing out the configs.

A valid username and password *may* also need to be specified, depending on remote setup/authentication requirements - currently the UPS user is hardcoded as upsmon.

In addition, a way to view the battery status is sorely missing from the GUI - the output from upsc should be sufficient, eg. upsc ups@localhost or upsc ups@192.168.0.2 etc.).

Ofc, there is a dummy-ups, but that doesnt emulate SNMP packets over the network ;)

Which OIDs are you interested in? Does the local UPS support SNMP, I don't see it loading or specifying any UPS MIBs.

 

[Milhouse]

In addition, when using a network UPS it doesn't appear as though upsd is required at all, therefore the prestart and poststop functions in /etc/local/rc.d/nut are not required, and the dependency on upsd.pid should be changed (I think the pgrep check - which is failing as there is no upsd - is causing the service startup to also fail).

If "Network or Local" (boolean), "Remote IP Address/Hostname" (string, mandatory for Network), "Remote Port" (string, or integer? Allow blank.), "Remote User" (string, mandatory for Network) and "Remote UPS identifier" (string, eg. "ups", mandatory for Network) can be added to the GUI/database, then most of the required code changes can be localised to rc.conf.local and ix-ups as it is these scripts that generate the configuration and start the NUT services (using mostly hard-coded localhost and "ups" identifiers).

However, this dependency on upsd in the nut service is possibly more problematic, and I'm not entirely sure how to solve that right now.

 

[William Grzybowski]

Looks like you have it all :)
I heard developers accept patches and freenas is community

 

[Milhouse]

Looks like you have it all :)
I heard developers accept patches and freenas is community

I'll be the first to admit I don't have the skill to provide patches for the GUI and settings database, but I had hoped you'd be willing to work with the community to identify the pain points preventing you from getting this working since you lack the equipment.

 

[William Grzybowski]

I'm, we are.

However by your comments everywhere looks like the developers don't have anything else to do but fix something that affects you.
Please understand that we have a lot of tasks to accomplish, and some of them are considered higher priority...

Network UPS will be there, at some point, with the help of community to test it (as it has been happening for long time), be patient.

 

[Milhouse]

However by your comments everywhere looks like the developers don't have anything else to do but fix something that affects you.
Please understand that we have a lot of tasks to accomplish, and some of them are considered higher priority...

Not everywhere, just where anyone expects to (or wonders if they should) use FreeNAS8 in a "pro" environment, for which it is not currently suited (whereas FreeNAS 7, is). And yes, I appreciate there are other more pressing issues, such as failed getting failed disk notifications to work, and reliable disk replacement etc. but the slow UPS progress (network or otherwise) is frustrating, which leads to the following...

Network UPS will be there, at some point, with the help of community to test it (as it has been happening for long time), be patient.

Bear in mind my post here is in relation to your previous comment from the other thread (reproduced below) which seemed to suggest you wanted help - obviously I misunderstood. However the problems with UPS functionality would indicate that code changes are never actually tested before release - I guess we now know why, and I won't hold my breath for Network UPS support but will continue to point out it's not working should anyone have expectations otherwise.

There are plans, yes, however this gonna be very hard to achieve without community patches as I don't have the equipment...

 

[William Grzybowski]

... and I won't hold my breath for Network UPS support but will continue to point out it's not working should anyone have expectations otherwise.

Network UPS?
This is your response in some thread:

Unfortunately however, FreeNAS 8.0.2 out-of-the-box isn't IMHO a suitable candidate for satisfying either of these priorities right now, as rsync can't push data to remote servers (unless you write your own script), and UPS support is buggy with offers of help to improve UPS support going unanswered.

AFAIK UPS support works, but for local ups :), which was the case for the guy in the respective thread

Also your statement about rsync is wrong, it works perfectly fine using rsync daemon (rsync over ssh is the one that doesn't)

Either way, enough of this, I won't reply to this thread anymore, it has gone too far for so little

 

[vanhaakonnen]

I have a ups nut-server installation on another machine. So I am very interested in using freenas with nut client support.

I have tried to edit some configfiles a few versions ago - but without success.

So - I am willing to try and verify upcoming client-nut implementations. In really big environments with one big usv there is no way to connect each server directly to the ups.

 

[Milhouse]

I have tried to edit some configfiles a few versions ago - but without success.

I did get Network UPS working with FreeNAS 7 just fine, here, but the current design of FreeNAS 8 makes it more difficult to hack in a similar solution, though the changes are more or less identical. I'll take another look at getting it working and post any further updates.

 

[Milhouse]

It's not nice, but the following commands will configure FreeNAS 8.0.2-RELEASE to monitor a remote UPS.

Replace 192.168.0.2 with your IP address (it occurs in two places):

Code:

mount -uw /

# Add IP address of remote UPS, and switch from master to slave
cd /conf/base/etc/rc.d
sed -i .bak 's/MONITOR ${ups_identifier} 1 upsmon ${ups_masterpwd} master/MONITOR ${ups_identifier}@192.168.0.2 1 upsmon ${ups_masterpwd} slave/' ix-ups
cp ix-ups /etc/rc.d
rm ix-ups.bak

# rc.conf.local uses a hard-coded "ups" identifier and hard-coded localhost - replace latter, possibly former too
cd /conf/base/etc
sed -i .bak 's/ups@localhost/ups@192.168.0.2/g' rc.conf.local
cp rc.conf.local /etc
rm rc.conf.local.bak

# Change GUI so that it depends on upslog instead of upsd (the latter is no longer required for network support)
cd /usr/local/www/freenasUI/middleware
sed -i .bak 's/upsd/upslog/g' notifier.py
python -m compileall .

mount -ur /
reboot

Following the reboot, configure the UPS service. You can use whatever USB driver and port you want (usbhid-ups and /dev/ugen0.1 work fine) as they're no longer relevant. Leave "Remote Monitor" unticked.

Once the UPS is configured, you should be able to start the UPS service and FreeNAS will be monitoring your network UPS. Check /var/log/messages for details, and /var/log/ups.log for battery status. Run "upsc ups@192.168.0.2" on FreeNAS to view battery details from the remote UPS.

During subsequent reboots, FreeNAS will attempt to load the USB driver - which will fail - but this isn't important as the USB UPS driver is no longer required when monitoring a remote UPS.

Now, although these changes work, the current 8.0.2 UPS configuration doesn't discriminate between being on battery and low battery - it will ALWAYS shut down as soon as the UPS (local or network, doesn't matter) goes on battery (ONBATT), even if the GUI requests a low battery shutdown (LOWBATT).

This is due to an incorrect configuration in upssched.conf, which always triggers the shutdown command when the ONBATT notification is received.

The following fix will ensure the UPS shuts down on LOWBATT, and not ONBATT:

Code:

cd /conf/base/etc/rc.d
sed -i .bak 's/AT ONBATT   \* START-TIMER/#AT ONBATT   \* START-TIMER/g' ix-ups
cp ix-ups /etc/rc.d
rm ix-ups.bak

Results of testing....

Code:

[root@freenas] /etc/local/nut# tail -f /var/log/messages
Oct 19 00:08:59 freenas freenas: Starting nut_upsmon.
Oct 19 00:08:59 freenas freenas: UPS: ups@192.168.0.2 (slave) (power value 1)
Oct 19 00:08:59 freenas freenas[1927]: Executing: /usr/sbin/service nut_upslog restart
Oct 19 00:08:59 freenas freenas: Stopping nut_upslog.
Oct 19 00:08:59 freenas freenas: Starting nut_upslog.
Oct 19 00:08:59 freenas freenas[1927]: Executing: /bin/pgrep -F /var/db/nut/upslog.pid upslog
# Mains power removed (ONBATT)
Oct 19 00:13:04 freenas upsmon[2717]: UPS ups@192.168.0.2 on battery
# Mains power restored (ONLINE)
Oct 19 00:34:45 freenas upsmon[2717]: UPS ups@192.168.0.2 on line power
# Mains power removed (ONBATT)
Oct 19 00:35:35 freenas upsmon[2717]: UPS ups@192.168.0.2 on battery
# Low Battery (LOWBATT)
Oct 19 00:43:50 freenas upsmon[2717]: UPS ups@192.168.0.2: forced shutdown in progress
Oct 19 00:43:50 freenas upsmon[2717]: UPS ups@192.168.0.2 battery is low
Oct 19 00:43:50 freenas upsmon[2717]: Executing automatic power-fail shutdown
Oct 19 00:43:50 freenas upsmon[2717]: Auto logout and shutdown proceeding
Oct 19 00:43:50 freenas upssched-cmd: Unrecognized command: FSD
Oct 19 00:43:50 freenas upssched-cmd: issuing shutdown
Oct 19 00:43:50 freenas shutdown: power-down by root:
Oct 19 00:43:54 freenas ntpd[1669]: ntpd exiting on signal 15
<shutdown achieved>

 

[William Grzybowski]

About shutting down ONBATT instead of LOWBATT i've changed the /usr/local/bin/custom-upssched-cmd script that may solve this issue,
if someone is willing to try it out, please grab it in http://support.freenas.org/browser/...es/usr/local/bin/custom-upssched-cmd?rev=8331

(making sure it has +x permission)

 

[Milhouse]

About shutting down ONBATT instead of LOWBATT i've changed the /usr/local/bin/custom-upssched-cmd script that may solve this issue,
if someone is willing to try it out, please grab it in http://support.freenas.org/browser/...es/usr/local/bin/custom-upssched-cmd?rev=8331

(making sure it has +x permission)

That should work, however assuming that upssched.conf remains configured as follows (which is how it is in 8.0.2-RELEASE)

Code:

CMDSCRIPT   /usr/local/bin/custom-upssched-cmd
PIPEFN      /var/db/nut/upssched.pipe
LOCKFN      /var/db/nut/upssched.lock

AT COMMBAD  * START-TIMER COMMBAD 10
AT COMMOK   * CANCEL-TIMER COMMBAD COMMOK
AT FSD      * EXECUTE FSD
AT LOWBATT  * START-TIMER LOWBATT 30
AT LOWBATT  * EXECUTE EMAIL
AT ONBATT   * START-TIMER ONBATT 30
AT ONBATT   * EXECUTE EMAIL
AT ONLINE   * CANCEL-TIMER ONBATT ONLINE
AT ONLINE   * CANCEL-TIMER LOWBATT ONLINE
AT REPLBATT * EXECUTE REPLBATT
AT SHUTDOWN * EXECUTE SHUTDOWN

then two emails will be kicked out when ONBATT or LOWBATT occurs, once for the EXECUTE EMAIL command, and then 30 seconds later if/when the timer expires.

The following change might be better:

Code:

case $1 in
        "ONBATT"|"LOWBATT")
                if [ "${ups_shutdown}" = "lowbatt" && "$1" = "LOWBATT" ] || [ "${ups_shutdown}" = "batt" && "$1" = "ONBATT" ]; then
                        if [ "${ups_emailnotify}" -eq 1 ]; then
                                echo "$1" | mail -s "$(echo "${ups_subject}"|sed "s/%d/$(date)/"|sed "s/%h/$(hostname)/")" "${ups_toemail}"
                        fi
                        logger -t upssched-cmd "issuing shutdown"
                        /usr/local/sbin/upsmon -c fsd
                fi
                ;;
        "SHUTDOWN|FSD")
                if [ "${ups_emailnotify}" -eq 1 ]; then
                        echo "$1" | mail -s "$(echo "${ups_subject}"|sed "s/%d/$(date)/"|sed "s/%h/$(hostname)/")" "${ups_toemail}"
                fi
                ;;
        "EMAIL"|"ONLINE"|"REPLBATT"|"COMMBAD"|"COMMOK")
                if [ "${ups_emailnotify}" -eq 1 ]; then
                        if [ "$1" = "ONLINE" ]; then
                                msg="ONLINE"
                        else
                                msg="$NOTIFYTYPE"
                        fi
                        echo "${msg}" | mail -s "$(echo "${ups_subject}"|sed "s/%d/$(date)/"|sed "s/%h/$(hostname)/")" "${ups_toemail}"
                fi
                ;;
        *)
                logger -t upssched-cmd "Unrecognized command: $1"
                ;;
esac

as this would mean the second email is only kicked out if the box is about to go down, which is what you want - I don't need two emails telling me the UPS is now on battery power as the second email is repeating what I already know from the first email.

Note also that I added support for REPLBATT, COMMBAD and COMMOK, so that an email is sent should these notifications occur, otherwise they'll be logged as unrecognised commands, and knowing your server has lost touch with it's UPS is quite important so deserves an email! :)

So, with the changes above, the following email notification workflow should occur:

Shutdown on ONBATT:

Code:

ONLINE -> ONBATT => Send email, UPS state has changed to ONBATT
(30 seconds timer elapses, UPS still ONBATT) => Send email - we're going down
SHUTDOWN (or maybe also FSD) => Send email - shutting down

Shutdown on LOWBATT:

Code:

ONLINE -> ONBATT => Send email, UPS state has changed to ONBATT
(30 seconds timer elapses, UPS still ONBATT) => No email sent, nothing new here
ONBATT -> LOWBATT => Send email, UPS state has changed to LOWBATT
(30 second timer elapses, UPS still LOWBATT) => Send email - we're going down
SHUTDOWN (or maybe also FSD) => Send email - shutting down

Arguably, the second "ONBATT" and "LOWBATT" email serves very little purpose in which case removing the entire email condition from the case statement might be a good idea, so that it becomes:

Code:

        "ONBATT"|"LOWBATT")
                if [ "${ups_shutdown}" = "lowbatt" && "$1" = "LOWBATT" ] || [ "${ups_shutdown}" = "batt" && "$1" = "ONBATT" ]; then
                        logger -t upssched-cmd "issuing shutdown"
                        /usr/local/sbin/upsmon -c fsd
                fi
                ;;

 

[William Grzybowski]

The purpose would be send two emails, one exactly when ONBATT or LOWBATT occurs and another when we the configured time elapses and shutdown is actually issued...
However, if you say its not necessary thats ok. I've committed your changes in r8357 -> http://support.freenas.org/browser/...es/usr/local/bin/custom-upssched-cmd?rev=8357

Please ty it out, if everything goes as you expected i'll merge it to 8.0.x branch

Thank you

 

[Milhouse]

However, if you say its not necessary thats ok. I've committed your changes in r8357 -> http://support.freenas.org/browser/f...d-cmd?rev=8357

Thanks.

Please ty it out, if everything goes as you expected i'll merge it to 8.0.x branch

There's a script error in the complex "if" statement for ONBATT|LOWBATT - /bin/sh is complaining about "[: missing ]". I'm not sure if /bin/sh supports an "if" condition of this complexity so I've split it out into two separate and simpler tests, while trying to minimise code duplication. The following now works:

Code:

case $1 in
        "ONBATT"|"LOWBATT")
                goingdown=0
                [ "${ups_shutdown}" = "batt" ] && [ "$1" = "ONBATT" ] && goingdown=1
                [ "${ups_shutdown}" = "lowbatt" ] && [ "$1" = "LOWBATT" ] && goingdown=1
                if [ ${goingdown} -eq 1 ]; then
                        echo logger -t upssched-cmd "issuing shutdown"
                        /usr/local/sbin/upsmon -c fsd
                fi
                ;;

Also, there's a typo in the SHUTDOWN|FSD case statement - my fault when I added it, sorry:

Code:

- "SHUTDOWN|FSD")
+ "SHUTDOWN"|"FSD")

I've tested the modified version, and it now works fine (at least, with my network hacks). I haven't tested a local UPS, but don't see any reason for it behaving differently.

 

[William Grzybowski]

Yeah, doing

Code:

[ "${ups_shutdown}" = "lowbatt" && "$1" = "LOWBATT" ]

is just wrong, -a should have been used:

Code:

[ "${ups_shutdown}" = "lowbatt" -a "$1" = "LOWBATT" ]

Fixed in r8366, thanks.

 

[vanhaakonnen]

Hi Milhouse and William Grzybowski,

Thanks for your work and the code. I will try this and post my results :)

But if it would be possible to make there changes directly at the gui it would be a great step in the right direction. Maybe the developers will add these function.. It would be a great benetit for freenas.