Redundant UPS?

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
Hi,

I want to add redundant UPS to my FreeNAS server. Actually the UPS is already there, but of course I also want to shutdown FreeNAS (and other connected servers) when battery is low.

Overview:
redundant-ups.png


There are two UPS, each connected to it's own RCD. The UPS are powering several servers, each with redundant PSU. So maybe server X runs FreeNAS, server Y runs some Linux and server Z runs whatever else.
My idea was adding SNMP cards to both UPS and let all servers query battery state over network. If battery runs low, all servers should shut down. Will this work or am I missing something?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Will this work or am I missing something?
Which battery? if you have low battery on UPS 1 and full battery on UPS 2, do you want your servers shutting down?

You may find you can use USB from each of the UPS units to a server and then use NUT to publish the UPS state to the network from the connected server(s). I use that method with success, but don't connect 2 UPS to the same server.
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
The perfect solution of course would be to shutdown only if both batteries are low. But that's not possible with the settings in FreeNAS web interface. It may be possible with running NUT in a jail or VM though. Any info about this would be appreciated.
EDIT: NUT can handle this: https://networkupstools.org/features.html "upsmon won’t initiate a shutdown until the total power situation across all source UPSes becomes critical (on battery and low battery). "

But also watching UPS 1 only would not be a huge problem:
* In the event when the whole site has a power outage (UPS 1 & 2 run on battery), the battery drain on both UPS will be roughly the same.
* In the case of mainteance or failure on UPS 1 of course it would be nice to have unlimited time while running on UPS 2.

How do others take care of this? Redundant PSUs are not that unlikely in a FreeNAS.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
It may be possible with running NUT in a jail or VM though. Any info about this would be appreciated.
I run it in a jail.

I think it was as simple as pkg install nut or something like that, then just editing the config files. Happy to provide a bit more help if needed.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
I run it in a jail.

I think it was as simple as pkg install nut or something like that, then just editing the config files. Happy to provide a bit more help if needed.
Can nut actually monitor a USB device and shutdown the server from inside a jail?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Can nut actually monitor a USB device and shutdown the server from inside a jail?
I'm actually not using it that way, rather with USB on other boxes and NUT querying those.

The theory goes that with the right devfs_ruleset, you can pass the USB through to a jail, so I would be doing that if I had the need.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
and shutdown the server from inside a jail?
Just understood that part of the question... actually what you do is have the FreeNAS host configured as a slave to look at NUT on the Jail IP and NUT port, then it's FreeNAS (UPS service) that does the shutting down on its own.
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
Just understood that part of the question... actually what you do is have the FreeNAS host configured as a slave to look at NUT on the Jail IP and NUT port, then it's FreeNAS (UPS service) that does the shutting down on its own.
I can set up a VM that monitors both UPS and using upsmon inside the VM it can shutdown itself when both UPS are on low battery. But how can I do this in the FreeNAS web interface? There I can still just monitor a single UPS as slave.

Seems the only way to do it is if the VM issues the shutdown command to the FreeNAS host via ssh.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

It may take a little doing with some scripts on your NUT server to set the condition that you want for the dummy based on the overall status, but FreeNAS would look only at the dummy.
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
Well, next question: If i chose "Slave" in FreeNAS:
2020-04-02 16_50_28-FreeNAS - 10.0.60.60.png


"Remote Host" and "Remot Port" is connection to the NUT master server.
But what about "Driver" and "Port"? My impression was that I should just enter the UPS name on the NUT server?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
It's s strange thing... on a server where I have no UPS config, if I enter the config page and select slave, driver disappears.

On another server that has been configured before as master, when changing to slave, the driver option remains.

I'm not exactly sure what to do about that... It seems to work as expected on the new config.
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
So you just get: Identifier (name of the UPS on the NUT server), Remote Host (NUT Server) and Remote Port? This is what I would have expected, too.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Looks like there was a fix for the "not disappearing driver field" which was delivered in 11.3-U2 yesterday. Perhaps it's a good time to try it out.
 

Dieter Toews

Dabbler
Joined
Jul 21, 2016
Messages
21
Hi all,

did anybody ever get a setup working with the jail approach? i have dual ups's with network card and I tried something a few years ago by directly the nut config files (it didn't work due, i think, to the complicated way the gui reads and writes config files in freenas...)

anyway - having never setup a jail before I'm a little gun shy.

 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
Actually I think running NUT in a jail is not the best idea, at least if you want to shutdown the UPS at the end. From https://networkupstools.org/docs/user-manual.chunked/ar01s06.html :
Shutdown design
When your UPS batteries get low, the operating system needs to be brought down cleanly. Also, the UPS load should be turned off so that all devices that are attached to it are forcibly rebooted.

Here are the steps that occur when a critical power event happens:

  1. The UPS goes on battery
  2. The UPS reaches low battery (a "critical" UPS), that is to say upsc displays:
    ups.status: OB LB
    The exact behavior depends on the specific device, and is related to:
    • battery.charge and battery.charge.low
    • battery.runtime and battery.runtime.low
  3. The upsmon master notices and sets "FSD" - the "forced shutdown" flag to tell all slave systems that it will soon power down the load.
    (If you have no slaves, skip to step 6)
  4. upsmon slave systems see "FSD" and:
    • generate a NOTIFY_SHUTDOWN event
    • wait FINALDELAY seconds - typically 5
    • call their SHUTDOWNCMD
    • disconnect from upsd
  5. The upsmon master system waits up to HOSTSYNC seconds (typically 15) for the slaves to disconnect from upsd. If any are connected after this time, upsmon stops waiting and proceeds with the shutdown process.
  6. The upsmon master:
    • generates a NOTIFY_SHUTDOWN event
    • waits FINALDELAY seconds - typically 5
    • creates the POWERDOWNFLAG file - usually /etc/killpower
    • calls the SHUTDOWNCMD
  7. On most systems, init takes over, kills your processes, syncs and unmounts some filesystems, and remounts some read-only.
  8. init then runs your shutdown script. This checks for the POWERDOWNFLAG, finds it, and tells the UPS driver(s) to power off the load.
  9. The system loses power.
  10. Time passes. The power returns, and the UPS switches back on.
  11. All systems reboot and go back to work.

If you run NUT (the upsmon master) in a jail the steps 5 and upwards cannot happen. Because the jail will already be shutdown in step 4 with the TrueNAS host.
It seems a better approach is to run NUT on a dedicated hardware. A Intel NUC or a Raspberry Pi will be fine.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
if you want to shutdown the UPS at the end
I have always viewed that step as undesirable.

If you can hold out even one second longer and the mains power comes back, you saved yourself an outage.

With a UPS shutdown, you're effectively doing an ungraceful power-off on the devices that couldn't be told to shut down gracefully anyway, so there's no additional damage (unless you want to argue about a fully depleted battery and that kind of thing... but for me, that battery only exists in order to prevent outage, so I'll use it to it's last watt).
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
But then how do you automatically start your servers again that have already gone to shutdown? Usually the servers automatically start again as soon as power resumes to their power supplies. But therefore it has to be switched off before.

You may use WakeOnLAN instead, but therefore you have to make shure that the servers really power off on shutdown and that they support WOL. Or maybe you can use IPMI to start the servers again.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Usually the servers automatically start again as soon as power resumes to their power supplies. But therefore it has to be switched off before.
I don't see how your scenario is different... when the power comes back, power will flow to the UPS attached devices again no matter which way the UPS reached the "off" state.
 

Exhorder

Explorer
Joined
Jul 12, 2019
Messages
66
If servers are shut down (and powered off) the load on the UPS is zero. So the UPS won't turn off soon. So there may be a large time frame where the UPS still is running. If power comes back within this time frame, the servers won't start automatically again.
 
Top