UPS and Restarting the server too early

[Rosin0416]

Hello,
I did a test with my UPS, but the result is not quite what I expected
Here are my settings :

• Shutdown mode : UPS goes on battery
• Shutdown timer : 30s
• Shutdown command : shutdown -r now
• "Power off UPS" checkbox : enabled (checked)

Behavior:
- Disconnection of the UPS power plug from the mains. The UPS immediately switches to battery power.
- After about 40 seconds, the system starts its shutdown protocol
- the server has shut down. Then it started to reboot.
- the UPS shut down, which caused the server to shut down abruptly.
- reconnecting the UPS power plug
- the UPS automatically turned on again
- the server automatically restarted

What bothers me is that the server started to reboot when the UPS had not yet shut down. And when the UPS shut down, it shut everything down abruptly.


What I was looking for was for the UPS to shut down after the server and when the power came back on, the UPS and then the server would restart.
The problem is that the "shutdown -r now" command causes an immediate reboot.
Is it not possible that the system waits for a while before rebooting, to give the UPS time to shut down?

Regards

 

[ChrisRJ]

You are using the wrong parameter for the shutdown command. The -r explicitly tells the machine to reboot. You should change to shutdown -h now. More on the man page.

 

[Rosin0416]

Hello, and thanks for your feedback.
Yes, I used the -r parameter to make the system reboot.
The default command that is sent to the system is unless I am mistaken is "/sbin/shutdown -p now" (seen here: /etc/local/nut/upsmon.conf)

I made several tests :

TEST 1 :

• Shutdown mode : UPS goes on battery
• Shutdown timer : 30s
• Shutdown command : shutdown -p now
• "Power off UPS" checkbox : disabled (unchecked)

Result:
The system shut down and immediately restarted (I had not yet reconnected the power)


TEST 2 :

• Shutdown mode : UPS goes on battery
• Shutdown timer : 30s
• Shutdown command : shutdown -h now
• "Power off UPS" checkbox : enabled (checked)

Result:
The system shut down and immediately restarted (I had not yet reconnected the power)
The UPS shut down, which caused the server to shut down abruptly.
When I reconnected the power, the server also automatically restarted
=> same behavior as my initial post


TEST 3 :

• Shutdown mode : UPS goes on battery
• Shutdown timer : 30s
• Shutdown command : shutdown -h now
• "Power off UPS" checkbox : disabled (unchecked)

Result :
Part of the system has stopped. But in this configuration, the fans were still running, which was not the case in the previous tests.
When I reconnected the power to the UPS, the system froze (see picture). Contrary to what is written, pressing a key did not cause the system to restart. I had to manually shut down the server and turn it back on.

By chance, I tried with the "/sbin/shutdown -h now" command, but I got the same result.

 

[ChrisRJ]

What I usually do is use a combination of BIOS/hardware functionality and software. In the BIOS I configure the motherboard to boot up, when power is restored. Together that has always worked well for me.

 

[Rosin0416]

I'll go look in the bios.

1] What command do you use?
nothing (default command)
or
shutdown -h now ?

2] Are you asking for UPS shutdown after system shutdown?

 

[WI_Hedgehog]

I'll go look in the bios.

1] What command do you use?
nothing (default command)
or
shutdown -h now ?

2] Are you asking for UPS shutdown after system shutdown?

0.) The UPS delay must be set for a minimum time delay, otherwise a temporary power outage will cause the following chain of events. See the note at the end to help analyze if this is the correct procedure.

1.) shutdown -P +10 "The server has lost power and will shut down. Save your work immediately and shut down your workstation."
-P: Requests the machine to be powered off once it is brought down. (-h leaves it up to the machine whether it powers down or just halts, in this case powering down is appropriate)
+10: 10 minutes from now. (gives the clients a chance to save their work, or for AutoSave (if enabled) to save their work; this may need to be 5 minutes, depending on many factors including backups that could be running)

2.) In BIOS set the server to automatically power itself on when power is restored (currently Disabled).


3.) Set the time delay to 30 seconds or a minute so the UPS can complete the self-diagnostic (the amount of time required depends on the UPS, (currently No Delay):



NOTE: The chain of shutdown events on the server could be set in motion (shutdown after a set amount of time),
during which time power could be restored, and the UPS not shut down.
This would cause the server to shut down without coming back up as the UPS would have not shut down and come back up (no "power restored" trigger event happens on the server).

In this case use -r instead of -P to reboot the server, in combination with a server BIOS delay of typically 2 minutes to allow the UPS to detect a no-load condition, cool itself appropriately, and shut down. This requires the BIOS has a Power On Delay option.

If the UPS shuts down and later restores power the server will boot.

If the UPS does not shut down the server will reboot.

 

[Patrick M. Hausen]

@WI_Hedgehog TrueNAS 13 is FreeBSD based, not Linux. That command won't work.

@Rosin0416 I don't know from the top of my head but the idea is for the system to shutdown to a halt (-h) or poweroff (-p) and the UPS to cut power a couple of seconds later. Then when line power is restored and the battery sufficiently charged, the UPS should restore power to the server. The server should boot on power loss&restore - most common setting in the BIOS: "always on".

 

[WI_Hedgehog]

@WI_Hedgehog TrueNAS 13 is FreeBSD based, not Linux. That command won't work.

@Rosin0416 I don't know from the top of my head but the idea is for the system to shutdown to a halt (-h) or poweroff (-p) and the UPS to cut power a couple of seconds later. Then when line power is restored and the battery sufficiently charged, the UPS should restore power to the server. The server should boot on power loss&restore - most common setting in the BIOS: "always on".

You're correct, the OP isn't using SCALE; therefore one would use -p (lower case).

FreeBSD shutdown manual page.

 

[ChrisRJ]

0.) The UPS delay must be set for a minimum time delay, otherwise a temporary power outage will cause the following chain of events. See the note at the end to help analyze if this is the correct procedure.

If I am not mistaken this is not relevant, if you configure the shutdown to be initiated only when the UPS is low on battery.

 

[Rosin0416]

Hello,

@Patrick M. Hausen​

@Rosin0416 I don't know from the top of my head but the idea is for the system to shutdown to a halt (-h) or poweroff (-p) and the UPS to cut power a couple of seconds later. Then when line power is restored and the battery sufficiently charged, the UPS should restore power to the server. The server should boot on power loss&restore - most common setting in the BIOS: "always on".

Yes, that's exactly it.
I just want that after for example 10mn (shutdown timer), the server shuts down followed by the UPS. When the power comes back on, the UPS turns on again (which it does), then the server turns on again.

What is surprising in my tests is that the command "shutdown -p now" caused the server to restart, when it should not have? On the other hand, if no command is indicated in the "Shutdown command" field, the default command is applied. The default command is "/sbin/shutdown -p now". In my test, it causes a real shutdown of the server.
This is curious.

 

[WI_Hedgehog]

If I am not mistaken this is not relevant, if you configure the shutdown to be initiated only when the UPS is low on battery.

Leaving the server on for an indeterminate amount of time is an inconsistent behavior, which may be perfectly acceptable in many environments.

In a production environment it may be the case that the server should remain up for a set amount of time in order to:

1. Allow other systems to shut down cleanly.
2. Not overly discharge the UPS batteries.

after which remaining "up" provides no useful function.

In some instances--such as laboratories running experiments--data logging is critical and the computers should be "always on." In the event a stand-by generator fails to bring backup power on-line it is often critical the computer systems be run for as long as possible with the hope standby power will be restored.

UPS batteries are typically sealed lead-acid Absorbed Glass Mat batteries with an 80% depth-of-discharge, although ideally should not be taken below 50% DOD

1. for longevity. UPS are "typically" sized to provide 15 minutes of standby power, enough time to withstand a temporary power outage (5 minutes) and enough reserve to shut down all systems cleanly (10 minutes).
2. and "second outage" purposes where power is restored temporarily and then fails again within a short amount of time.

How long the systems remain up is generally part of a Managed Outage Plan or Disaster Recovery Plan or whatever applicable plan is in place. As such, Peak Load Events tend to be a main focus as System-Wide Shutdown causes a lot of events to happen in a compressed timeframe. UPS are typically rated at a percentage of full-load to manage the event of UPS failure--which does happen given the heavy load placed on the UPS circuitry (often rated at double the capacity of maximum peak load).

At home that "plan" may relate to "on as long as possible." I personally don't see that as a good strategy, but maybe it is.

----
Power consumption calculator
UPS runtime formula

 

[Rosin0416]

UPS batteries are typically sealed lead-acid Absorbed Glass Mat batteries with an 80% depth-of-discharge, although ideally should not be taken below 50% DOD

Thanks for the 50% threshold.
In Truenas, the default "battery.charge.low" threshold, so the DOD (Depth-Of-Discharge) was set to 30%. I wanted to raise this threshold, but I didn't know where to put it.
So thank you, this gives me an idea.

To do this, Web Interface Truenas > Services > UPS > edit > auxiliary parameters > add lines

ignorelb
override.battery.charge.low = 50

We can also add this parameter :

override.battery.runtime.low = to be determined (in second)

By the way, in my many tests with this paramater, I found one thing positively amazing :

• Shutdown mode : UPS goes on battery
• Shutdown timer : 1000s
• Shutdown command : default
• "Power off UPS" checkbox : disabled (unchecked)
• auxiliary parameter :
  • override.battery.charge.low = 97

Normally, the server was configured to start the wait timer from the battery switchover and should wait about 16mn (1000s) before starting its shutdown procedure.
Except that as I raised the low battery threshold to 97% (it was just for the test), the server reached the low battery threshold before reaching the end of the waiting timer (16mn).
The server started its shutdown when it went from 97% to 96%.

So finally, even if the server is not configured on the "UPS goes on battery" shutdown mode, it still takes into account a shutdown if this threshold is reached.
Which is a very good thing, I think.

 

[Rosin0416]

Hello
I managed to get the behavior of the server+UPS couple I was looking for.
In my configuration, I ask the UPS to shut down after the server (I checked the case provided for this in the UPS service configuration in the web interface.)
Having performed a test, I find that the inverter cuts the power supply a few seconds (about 5s) after what seems to be the shutdown of the server. I would like to increase this time. But I did not find in the doc the parameter. Is there such a parameter? If so, which one?
Regards
En savoir plus
https://www.deepl.com › tran

 

[Patrick M. Hausen]

Only someone knowing your particular UPS model better than the manual could ever know. If the UPS implementation is doing the right thing, it cuts power after a significant drop in the current drawn from the server socket, not after a timeout. I'd check with the manufacturer. Open a support ticket or just send an email to their general technical contact.

 

[Rosin0416]

Hello,
Sorry, that's not what I'm talking about here.

I checked the "Power off ups" box.
Here are the phases that occurred:

1. When I unplugged the power supply from the UPS, the UPS switched to battery power.
2. Then after about 30 seconds, the server started its shutdown procedure.
3. Once the server appeared to have shut down, the UPS shut down as well.
4. When I plugged the power back in, the UPS automatically restarted and then so did the server.

My question is about phase 3.
I would like to increase the time (timer) before the UPS turns itself off.
I don't know how to get out the list of configurable parameters on the inverter.

My UPS is an old MGE pulsar evolution 800 (MGE has been bought by Eaton)
When I connect it to a PC, I can use the software "Personnal Solution PAC" which allows me to configure the UPS.
I would like to have access to these settings on Truenas.

When we configure the UPS to shut down after the server, does the UPS have the information that the server is shut down?
If not, what is the trigger?

Regards

 

[Patrick M. Hausen]

And I meant "ask the manufacturer of the UPS". Nobody else will be able to tell. If the UPS does a reasonable thing it will shut down after power consumption by the server drops to zero. Whether it does that or not, only the manufacturer ... if it is not explicitly stated in the manual.

Or someone who happens to own the exact same model of UPS, is an electrical and software engineer, reverse-engineered the whole thing ... etc. Nobody here on the forum can tell you what "piece of hardware X" actually does.

 

[WI_Hedgehog]

Often a UPS will have a serial or USB connection to a computer (server), and if the link drops the UPS knows the system (computer/server) is down. It could be the case the installed UPS software on the computer/server sends a note to the UPS it is in the process of shutting down and the UPS should then shut down.

UPS controller software is not uncommon and can set on-chip UPS configurations/behaviors. Although a timeout as requested is possible, it's uncommon. Usually the UPS is set to shut down immediately after server shutdown.

What triggers that chain of shutdown events is a question, either the local UPS, attached machine, or remote UPS manager can govern the task depending on the level of UPS service provided.