Resource icon

Configuring UPS support for single or multiple FreeNAS servers

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Spearfoot submitted a new resource:

Configuring UPS support for single or multiple FreeNAS servers - Details on configuring the NUT UPS tools for use with FreeNAS servers connected to a UPS

If you are running a FreeNAS server you definitely need to connect it to a UPS.

Standalone Configuration

The official FreeNAS 9.10 UPS service documentation is available at doc.freenas.org and you should read it before proceeding.

Configuring a standalone FreeNAS server is fairly simple, provided your UPS is supported by the Network UPS Tools driver used by FreeNAS. You may consult networkupstools.org's...

Read more about this resource...
 

marksibert

Cadet
Joined
Jun 5, 2018
Messages
7
I agree - great resource! Is there any documentation on how to set up multiple UPSs, in the case where there is a server with redundant power supplies, each plugged into its own UPS?
 

jp83

Dabbler
Joined
Mar 31, 2017
Messages
23
Trying to set this up myself. Spearfoot's guide "ssh root@esxihost poweroff" and neb50's "ssh root@ipaddress /sbin/poweroff" both seem to Power Off VMs as opposed to Shutdown Guest (using VM Tools for a safe shutdown). I was watching the tasks at the bottom. I also looked in the config in the esxi NUT client and it appears to use the poweroff command as well. Sounds like permissions issues prevent a wrapper script from working, so how are you safely shutting down ESXi VMs?

Cross-post link: https://forums.freenas.org/index.php?threads/ups-shutdown-esxi-6-5.63211/#post-486371
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Trying to set this up myself. Spearfoot's guide "ssh root@esxihost poweroff" and neb50's "ssh root@ipaddress /sbin/poweroff" both seem to Power Off VMs as opposed to Shutdown Guest (using VM Tools for a safe shutdown). I was watching the tasks at the bottom. I also looked in the config in the esxi NUT client and it appears to use the poweroff command as well. Sounds like permissions issues prevent a wrapper script from working, so how are you safely shutting down ESXi VMs?

Cross-post link: https://forums.freenas.org/index.php?threads/ups-shutdown-esxi-6-5.63211/#post-486371
No, the 'poweroff' command shuts down the ESXi host, not the VMs. The latter are shutdown as part of the normal ESXi server shutdown sequence -- but only if you have configured it correctly! Otherwise (if memory servers) ESXi will simply power them down. So you must configure your ESXi server to send a graceful shutdown command to at least the FreeNAS VM, and any other VMs not homed on storage provided by the FreeNAS VM.

With this setup, as part of the ESXi host shutdown sequence, the FreeNAS VM itself receives a shutdown command from its ESXi host. This invokes your FreeNAS shutdown script, which should send shutdown commands to each VM it stores (either on NFS or iSCSI storage). These VMs should have the VMware tools installed.

See my resource Utility scripts for FreeNAS and VMware ESXi for scripts you can modify to use on your AIO (All-In-One) server to do all of this stuff.

It can be a little tough to wrap your mind around all of this, but it really does work very well. I've been using this setup for several years on multiple AIO servers.

Good luck!
 

NasKar

Guru
Joined
Jan 8, 2016
Messages
739
Thanks for the great guide. I'm not running ESXi so is the Shutdown Command
Code:
/sbin/shutdown -p now
ok for the Master and Slave?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thanks for the great guide. I'm not running ESXi so is the Shutdown Command
Code:
/sbin/shutdown -p now
ok for the Master and Slave?
Yes
 

Sundog

Cadet
Joined
Jun 10, 2020
Messages
2
Thanks for posting the guide it was very useful! I'm a little stuck on one part though. If I run the poweroff command as a test in Freenas shell, I'm still getting prompted for a password when I connect to my ESXI server. I don't think this should be happening because I setup SSH keypairs and an SSH connection under System in FreeNAS and I copied the public key to ESXI and pasted it into the authorized keys list. If I run the upsmon -fsd command it doesn't do anything other than stop the UPS service.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thanks for posting the guide it was very useful! I'm a little stuck on one part though. If I run the poweroff command as a test in Freenas shell, I'm still getting prompted for a password when I connect to my ESXI server. I don't think this should be happening because I setup SSH keypairs and an SSH connection under System in FreeNAS and I copied the public key to ESXI and pasted it into the authorized keys list. If I run the upsmon -fsd command it doesn't do anything other than stop the UPS service.
Thank you, @Sundog

I admit skipping over the details of configuring SSH keypairs, saying only "first enable the SSH service on the ESXi server and set up private key authentication", but in my defense there are probably hundreds of tutorials about this subject on the internet and I didn't want to get bogged down.

If you're being prompted for a password, something still isn't right with your VMware setup. VMware is a little different from everyone else, their SSH implementation looks in /etc/ssh/keys-root for the authorized_keys file. Also, you need to enable the ESXI shell service in addition to SSH; this service gives you access to CLI commands such as poweroff, etc.
 

Sundog

Cadet
Joined
Jun 10, 2020
Messages
2
Thanks for replying @Spearfoot

The strange part is that it's only an issue if I try and SSH from Freenas shell into the ESXi host. If I do it from Putty SSH client with it's own keypair then it works fine and I don't get prompted for a password. Both public keys are in the authorized_keys file each on a separate line. Maybe there's a syntax for separating them? I also recall before getting the Keypairs setup in Freenas I got prompted for whether I wanted to accept an unknown server key I think (something like that). I entered yes but I'm not sure if that could be the issue.

I currently have it sort of working by installing the NUT tools on the ESXi host, but that's not as clean as just doing it through SSH the way you did. The issue I've noticed with the poweroff command is that it doesn't seem to be gracefully powering off the VMs and isn't respecting the ESXi shutdown delay settings. I'll take another look at it today though and see if I can figure something out.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thanks for replying @Spearfoot

The strange part is that it's only an issue if I try and SSH from Freenas shell into the ESXi host. If I do it from Putty SSH client with it's own keypair then it works fine and I don't get prompted for a password. Both public keys are in the authorized_keys file each on a separate line. Maybe there's a syntax for separating them? I also recall before getting the Keypairs setup in Freenas I got prompted for whether I wanted to accept an unknown server key I think (something like that). I entered yes but I'm not sure if that could be the issue.

I currently have it sort of working by installing the NUT tools on the ESXi host, but that's not as clean as just doing it through SSH the way you did. The issue I've noticed with the poweroff command is that it doesn't seem to be gracefully powering off the VMs and isn't respecting the ESXi shutdown delay settings. I'll take another look at it today though and see if I can figure something out.
Yikes! I wouldn't recommend installing the NUT tools on your ESXi server.

I believe you will have success if configure your system according to these instructions. I'm using this setup on four FreeNAS-on-ESXi All-In-One systems here at my home, and I've been using this setup for ~5 years.

1> Until you can log on to your ESXi host via SSH using keys instead of a password, there's no hope of getting the FreeNAS UPS support to shut down your ESXi server. Did you double-check the key location I mentioned above? ESXi is a little different from other operating systems. In brief, your authorized_keys file for user root must be located here:

/etc/ssh/keys-root/

2> In addition to the SSH service (and it's key setup), you need to make sure the ESXi Shell service is running, too. See the screen shot below:
esxi-services.jpg


3> Lastly, the burden is on you to configure ESXi for graceful shutdown, it doesn't work that way out of the box. You'll have to install the VMware tools on all VMs and configure ESXi to shut them down instead of simply powering them off, see 'Stop Action' below:
esxi-startup-config.jpg
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Joined
Jan 4, 2014
Messages
1,644
Unfortunately, (unless I've missed something in the interim), while FreeNAS is the perfect platform for Plex, it doesn't support TV tuner cards. I run a second Plex server on a Windows 10 Home PC with a Hauppage TV tuner card just so I can have Plex manage recorded TV programs such as my favourite cooking shows.

I've been using WinNUT for several years now on that PC as a NUT slave. For such an old piece of software, it's as solid as a rock and is dead easy to set up.

@Spearfoot This extract from the FreeNAS User Guide UPS

screenshot.542.png


I believe the port needs to be included in this line that's in the Overview...

MONITOR ups@172.16.10.13 1 upsmon 'your_upsmon_password' slave

so it operates with a FreeNAS NUT master i.e.

MONITOR ups@172.16.10.13:3493 1 upsmon 'your_upsmon_password' slave

Furthermore, as WinNUT is a partial port of the client side of NUT for Windows, I vaguely recall that I wasn't successful in setting it up as a NUT master.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Unfortunately, (unless I've missed something in the interim), while FreeNAS is the perfect platform for Plex, it doesn't support TV tuner cards. I run a second Plex server on a Windows 10 Home PC with a Hauppage TV tuner card just so I can have Plex manage recorded TV programs such as my favourite cooking shows.

I've been using WinNUT for several years now on that PC as a NUT slave. For such an old piece of software, it's as solid as a rock and is dead easy to set up.

@Spearfoot This extract from the FreeNAS User Guide UPS

View attachment 41369

I believe the port needs to be included in this line that's in the Overview...

MONITOR ups@172.16.10.13 1 upsmon 'your_upsmon_password' slave

so it operates with a FreeNAS NUT master i.e.

MONITOR ups@172.16.10.13:3493 1 upsmon 'your_upsmon_password' slave

Furthermore, as WinNUT is a partial port of the client side of NUT for Windows, I vaguely recall that I wasn't successful in setting it up as a NUT master.
Interesting... Did you have to specify a port to get WinNUT to work? Because I just pasted in the MONITOR spec from my working WinNUT configuration file, which as you see, has no port specifier. Perhaps WinNUT defaults to port 3493 and you only need to give it explicitly when you use a different port?

My master NUT system contains only an IP address (no port) in the Auxillary parameters (upsd.conf) settings:
Code:
LISTEN 172.16.10.13
Does your master system include a port in the LISTEN setting?

I will gladly update the resource once we figure out how this part of the puzzle actually works. Thanks!
 
Joined
Jan 4, 2014
Messages
1,644
Did you have to specify a port to get WinNUT to work? Because I just pasted in the MONITOR spec from my working WinNUT configuration file, which as you see, has no port specifier. Perhaps WinNUT defaults to port 3493 and you only need to give it explicitly when you use a different port?
I think you're right. I've just always explicitly specified the port, but I just tried it without the port in the WinNUT upsmon.conf and it still works so this suggests that it must implicitly assume the master defaults to 3493.

Does your master system include a port in the LISTEN setting?
I have a FreeNAS server acting as NUT master.

screenshot.543.png


FreeNAS servers acting as NUT slaves do need to have the port specified.

screenshot.544.png
 
Last edited:

kekoakeakane

Dabbler
Joined
Apr 20, 2019
Messages
12
I was able to test NUT executing a shell script /sbin/doshutdown twice. I tried a couple different directories, but /sbin was the most consistent and survived reboots. I have 3 ESXI servers that all need to be gracefully shutdown before FREENAS since FREENAS provides the datastore. These scripts were modified from the versions found here.

The following script does a root ssh into each server and executes a script to shutdown the VMs and then ESXI. I do not run the process in the background (i.e. &) since I need the process to wait until all VMs are shutdown.

*****************************
#!/bin/sh
WALL=wall

# this script is triggered in case the power fails
# configured ssh-automatic-access with keys

echo "Shutting Down all VMs on SERVER01..." | ${WALL}
ssh root@10.1.10.1 "nohup /vmfs/volumes/datastore01/scripts/shutdown_esxi.sh"

echo "Shutting Down all VMs on SERVER02..." | ${WALL}
ssh root@10.1.10.2 "nohup /vmfs/volumes/datastore02/scripts/shutdown_esxi.sh"

echo "Shutting Down all VMs on SERVER03..." | ${WALL}
ssh root@10.1.10.3 "nohup /vmfs/volumes/datastore03/scripts/shutdown_esxi.sh"

echo "finished shutting ESXI down..." | ${WALL}
sleep 300 # Wait 5 minutes for ESXI Servers to Completely Shutdown

echo "Shutting Down BACKUP FREENAS Server..." | ${WALL}
ssh root@10.1.10.4 "nohup /sbin/shutdown -p now"

echo "Shutting Down FREENAS Server..." | ${WALL}

/sbin/shutdown -p now
**************************

The following script runs on each ESXI host to shut the VMs and Host down. The process on FREENAS will wait until this script completes on each ESXI host before moving on. This will shutdown each VM that has VMWare Tools installed. It does not rely on having VMs setup in the Auto Startup/Shutdown setting.

**********************************
#/bin/sh

##########################################################################################################################
#courtesy of http://www.c-note.dk/2011/12/04/wmware-esxi-suspend-all-guests/
#
#copy this file somewhere to datastore
#e.g. ‘/vmfs/volumes/myDataStore/scripts/’
#
# looks which vm's are running, sends them into Shutdown, waits until Shutdowhn-process is finished, then powers off esxi
# completely
##########################################################################################################################

VMS=`vim-cmd vmsvc/getallvms | grep -v Vmid | awk '{print $1}'`
for VM in $VMS ; do
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
if [ "$PWR" == "Powered on" ] ; then
name=`vim-cmd vmsvc/get.config $VM | grep -i "name =" | awk '{print $3}' | head -1 | cut -d "\"" -f2`
echo "Powered on: $name"
echo "Shutting Down: $name"
vim-cmd vmsvc/power.shutdown $VM > /dev/null &
fi
done

while true ; do
RUNNING=0
for VM in $VMS ; do
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
if [ "$PWR" == "Powered on" ] ; then
echo "Waiting..."
RUNNING=1
fi
done
if [ $RUNNING -eq 0 ] ; then
echo "Gone..."
break
fi
sleep 1
done

echo "Now we shutdown the host..."
/sbin/shutdown.sh && /sbin/poweroff
******************************************************
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I was able to test NUT executing a shell script /sbin/doshutdown twice. I tried a couple different directories, but /sbin was the most consistent and survived reboots. I have 3 ESXI servers that all need to be gracefully shutdown before FREENAS since FREENAS provides the datastore. These scripts were modified from the versions found here.

The following script does a root ssh into each server and executes a script to shutdown the VMs and then ESXI. I do not run the process in the background (i.e. &) since I need the process to wait until all VMs are shutdown.

*****************************
#!/bin/sh
WALL=wall

# this script is triggered in case the power fails
# configured ssh-automatic-access with keys

echo "Shutting Down all VMs on SERVER01..." | ${WALL}
ssh root@10.1.10.1 "nohup /vmfs/volumes/datastore01/scripts/shutdown_esxi.sh"

echo "Shutting Down all VMs on SERVER02..." | ${WALL}
ssh root@10.1.10.2 "nohup /vmfs/volumes/datastore02/scripts/shutdown_esxi.sh"

echo "Shutting Down all VMs on SERVER03..." | ${WALL}
ssh root@10.1.10.3 "nohup /vmfs/volumes/datastore03/scripts/shutdown_esxi.sh"

echo "finished shutting ESXI down..." | ${WALL}
sleep 300 # Wait 5 minutes for ESXI Servers to Completely Shutdown

echo "Shutting Down BACKUP FREENAS Server..." | ${WALL}
ssh root@10.1.10.4 "nohup /sbin/shutdown -p now"

echo "Shutting Down FREENAS Server..." | ${WALL}

/sbin/shutdown -p now
**************************

The following script runs on each ESXI host to shut the VMs and Host down. The process on FREENAS will wait until this script completes on each ESXI host before moving on. This will shutdown each VM that has VMWare Tools installed. It does not rely on having VMs setup in the Auto Startup/Shutdown setting.

**********************************
#/bin/sh

##########################################################################################################################
#courtesy of http://www.c-note.dk/2011/12/04/wmware-esxi-suspend-all-guests/
#
#copy this file somewhere to datastore
#e.g. ‘/vmfs/volumes/myDataStore/scripts/’
#
# looks which vm's are running, sends them into Shutdown, waits until Shutdowhn-process is finished, then powers off esxi
# completely
##########################################################################################################################

VMS=`vim-cmd vmsvc/getallvms | grep -v Vmid | awk '{print $1}'`
for VM in $VMS ; do
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
if [ "$PWR" == "Powered on" ] ; then
name=`vim-cmd vmsvc/get.config $VM | grep -i "name =" | awk '{print $3}' | head -1 | cut -d "\"" -f2`
echo "Powered on: $name"
echo "Shutting Down: $name"
vim-cmd vmsvc/power.shutdown $VM > /dev/null &
fi
done

while true ; do
RUNNING=0
for VM in $VMS ; do
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
if [ "$PWR" == "Powered on" ] ; then
echo "Waiting..."
RUNNING=1
fi
done
if [ $RUNNING -eq 0 ] ; then
echo "Gone..."
break
fi
sleep 1
done

echo "Now we shutdown the host..."
/sbin/shutdown.sh && /sbin/poweroff
******************************************************
Looks like a lot of work! I just send a poweroff command to the ESXi host, like this: ssh root@esxihost poweroff

I configure the ESXi host Autostart settings like this:
esxi-autostart.jpg


...and install the VMware tools on all virtual machines. The ESXi host gracefully shuts down all of its guests before shutting itself down. Simple, but effective.
 

kekoakeakane

Dabbler
Joined
Apr 20, 2019
Messages
12
My situation is complicated by the fact that I cannot shutdown the FREENAS Server until I'm sure that all the VMs have shutdown. Note: My FREENAS server is a separate machine from my ESXI servers. I need to ensure that all the data from those VMs are flushed to the FREENAS iSCSI datastore before I shut down the FREENAS server. I have a couple Windows Domain Controllers on the datastore and they complain bitterly, if not break, when they are shutdown improperly. That would happen if they could not flush their DBs to the disk during shutdown. The poweroff command returns an exit code immediately after it's issued and would not give any indication to the FREENAS server that it should not shut down yet. The use of the script with the loop ensures that no exit code is returned until all the VMs have shutdown. I thought about using NUT on each of the ESXI servers, but since it's a manual install, that needs to be removed prior to upgrading, it was more work that could get messed up on the next upgrade. This script method has survived several ESXI upgrades. Also, I have discovered that migrating VMs will drop them out of the Autostart setting.
 
Joined
Jan 4, 2014
Messages
1,644
During a recent power outage, I noticed a pair of backup CORE servers dropped like a lead balloon rather than shutting down gracefully when the UPS they were connected to (in a master-slave arrangement) reached a low battery condition. I realised then that the UPS batteries were no longer holding their charge. I replaced the batteries this w/e and all is well again.

The question I have is 'Short of periodically testing the UPS, is there a way in software to determine if UPS batteries are nearing EoL and should be replaced?'. This is the output for the upsc command on the PowerShield Defender1200VA UPS the CORE servers are connected to.

Code:
root@truenas-b2:~ # upsc ups
battery.charge: 100
battery.voltage: 26.10
battery.voltage.high: 26.00
battery.voltage.low: 20.80
battery.voltage.nominal: 24.0
device.type: ups
driver.name: blazer_usb
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.internal: 0.12
input.current.nominal: 5.0
input.frequency: 50.1
input.frequency.nominal: 50
input.voltage: 233.6
input.voltage.fault: 233.6
input.voltage.nominal: 240
output.voltage: 233.6
ups.beeper.status: enabled
ups.delay.shutdown: 30
ups.delay.start: 180
ups.load: 17
ups.productid: 5161
ups.status: OL
ups.type: offline / line interactive
ups.vendorid: 0665
 
Last edited:
Top