UPS shutdown ESXI 6.5

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
Hello to all,
If this is covered in one of the many threads on Freenas/ESXI apologies in advance.

I am running a new system on ESXI 6.5 FREE. I have not found a suitable solution for UPS shutdown for the sever itself. I currently have the UPS USB port passed through to the Freenas VM.

But would much prefer an alternative that would shutdown all the running VM's and then shutdown the server.
 

neb50

Dabbler
Joined
Aug 15, 2017
Messages
36
I setup a public/private key that enables the FreeNAS VM to send SSH commands to the ESXi machine and then send the shutdown command from FreeNAS to ESXi when the battery power gets low.

FreeNAS has the USB for the UPS passed thru to it from ESXi.

ESXi will need to have the AutoStartup/Shutdown properly configured to allow the VM's to properly shutdown.

I don't have access to my server now, but can get the proper syntax later.
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
I don't have access to my server now, but can get the proper syntax later.

Please if you could. Your post has me pointed in the right direction.

Thanks
 

neb50

Dabbler
Joined
Aug 15, 2017
Messages
36
Put this in the shutdown command on the UPS settings "ssh root@ipaddress /sbin/poweroff" where ipaddress is the ip address of the ESXi server.

You will need to get the public/private rsa keys setup first so that this will work. You can test this with a cron job or a console window to ensure that you have the keys setup correctly.

Also, you will need the AutoStartup setup correctly in ESXi so that it will start your VM's on boot up (set the start up order in the VM setup page and the rest is in the Host section.) The VM's will shutdown in the reverse order that they started in before the host is shutdown.
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
Thanks to neb50, this solution worked for me. When I created the ssh keys for freenas/esxi I was logged in as root on the Freenas VM. The ups shutdown command did not work until I added the root user to the ups additional users configuration setting.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
I have not found a suitable solution for UPS shutdown for the sever itself.
What UPS are you using?

Of course you would not have the UPS USB port passed through to FreeNAS, you would just use ESXi to shut down the VM's on your system.

But it sounds like you have things working for you so that is okay.

The only thing you should ensure is correct is the shutdown sequence for the VMs. This is based on dependancies of course, for example if you have a FreeNAS VM and it's hosting a iSCSI drive and you have a second VM which uses that iSCSI drive then you want to ensure you shutdown the second VM first and wait long enough for that to happen before shutting down the FreeNAS VM. My FreeNAS VM is the last VM to shutdown and the first VM to startup. This is the tricky part with virtualizing FreeNAS. Of course you may be all over this already but I just wanted to point it out.
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
Thanks for this Joe,

Presently all my VM's reside on a ESXI data store rather than on a Freenas pool. I plan to reconfigure this as time permits.
 

neb50

Dabbler
Joined
Aug 15, 2017
Messages
36
What UPS are you using?

Of course you would not have the UPS USB port passed through to FreeNAS, you would just use ESXi to shut down the VM's on your system.

But it sounds like you have things working for you so that is okay.

The only thing you should ensure is correct is the shutdown sequence for the VMs. This is based on dependancies of course, for example if you have a FreeNAS VM and it's hosting a iSCSI drive and you have a second VM which uses that iSCSI drive then you want to ensure you shutdown the second VM first and wait long enough for that to happen before shutting down the FreeNAS VM. My FreeNAS VM is the last VM to shutdown and the first VM to startup. This is the tricky part with virtualizing FreeNAS. Of course you may be all over this already but I just wanted to point it out.
How do you get the free ESXi 6.5 to perform the shutdown of the server with the UPS connected but not passed through to any of the VM's?

I tried to find solutions that would do that but all the information that I found stated that you needed to do it from within the VM's. If you have an alternative solution that would allow the UPS to be connected to the server by a USB cable that would allow proper shutdown, would you be able to provide that solution since that would be a much cleaner solution than the one that I came up with.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
How do you get the free ESXi 6.5 to perform the shutdown of the server with the UPS connected but not passed through to any of the VM's?

I tried to find solutions that would do that but all the information that I found stated that you needed to do it from within the VM's. If you have an alternative solution that would allow the UPS to be connected to the server by a USB cable that would allow proper shutdown, would you be able to provide that solution since that would be a much cleaner solution than the one that I came up with.
You should read my super long thread about my dream system. Do a google search for "joeschmuck my dream system" and it should pop up. Hopefully this is the proper link.
 

jp83

Dabbler
Joined
Mar 31, 2017
Messages
23
Trying to set this up myself. Spearfoot's guide "ssh root@esxihost poweroff" and neb50's "ssh root@ipaddress /sbin/poweroff" both seem to Power Off VMs as opposed to Shutdown Guest (using VM Tools for a safe shutdown). I was watching the tasks at the bottom. I also looked in the config in the esxi NUT client and it appears to use the poweroff command as well. Sounds like permissions issues prevent a wrapper script from working, so how are you safely shutting down ESXi VMs?
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Did you read my posting two posts up? I installed a third party application into ESXi which monitors the FreeNAS to send the signal that it's time to shutdown, ESXi sees it and then it shuts down the VMs and then itself.
 

jp83

Dabbler
Joined
Mar 31, 2017
Messages
23
Yes, that's what I was referring to the ESXi NUT client. I started setting it up and realized it uses the same shutdown command we were trying to do via ssh. I watched the web gui and it powers off all the VMs, not shutdown. I have the vmtools installed in all my VMs and the startup order set and working (poweroff in reverse). When ESXi powers off FreeNAS (AIO) I get an email notification from FreeNAS about unscheduled system reboot.

On my ESXi 6.7 (free) box:

more /opt/nut/etc/upsmon.conf.template
# Network UPS Tools: upsmon configuration
MINSUPPLIES 1
SHUTDOWNCMD "poweroff"
 

neb50

Dabbler
Joined
Aug 15, 2017
Messages
36
Have you ensured that the AutoStart setup is configured to Shutdown the VM's or is it set to Power off the VM's?

The default in ESXi is to set them to Power off, which is not a clean shutdown. If you change them to be Shutdown, then it should do a clean shutdown.

You can test all of this out without using the command by just shutting down the ESXi server from the GUI interface to see if it is working the way you need it to. It should shutdown cleanly from either the GUI or the remote command if it is all configured correctly.
 

jp83

Dabbler
Joined
Mar 31, 2017
Messages
23
Ah, that's it, thank you neb50. I've manually shutdown VMs related to NFS and didn't realize the other few local VMs were just getting powered down when I shut down the host. Now, I just need to figure out why my APC SUA1500 isn't power cycling. Oops, it just did like 10 min later (14 min after initiating shutdown). I add these aux parameters to ups.conf: offdelay=180 and ondelay=190. I thought that gives ESXi 3 min to shutdown everything, not sure why the UPS is waiting so long.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
I thought that gives ESXi 3 min to shutdown everything
3 whole minutes? I guess it depends on how many VMs you have running and how long it takes for them to shutdown. I'd be perfectly fine with the UPS taking 14 minutes to shutdown, that give the VMs and ESXi time to shutdown.

I really wish NUT was supported better in the Free version of ESXi, this would make things just so much simpler. Maybe I should upgrade to ESXi 6.7 and see what it has to offer.
 
Joined
Dec 29, 2014
Messages
1,135
I really wish NUT was supported better in the Free version of ESXi, this would make things just so much simpler. Maybe I should upgrade to ESXi 6.7 and see what it has to offer.

If it is an APC UPS, the Powerchute virtual appliance is free. That is what I use.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
If it is an APC UPS, the Powerchute virtual appliance is free. That is what I use.
Guess I will have to check that out. Yes, I do have an APC UPS, never tried the appliance. I'll have to read up on it.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Trying to set this up myself. Spearfoot's guide "ssh root@esxihost poweroff" and neb50's "ssh root@ipaddress /sbin/poweroff" both seem to Power Off VMs as opposed to Shutdown Guest (using VM Tools for a safe shutdown). I was watching the tasks at the bottom. I also looked in the config in the esxi NUT client and it appears to use the poweroff command as well. Sounds like permissions issues prevent a wrapper script from working, so how are you safely shutting down ESXi VMs?
The poweroff shuts down ESXi, which will gracefully shut down all virtual machines first; but you must configure ESXi to do this.

With ESXi v6.0 (which I use) you do this using the vSphere client's Configuration tab, using the Virtual Machine Startup/Shutdown section to configure what order to shut down the virtual machines. The key property to configure is 'Guest Shutdown' as the shutdown action; I believe the default is 'Power Off', which isn't very graceful at all.
 

dknm

Dabbler
Joined
Aug 27, 2018
Messages
25
I setup a public/private key that enables the FreeNAS VM to send SSH commands to the ESXi machine and then send the shutdown command from FreeNAS to ESXi when the battery power gets low.

I've watched this thread and thought of giving it a go with a CyberPower UPS. Tried to get the two to get to talk to one another but I must've done something wrong.

1. generated the keys in freenas
2. copied it to esxi 6.5 like so

Code:
cat esxi_key.pub | ssh root@esxi_address \ 'cat >> /etc/ssh/keys-root/authorized_keys'
Password: 

3. added 'ChallengeResponseAuthentication no' to sshd_config (on esxi side). That's the only change made.
4. restarted ssh service.
5. I'm getting a permission denied ssh-ing from freenas like so

Code:
root@foxtrot:/mnt/Data/dknmssh # ssh -v root@esxi_address
OpenSSH_7.5p1, OpenSSL 1.0.2k-freebsd  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 3: Deprecated option "useroaming"
debug1: Connecting to esxi_address [esxi_address] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: Fssh_key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5 FreeBSD-20170903
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5
debug1: match: OpenSSH_7.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to esxi_address:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:yuer14MB3VhkBtT8fhWEpwcYr6mmK2PFkVXO/A8Ba5s
debug1: skipped DNS lookup for numerical hostname
debug1: Host 'esxi_address' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey). 


Goal is to get ssh root@esxi_address /sbin/poweroff to work.

Any thoughts?
 
Top