How to verify proper shutdown of Freenas VM on ESXi host?

[Jopi]

I'm running Freenas 11.1-U5 on a ESXi 6.7 host and decided to install APC Back-UPS Pro 900 VA UPS to protect it from power outages.

I've managed to get NUT up and running on my Freenas VM and it sends "poweroff" command to ESXi via SSH when ever UPS switches to battery power to shutdown all the VMs gracefully and then powering off the host itself.

BUT whenever I do a trial run of the UPS shutdown sequence I get unauthorized system reboot notifications via email telling me that the system was not shut down properly?

If I use ESXi web thingy to tell Freenas VM to shutdown I get "system going down" messages in open SSH sessions etc, but when the poweroff sequence is initiated by NUT I get no messages.

How can I verify the proper shutdown of Freenas VM?

 

[neb50]

Put this in the shutdown command on the UPS settings "ssh root@ipaddress /sbin/poweroff" where ipaddress is the IP address of the ESXi server.
You will need to get the public/private rsa keys setup first so that this will work. You can test this with a cron job or a console window to ensure that you have the keys setup correctly.

Also, you will need the AutoStartup setup correctly in ESXi so that it will start your VM's on boot up (set the start up order in the VM setup page and the rest is in the Host section.) The VM's will shutdown in the reverse order that they started in before the host is shutdown.

See this thread for more information
https://forums.freenas.org/index.php?threads/ups-shutdown-esxi-6-5.63211/#post-452312

 

[Jopi]

Thanks, but I've already managed to get the ssh/poweroff part working and was looking for a way to verify the Freenas shuts down correctly?

Is there any logs in Freenas (or in ESXi) that could shed some light on wether the shutdown was graceful or not? I'm spooked by the "system was not shut down properly" warnings I keep getting in email.

 

[neb50]

If you are getting the not shutdown properly message, then it is probably just turning the VM off without a controlled shutdown.

Did you see/verify this part?
Also, you will need the AutoStartup setup correctly in ESXi so that it will start your VM's on boot up (set the start up order in the VM setup page and the rest is in the Host section.) The VM's will shutdown in the reverse order that they started in before the host is shutdown.

The AutoStart is what ESXi uses to Start and Stop the VM's on startup and shutdown. You should be able to verify the startup is working correctly first and then can use the VM desktop display in ESXi to verify that the VM's are shutting down correctly when given the Shutdown command. You can also watch the output log in FreeNAS and see a message about getting a shutdown command when it is run. This will work with the shutdown command from within ESXi or from the remote command that you are sending from FreeNAS.

Also, in the AutoStartup setup I believe the default is PowerOff and it needs to be changed to Shutdown to do a controlled shutdown in the VM's.

The last thing that may be causing you issues is if the VM does not have the VMWare tools installed/recognized within ESXi. This is what ESXi uses to issue the controlled shutdown command to the VM.

 

[Jopi]

Thanks for your help! I've got it working! :)

You were right, the autostart settings were the problem. The default Stop action in Host -> Manage -> System -> Autostart was "Power off" so I changed it to "Shut down" and now everything seems to be working correctly.

I also set the Autostart order so that my syslog server (Ubuntu) goes up first and down last so it can gather all the logs from other VMs.

 

[usergiven]

After 11.2 upgrade I've apparently broken my UPS monitoring on FreeNAS that shuts ESXI down gracefully via SSH. Anyone having similar issues?

Code:

undefinedDec 29 08:38:51 jeep upsmon[12286]: UPS ups on battery
undefinedDec 29 08:39:07 jeep upssched-cmd: issuing shutdown
Dec 29 08:39:07 jeep upsmon[12286]: Executing automatic power-fail shutdown
Dec 29 08:39:07 jeep upsmon[12286]: Auto logout and shutdown proceeding
undefinedDec 29 08:39:28 jeep upsmon[12285]: parent: Unable to call shutdown command: ssh -i id_esxi root@192.168.1.150 '/sbin/poweroff'

I get proper notifications of onbattery and shutdown events. I originally configured and setup SSH Keys (id_esxi) for root and another user, applying them to my UPS configuration. I can also ssh into FreeNAS under one of those users and execute the ESXI shutdown successfully. For some reason I don't know why the UPS service doesn't have the right to call the shutdown command. Any help would be greatly appreciated!

*Edit* I just deleted the ESXi authorized_keys file and generated a new set on freenas, sent them back to ESXi and all works well now.