How do I set up FreeNAS to send a shutdown signal to my VMs, before shutting down

[Zaaphod]

In the event of a power failure, of which there are many in my area... I have a battery backup and can initiate a safe shutdown of FreeNAS, but how can I get FreeNAS to send a proper shutdown signal to the VM before it attempts to shutdown? This is going into a business so there will be nobody there to know there even was a power failure outside business hours to do any kind of manual shutdown procedure. It would be nice for a solution that would always send a shutdown signal when FreeNAS was trying to shut down for any reason, not just power failures, however power failures are my main concern.

 

[Allan Wilmath]

Configure SSH with keys in all of the VMs and use a shell script that is put in to the Tasks>init/shutdown section of FreeNAS.

 

[wblock]

Not all VMs are even SSH-capable. @Zaaphod, please enter a bug to get this feature added (sending a hardware shutdown signal to all running VMs).

 

[Zaaphod]

Also Enabling SSH is not allowed by some system administrators due to security risks.

Report Entered Here

 

[Zaaphod]

I came up with a way to shut down my VMs with a script that does not rely on ssh to work on the vm. I plan to expand this to also warn / shutdown cliet computers of an impending server shutdown, so their files have a chance to be properly commited to the server before the shutdown. My scripts work fine if I run them from a puTTY terminal and everything happens as I expect it to, but I'm having a heck of a time getting them to run either under init/shutdown scripts or under UPS service. I'm not sure if I'm not getting the syntax correct to call the script, or if there is a permission issue... it's just not working. The one I need to work the most is from a UPS event triggered shutdown, because an admin can easily run the scipt manually before shutting down the server for scheduled reasons.


Below are my scripts - there are 3 of them, the first one is the one I call to initate the shutdown procedure. It checks to see if the client computer is actually running or not with a ping, if it's not running, it simply creates a specific file in a specific location... then it waits for the file to be deleted. Next I have a batch file running on the client computer that just sits around all day checking to see if the specific file exists, then a delay, I start of with testing for the file once a second. If the file exists, I start a 30 second countdown, then I force my monitor batch file to restore from minimize and give the user a chance to extend the countdown to 5 minutes. After the timeout, a shutdown is initiated (I could also launch other commands to shut down other specific processes beforehand) and then the specific file is deleted. So now I have a proper shutdown in progress, but just because I gave the shutdown command, doesn't mean it's all the way down, so the initial script upon detection that the specific file no longer exists, now starts checking the IP address of the client computer up to a maximum of 15 minutes, if it hasn't stopped responding by then, it's determined that something happened to the shutdown, but the server can't wait for a proper shutdown any longer as the clock is ticking on the battery life. Once the IP address is no longer present (or the timeout) there is an addional 30 second delay just to make sure it's down. After this, the script exits.
The third script just launches the first one, waits for it to complete, then issues the shutdown command to the server. I did it this way because I can use the third script to launch the first script on multiple machines, wait for them all to complete, then continue.

I have tried putting ShutDownAll.sh in UPS under shutdown command with full path like this:
/mnt/Kirkland_Audio_4TB/4TB/User Datasets/QB_Server/ShutDownAll.sh
I have tried putting bash in front like this:
bash /mnt/Kirkland_Audio_4TB/4TB/User Datasets/QB_Server/ShutDownAll.sh
I've tried various locations such as /root or /etc
I always get an error "unable to call shutdown command"

so I thought I would let the UPS service just do it's normal shutdown and put my script under init/shutdown tasks and again it just never seems to run my script. I even did a test where I just write a file first thing... with echo test > test but the file never gets created so I know my script is not running. I seem to be missing some important detail on how to launch my script from either of these locations.

Any Help getting this to run is greatly appreciated!

below is the first script that initiates the shutdown
SendShutDown.sh

Code:

#!/usr/local/bin/bash
cd /mnt/Kirkland_Audio_4TB/4TB/User Datasets/QB-Server
RemoteSystem="192.168.0.234"
WaitReceive="118"
WaitShutdown="900"
WaitExtra="30"
echo $Thing
ping -c 1 -o -q $RemoteSystem
if [ $? -eq 0 ]; then
   echo Shutdown >> Shutdown
   SECONDS=0
   sleep 1
   while [ -f Shutdown ] && [ "$SECONDS" -le "$WaitReceive" ]; do
	  clear
	  printf "$(hostname)  "
	  date
	  printf "Waiting %01d:%02d for $RemoteSystem to be Receive Shutdown Request\n" "$(($WaitReceive / 60))" "$(($WaitReceive % 60))"
	  TimeWaiting=$SECONDS
	  printf "%01d:%02d Elapsed	  " "$(($TimeWaiting / 60))" "$(($TimeWaiting % 60))"
	  printf "%01d:%02d Remaining\n" "$((($WaitReceive-$TimeWaiting) / 60))" "$((($WaitReceive-$TimeWaiting) % 60))"
	  sleep 1
   done
   PreviousTime=$SECONDS
fi
SECONDS=0
ping -c 1 -o -q $RemoteSystem
while [ $? -eq 0 ] && [ "$SECONDS" -le "$WaitShutdown" ]; do
   clear
   printf "$(hostname)  "
   date
   if [ "$PreviousTime" -le "$WaitReceive" ]; then
	  printf "Shutdown Request Received By $RemoteSystem\n"
   else
	  printf "Time allowed of %01d:%02d has been exceeded - No response from $RemoteSystem\n" "$(($WaitReceive / 60))" "$(($WaitReceive % 60))"
   fi
   printf "Waiting %01d:%02d for $RemoteSystem to be Receive Shutdown Request\n" "$(($WaitShutdown / 60))" "$(($WaitShutdown % 60))"
   TimeWaiting=$SECONDS
   printf "%01d:%02d Elapsed	  " "$(($TimeWaiting / 60))" "$(($TimeWaiting % 60))"
   printf "%01d:%02d Remaining\n" "$((($WaitShutdown-$TimeWaiting) / 60))" "$((($WaitShutdown-$TimeWaiting) % 60))"
   sleep 1
   ping -c 1 -o -q $RemoteSystem
done
PreviousTime=$SECONDS
SECONDS=0
while [ "$SECONDS" -le "$WaitExtra" ]; do
   clear
   printf "$(hostname)  "
   date
   if [ "$PreviousTime" -le "$WaitShutdown" ]; then
	  printf "$RemoteSystem Is OffLine\n"
	  printf "Delay %01d:%02d to make sure $RemoteSystem is completly Shutdown\n" "$(($WaitExtra / 60))" "$(($WaitExtra % 60))"
   else
	  printf "$RemoteSystem Is Still Online but MUST Shutdown in %01d:%02d\n$" "$(($WaitExtra / 60))" "$(($WaitExtra % 60))"
   fi
   TimeWaiting=$SECONDS
   printf "%01d:%02d Elapsed	  " "$(($TimeWaiting / 60))" "$(($TimeWaiting % 60))"
   printf "%01d:%02d Remaining\n" "$((($WaitExtra-$TimeWaiting) / 60))" "$((($WaitExtra-$TimeWaiting) % 60))"
   sleep 1
done

Here is my batch file that I run on startup on my Windows 10 vm
Wait4Shutdown.cmd

Code:

@Echo off
echo %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% System started >> Wait4Shutdown.log
if DEFINED IS_MINIMIZED Goto CheckforShutdown
timeout /t 10 /nobreak >Nul
If exist "\\192.168.0.252\QB_Server\SendShutdown.sh" Goto CheckforShutdown
echo [91mUnable to find network share!  Will not be able to detect Server Shutdown[0m
echo %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% Unable to find network share!  Will not be able to detect Server Shutdown >> Wait4Shutdown.log
Pause
Exit
:CheckforShutdown
if not DEFINED IS_MINIMIZED set IS_MINIMIZED=1 && start /min %~dpnx0 && exit
echo %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% Waiting for Shutdown >> Wait4Shutdown.log
:WaitforShutdown
Cls
Echo [92mWaiting for Shutdown[0m 
echo [93m%date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2%[0m
If exist "\\192.168.0.252\QB_Server\Shutdown" Goto ShutitDown
timeout /t 1 /nobreak >Nul
Goto WaitforShutdown
:ShutitDown
if %IS_MINIMIZED% == 1 set IS_MINIMIZED=0 && start %~dpnx0 && exit
echo %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% Shut Down Request Received from Server>> Wait4Shutdown.log
CLS
echo [91mShut Down Request Received from Server at: %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% [0m
Echo  [35m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in 30 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [36m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in 25 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [32m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in 20 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [33m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in 15 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [31m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in 10 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [95m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 5 /D ê /n /M "Initiating Shutdown in  5 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [94m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 1 /D ê /n /M "Initiating Shutdown in  4 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [96m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 1 /D ê /n /M "Initiating Shutdown in  3 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [92m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 1 /D ê /n /M "Initiating Shutdown in  2 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Echo  [93m
choice /c ê1234567890qwertyuiopasdfghjklzxcvbnm /T 1 /D ê /n /M "Initiating Shutdown in  1 Seconds.  Press a Character for Extended Time"
If Errorlevel=2 goto DelayShutDown
Goto GoShutDown
:DelayShutDown
Echo  [92m
Echo Automatic Shutdown Aborted....
Echo [93mServer is will force Shutdown of this VM in [96m5 Minutes[95m
Timeout /t 600
:GoShutDown
Echo  [91m
Echo Initiating Shutdown at: %date% %time%[0m
echo %Date% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2% Initiating Shutdown >> Wait4Shutdown.log
Del \\192.168.0.252\QB_Server\Shutdown
Shutdown /s
:Finish
Exit

This is the third script which changes directories to where the first script is and calls it, that way I can call multiple versions of the first script, wait for them all to finish, then issue the shutdown to the server.
ShutDownAll.sh

Code:

#!/bin/bash
cd "/mnt/Kirkland_Audio_4TB/4TB/User Datasets/QB_Server" > /dev/null
/bin/bash SendShutDown.sh
wait
Echo Finished the Script
sleep 10
poweroff

 

[Haggis663]

Why don't you just send a poweroff signal to all the VMs in the shutdown script e.g.
bhyvectl --vm=<vm name> --force-poweroff

And then give it a few seconds to shutdown

 

[KrisBee]

Why don't you just send a poweroff signal to all the VMs in the shutdown script e.g.
bhyvectl --vm=<vm name> --force-poweroff

And then give it a few seconds to shutdown

Man bhyvectl(8) doesn't explicitly state it, but this thread confirms that particular bhyve command is the equivalent of a hard power off on real hardware.

https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-April/004360.html

 

[Haggis663]

Okay, from https://github.com/ateamsystems/bmt/blob/master/bmt
tpid=`ps aux -ww | grep "bhyve: $VM_NAME (bhyve)" | grep -v grep | awk '{print $2}'`

kill -TERM $tpid

This should trigger an equivalent of the shutdown power button

 

[Patrick M. Hausen]

Uh oh ... grep | awk ... bad style ;)

Code:

#! /bin/sh

WAIT=120

ps auwwx | awk '/[0-9] bhyve:.*(bhyve)/ {print $2}' | while read pid
do
	kill -TERM "${pid}"
done

sleep "${WAIT}"

exit 0

Thanks a lot for that initial info!
Patrick

P.S. Set this as a shutdown script and it will cleanly shut down all of your bhyve VMs.

 

[Haggis663]

Nice.
Wouldn't it be better to scan every second for the PIDs to die rather than waiting 120s?

 

[Patrick M. Hausen]

Nice.
Wouldn't it be better to scan every second for the PIDs to die rather than waiting 120s?

Left as an exercise to the reader ;)

Patrick

 

[Patrick M. Hausen]

Code:

#! /bin/sh

PAUSE=1

ps auwwx | awk '/[0-9] bhyve:.*(bhyve)/ {print $2}' | while read pid
do
	kill -TERM "${pid}"

	while kill -0 "${pid}" >/dev/null 2>&1
	do
		sleep "${PAUSE}"
	done
done

exit 0

Better? ;)

Patrick

 

[Haggis663]

Good, but you are better off killing then both first and then waiting.

Code:

PAUSE=1
ps auwwx | awk '/[0-9] bhyve:.*(bhyve)/ {print $2}' | while read pid
do
	kill -TERM "${pid}"
done

ps auwwx | awk '/[0-9] bhyve:.*(bhyve)/ {print $2}' | while read pid
do
	while kill -0 "${pid}" >/dev/null 2>&1
	do
		sleep "${PAUSE}"
	done
done

exit 0

 

[Zaaphod]

The question remains, how do I execute a script from init/shutdown or UPS Service.

Why don't you just send a poweroff signal to all the VMs in the shutdown script e.g.
bhyvectl --vm=<vm name> --force-poweroff

And then give it a few seconds to shutdown

I do not want a force poweroff... I want to properly close down programs and save data, then do a shutdown. My script would allow me to do that. I can purposefully close specific programs and even automate saving of opened files before the shutdown command is given. This also allows me to shut down and confirm the shutdown of non-VM client computers before I shut down the server where clients may have open files or unsaved work still. I put a ridiculous battery backup on the server, 6 hours of battery reserve, so shutting down the server fast is not a concern, giving the clients a chance to shut down properly and close any open files is of greater concern.

Nice.
Wouldn't it be better to scan every second for the PIDs to die rather than waiting 120s?

I didn't even know there was a way to scan for the PID, thanks for the suggestion, I didn't know I could do such a thing, thanks for the suggestion. For non-VM clients I will need to rely on the IP address.

 

[Zaaphod]

I'm still having dificulty running any script from init/shutdown or the UPS service. I think I must be getting the syntax wrong? or perhaps it's a permission issue? It's just ignoring any script I put in either location. Does anyone have any advice?

 

[Patrick M. Hausen]

You can put you script in any location available to the FreeNAS system. I use /mnt/zfs/scripts. The script should start with #! /bin/sh if it is a shellscript, belong to root:wheel and have permissions 0755 or 0750.

Then you just add it through the UI - Tasks - Init/Shutdown Scripts. Enter the full path at "Script" and "Shutdown" at "When".

HTH,
Patrick