SOLVED Problem with running VM when virtualbox jail stops - how to fine tune how your jail is stopped

[Doc Chacha]

Hi all.

I've been wondering quite some days wether I should post thit or not. Having read a bunch of things on the forums, I think I should now ask for help.

Here's my setup:
-I have a VirtualBox jail (the default one), in which I created a Debian64 VM, running a Debian 8.6.0 64 bits distro. (First point -> I remember reading somewhere that 64 bits in virtualbox in FreeNAS were bad. Anyway, I tested it by replacing my 64 bit VM with a 32 bit one, and the problem stays the same)
-In that VM, I run an instance of Goodsync server, which has access to my main datasets via NFS shares (I did not go with the shared folders over a jail storage for various uid/gid problems)

The problem occurs at server shutdown... At this time, I'm stting up UPS at home on my two NAS, and I discovered this problem when FreeNAS shuts down at UPS warning threshold.
When I shutdown the server manualy, or when UPS service shuts it down, I get a system hang. My IPMI remote console is flooded with trap errors and kinds of memory dump... And finaly, it reboots (watchdog must play a part in the reboot). The problem is : it REBOOTS ! It doesn't just shuts down. So if I'm on UPS power, it's a real issue.
If I first shutdown the VM manualy, either using the web GUI or CLI via SSH, and *then* shut the server down, everything goes fine.

Long story short: I read a lot about this. I tried various comands in stop scripts, both in VM, in virtualbox jail... I tried to debug using commands to write things in files...
I finaly came on a post about stopping jails scripts. And digging in stopping scripts and pre stopping scripts, I realised that when stopping a jail, the process is:
1- stop networking
2-stop user mounts
3-launch custom stop script
4-unmount jail filesystem
5-other things

I think my problem occurs because the warden disconnects the jail from the network (and thus, the NFS shares) before it unmounts the NFS folders. I could tweak things there, in the stop scripts, but it's probably not a good idea to tinker with things like that.
I would like to have your point of view: is this behaviour causing my problem ? If so, could it be considered to be a bug ?
If not, what could cause that problem, and why would it work when I stop the VM manualy *before* I shutdown the server ?

(I know I will get some "a server should be up 24/7" answers. And I agree. My concern is to manage a worst case scenario, where I get a power loss long enough to drain my UPS battery, and the servers have to stop. They should have a chance to do it properly...)

Looking forward to read from this awesome community !

 

[Robert Trevellyan]

Leaving a VM running when FreeNAS is shutting down us like pulling the power cord out of a physical machine.

Does this help?
https://forums.freenas.org/index.php?threads/autostart-virtualbox-vm.22116/

 

[Doc Chacha]

I understand your real life example.
My problem is : how do I get FreeNAS to press the "shutdown" button instead of pulling the cord of the VM when the UPS service tells it it'd better shut the lights for now....

I had already tried some of these tricks.
Just got another try for all of them after reading this post.
No luck...

Anyway, thanks for you lightspeed answer ! :)

 

[pasiz]

Listen with your vm system underlying nut message?

 

[Doc Chacha]

Listen with your vm system underlying nut message?

That might work. But it would on be a workaround, at best...
I'm going to investigate a bit further, but if I cannot find another way to proceed, your idea is definitely something I'll try. Thanks.

 

[Doc Chacha]

At this point, here's where I am:
-when shutdown occurs, the warden calls /usr/local/share/warden/scripts/backend/stopjail.sh [jailname] which ultimately calls the jail's rc.shutdown (as I read in this post : https://forums.freenas.org/index.php?threads/proper-shutdown-of-jail-contained-services.15555/)
-the stopjail.sh does some checking before actually beginning the dhutdown process. (Checking path, variable, and so on...)
-it also calls a jail-pre-stop.sh script that lives in the .[jailsname].meta folder, situated in your jails base dir.
-this script launches /etc/rc.freenas which defines the jail-pre-stop base function. It's main purpose seems to look if the jail is a plugin jail, and if it is, to stop the plugin services in the jail before going on in the jail shutdown process (it uses jexec [jail id variable] [plugin service name variable] stop)

So, I thought I might use the space in my jail's meta folder jail-pre-stop.sh, before it calls the base function, to shutdown my VM (/usr/sbin/jexec [script in jail to shutdown the VM] && sleep a bit)

... which turns out to work perfectly !

(Marked as Solved, and title edited to better reflect the post content)

 

[pasiz]

... which turns out to work perfectly !

Good to hear. Must remember if I ever run VM instances in freenas.