Proper shutdown of jail & contained services

[frenziedengi]

I have a few jails setup on my FreeNAS 9.1.1 installation.
I am having a problem with the jails when they get stopped: it seems like when the jail get's stopped (via the Stop button in FreeNAS GUI, or via system reboot/shutdown), it's like "the plug is getting pulled" from the server (i.e. none of the processes/services get a chance to shutdown properly).

For example, I have a jail with MySQL installed (via pkg_add). I have added mysql_enable="yes" to /etc/rc.conf and MySQL will startup fine when the jail starts. If I stop the jail, the *pid file does not get deleted (which is what lead me to believe the service is not getting stopped properly), and some of the files that it uses are still "locked" which causes problems the next time that the jail starts up (MySQL fails to start).

So here are my questions:
1. What mechanism is used to stop the jail (how is the jail terminated behind the scenes of pressing stop button, or during a reboot)?
2. How to properly shutdown a jail, including shutting down services that are part of a jail?

My findings so far:
FreeBSD jail documentation: http://www.freebsd.org/cgi/man.cgi?...=FreeBSD+9.1-RELEASE&arch=default&format=html

This indicates that you can use the exec.stop argument to list what scripts to run when the jail is terminated, typically /etc/rc.shutdown. I am guessing that either this script is being run, and it is not effective, or it is not being run at all.

Thank you.

 

[Dusan]

1. What mechanism is used to stop the jail (how is the jail terminated behind the scenes of pressing stop button, or during a reboot)?

The button runs "warden stop [jailname]" which in turns calls "/usr/local/share/warden/scripts/backend/stopjail.sh [jailname]". stopjail.sh does run the jail's rc.shutdown (https://github.com/freenas/freenas/blob/master/src/pcbsd/warden/scripts/backend/stopjail.sh#L177).
I checked and it works properly in my jails. You may want to insert some debugging outputs into your rc.shutdown to see what is going on (warden redirects the output of the script to /dev/null so send your logging output into a file).

 

[frenziedengi]

It doesn't seem like it is getting run. I put some prints into the /etc/rc.shutdown in the jail, and the file never gets created that I print to.

When I manually run stopjail.sh <jailname> (not via the stop button in the GUI), it shuts down properly (and writes the log).

What is going on?

 

[frenziedengi]

OK, I think I figured out the problem....

I had created a mount point for mysql to store its database. The problem is that the mount point gets unmounted before the /etc/rc.shutdown script is run. So mysql does not get a chance to gracefully shutdown before the mountpoint (containing the database) is unmounted.

Why is it that mountpoints get unmounted first? It seems like that should be one of the last things done...?

From stopjail.sh:
# Check if we need umount x mnts
if [ -e "${JMETADIR}/jail-portjail" ] ; then umountjailxfs ${JAILNAME} ; fi

... some other stuff

calls /etc/rc.shutdown

... then later on

if [ -e "${JMETADIR}/jail-portjail" ] ; then
umountjailxfs
fi


EDIT:
The more I think about this, the more it seems wrong... What if you have a service busy writing data to the mount? You can't just unmount it. You need to gracefully stop the service first and then unmount.

Also, what is the point of calling the unmount twice? The second time without a jailname?

 

[Dusan]

You are right. I scanned the stopjail.sh script only briefly yesterday. I do not experience this problem as I don't use the jail mountpoints. You may want to create a bug report here: https://bugs.freenas.org/ (and/or prepare a patch and submit a pull request: https://github.com/freenas/freenas/pulls?direction=desc&page=1&sort=created&state=closed).
Also, warden comes from PCBSD (http://wiki.pcbsd.org/index.php/Warden®), so maybe create a bug report also here?: http://trac.pcbsd.org/

 

[frenziedengi]

Created: https://bugs.freenas.org/issues/3296

 

[Tsaukpaetra]

Strangely this seems to still be an issue. I noticed in the commit reference in the above ticket that specific references to the portjail jail type were removed, but why is this happening at all?
Regardless of jail types, unmounting of mount points should never ever ever happen before the jail is shut down. Can you image FreeNAS unmounting all of the pools before stopping all of the services? Havoc would ensue!

 

[cyberjock]

True. You'll also notice if you look closely that you are responding to a ticket from a year ago. :P

 

[Tsaukpaetra]

True. You'll also notice if you look closely that you are responding to a ticket from a year ago. :p

That's why I'm confused, as it's apparently still happening. This was the latest post about it, and I seem to be having problems with a year-old bug that was promised as fixed. :/

 

[aran kaspar]

That's why I'm confused, as it's apparently still happening. This was the latest post about it, and I seem to be having problems with a year-old bug that was promised as fixed. :/

Hey Tsaukpaetra, how much did you pay for FreeNAS?

 

[Tsaukpaetra]

Hey Tsaukpaetra, how much did you pay for FreeNAS?

I love people who resurrect things, thanks!
In response, I paid my time, talents, and effort. Because of CyberJock and I, FreeNAS has the ability to host VirtualBox. What's your claim to fame?

 

[cyberjock]

I love people who resurrect things, thanks!
In response, I paid my time, talents, and effort. Because of CyberJock and I, FreeNAS has the ability to host VirtualBox. What's your claim to fame?

Boom Shaka-laka!