Unkillable process in jail; any tips?

[Jon Rosebaugh]

Twice now I've managed to get a process in a jail wedged in such a way that it cannot be killed. I don't have precise reproduction steps; once it happened while running mkvinfo on an avi file, and once while running a python script that should have imported os and sys and then exited. (In both cases it was in a directory mounted into the jail via the storage system.)

When this happens, I can't kill the process, even with -9. I can kill the shell that I launched it from, but all that does is make init the parent process of this unkillable process. I cannot shut down the jail; FreeNAS tries to do it, but the warden job just waits forever for something. Even shutting down the NAS is blocked by this process.

It doesn't seem to be actually a zombie process when it happens; ps shows it as runnable and using 100% of a CPU (I'm on a FreeNAS mini with eight cores). I can renice it up to 20, but I can't kill it. Each time this has happened I've resorted to powercycling the NAS to clear it, which is pretty annoying. Any tips if I manage to repro this again?

 

[DrKK]

That is certainly very odd. The storage you add to jails has always been slightly wonkadelic.

What if you try to kill the process from the FreeNAS appliance level itself, not from the jail? The pid is the same, and while you cannot kill a non-jail process from within the jail, you can certainly kill a jail process from the outer/main environment. Would that make a difference? I've not heard of this problem before.

 

[Jon Rosebaugh]

I've tried that as well, and certainly shutting down the jail should kill the processes from outside the jail. It doesn't seem to work.

(Note, I've never tried killing them from outside the jail FIRST, because how do you know it's happened until you can't kill it?)

 

[meku]

I experienced the same effect after running 'pkg update' from within a jail. The pkg process took 100% CPU and appeared to stay in the run state. Any attempts to kill the process with sig KILL had no effect. I also noticed that any subsequent commands in the jail that required disk access hung in state 'Uninterruptible sleep'.

FreeNAS was unable to shutdown cleanly so I had to do a hard-reset. This is running on 9.3 Stable.

 

[DrKK]

So, just so we're all on the same page, when you update a package or a port, the final stage will involve a potentially pinned pkg process for up to several minutes while it massages the database and whatnot. You guys are talking about something more severe than this, right? There are some packages that after an update may hang around pinning pkg for a good ten minutes.

 

[meku]

I wouldn't rule that out, though by the time I decided to reboot it had been running on CPU for 40min producing no output or logs I could see.

 

[Jon Rosebaugh]

In my case it had nothing to do with package installation, so.

 

[DrKK]

I'll mention this thread to John Hixson, one of the devs with a strength in jails. See what he thinks.

 

[Bytor]

I'll mention this thread to John Hixson, one of the devs with a strength in jails. See what he thinks.

I too, am having this issue. A process in a jail which is consuming 100% CPU and which is unkillable. The only sign (othe rthan actually logging in and doing top periodically) is that eventually NFSv4 shares can no longer be mounted from any other machines on my network.

When trying to reboot the NAS after this it takes 10+ minutes to shutdown with a screenfull of "Freed UMA keg... Lox X pages of memory" errors, "hhook_vnet_uninit" errors and one "ifa_del_loopback_route: deletion failed" message. Actually, just now I gave up in frustration and hit the power switch after 17 minutes.

This happens on both recent stable versions.

 

[Bytor]

I made a bug report. https://bugs.freenas.org/issues/10757

 

[DrKK]

Yeah I added this thread to the bug report. The bug report appears to be marked "meh" by Jordan. I will try to get more details from him later.

 

[Zubin Singh Parihar]

I too am having a similar issue as Jon Rosebaugh.

I have a Plex Jail. I decided to a do a 'pkg install rsync' to so that I could setup a cronjob to rsync my Music Directory from my workstation. The package installed, however, i was left with a 'pkg install' process in the jail that was consuming 100% CPU which was unkillable both from within and outside of the jail. I noticed because I also had an NFSv4 Share to the plex media directories which were not responding (nor was I unable to Unmount them). I was not able to restart the Jail from the GUI. I had to do a Full FreeNAS Reboot.

Build FreeNAS-9.3-STABLE-201512121950
Platform Intel(R) Atom(TM) CPU C2750 @ 2.40GHz
Memory 16349MB

Perhaps this is a FreeBSD bug rather then a FreeNAS bug, but was hoping that you'd be able to check on your end and then report it upstream?

Cheers,

Zubin Singh Parihar

 

[trumee]

I had the same problem. Here is my post regarding this http://freebsd.1045724.n5.nabble.com/Zfs-locking-up-process-td6043610.html

It seems to be freenas/freebsd 9.x problem. I changed over to vanilla. Freebsd the 10.2 and no longer have this issue.

 

[nightshade00013]

I have a Plex Jail. I decided to a do a 'pkg install rsync' to so that I could setup a cronjob to rsync my Music Directory from my workstation. The package installed, however, i was left with a 'pkg install' process in the jail that was consuming 100% CPU which was unkillable both from within and outside of the jail. I noticed because I also had an NFSv4 Share to the plex media directories which were not responding (nor was I unable to Unmount them). I was not able to restart the Jail from the GUI. I had to do a Full FreeNAS Reboot.

You should not need to do a rsync inside a jail. If you attach storage you should be able to set this up in the FreeNAS and let the jail do it's plex work. You could then either rsync the storage directory or snapshot it and back it up that way. If you are not using attached storage to the jail you should seriously consider it. Something happens to your jail and everything can easily go up in smoke.

You could also look up Syncthing and use it to sync the music in real time automatically. Just attach the same storage mount point in the Syncthing plugin jail.