Lots of pkg upgrade processes

[rwfitzy]

Yesterday I start getting alerts about a large number of processes from nagios running in a Jail. Again this morning over 1000 pkg upgrade processes. The server is running FreeNAS-11.3-U1 with Jail running 11.3-RELEASE-p7. I can see all the processes and kill from the Jail. How could I determine the cause? I was not able to find anything related to pkg in the system log. We haven't made any changes since way before this started happening as you can see my FreeNAS install needs to be updated.

 

[rwfitzy]

So, looking at this again this morning, I had to set up a cron task to `pkill pkg` every hour as a workaround until I had a chance to examine closer. When I run `pkg upgrade` from the Jail console, it loops indefinitely...

Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating FreeBSD_latest repository catalogue...
FreeBSD_latest repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating FreeBSD_latest repository catalogue...
FreeBSD_latest repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
Checking integrity... done (0 conflicting)
Your packages are up to date.

...

I've checked cron jobs, all are system other than those set by myself that have nothing to do with `pkg`. I have not been able to find how this process is getting started and believe the loop is what is causing so many processes to build up. I'm assuming since I see this from inside the Jail, that is where the processes are getting started. Any help appreciated.

 

[Samuel Tai]

Inside each jail, look at root's crontab via crontab -l. Typically, there will be a daily pkg update there. You can modify root's crontab per jail via crontab -e, or editing it directly in /var/cron/tabs.

 

[rwfitzy]

Thanks. I've never known FreeBSD or FreeNAS jail to run pkg upgrade automatically. Yes, I checked all cron jobs including root and find only my own cron jobs with nothing to do with pkg. I'm also familiar with /etc/periodic and only see system jobs. At this point I cannot use pkg in the jail, the same loop with pkg install. This jail has been in production for years. I'm thinking it must have been something done at one point and left hanging, but I've killed all pkg processes and just can't find where they are getting started from again. This jail is a maintenance server for Nagios monitoring and backups, the only other updates we do besides pkg upgrade are to those configurations.

Would it be wise to go ahead and update FreeNAS to latest version in this state? I'm always skeptical when things aren't right first.