lithium3141
Cadet
- Joined
- May 17, 2020
- Messages
- 2
Hello all! I'm having an issue with my FreeNAS 11.3-U2 installation, and I've been pretty stuck on it. I'd be grateful for any pointers or clues anyone can offer.
As far as I can tell, the problem is this: on occasion, some regular task in the system spawns a
Since around the time I upgraded to FreeNAS 11.2, these commands have been hanging and slowly consuming all the available RAM in the system. If I let them run long enough, they consume all the swap as well, and I'm forced into rebooting the system, since at that point I'm not able to log in or take any other action. It takes more than a week to get to that point, though, so for the time being I've been logging in and killing the offending processes every couple of days.
The actual commands getting spawned look like the following:
After identifying these processes, I searched around and found a FreeBSD forums thread which referenced exactly these periodic jobs. It recommended that I set
I noticed in
I also checked the output of
Nevertheless, this info correlates roughly with the timing of when I started noticing the issue. I never had problems with the Plex plugin as a warden jail, but after reinstalling it in an iocage jail shortly after updating to 11.2, this problem cropped up. It persists in 11.3-U2, which I'm running now.
Does anybody know what's going on here? I'm about at the end of my useful diagnostic knowledge, and I'd be incredibly grateful for any ideas about next steps I can take.
As far as I can tell, the problem is this: on occasion, some regular task in the system spawns a
find command, which gets "stuck" and starts consuming tons of memory. There are two jobs in /etc/periodic/security, 100.chksetuid and 110.neggrpperm, which were particularly noticeable culprits; these commands are meant to look for files with particular combinations of permissions as a regular security check. A third command seems like it's related to software updates, but I haven't been able to find its source.Since around the time I upgraded to FreeNAS 11.2, these commands have been hanging and slowly consuming all the available RAM in the system. If I let them run long enough, they consume all the swap as well, and I'm forced into rebooting the system, since at that point I'm not able to log in or take any other action. It takes more than a week to get to that point, though, so for the time being I've been logging in and killing the offending processes every couple of days.
The actual commands getting spawned look like the following:
Code:
# From 100.chksetuid
find -sx / /dev/null ( ! -fstype local ) -prune -o -type f \
( -perm -u+x -or -perm -g+x -or -perm -o+x ) \
( -perm -u+s -or -perm -g+s ) -exec ls -liTd {} +
# From 110.neggrpperm
find -sx / /dev/null ( ! -fstype local ) -prune -o -type f \
( ( ! -perm +010 -and -perm +001 ) -or \
( ! -perm +020 -and -perm +002 ) -or \
( ! -perm +040 -and -perm +004 ) ) \
-exec ls -liTd {} +
# From somewhere I can't find
find -s / ! ( -fstype tmpfs -or -fstype zfs ) -prune \
-or -path /tmp -prune \
-or -path /usr/tmp -prune
-or -path /var/tmp -prune \
-or -path /var/db/portsnap -prune \
-or -path /var/db/freebsd-update -prune \
-or -name .zfs -type d -prune \
-or -print
After identifying these processes, I searched around and found a FreeBSD forums thread which referenced exactly these periodic jobs. It recommended that I set
setuid=off for several of my datasets, since the jobs only run on datasets which have both exec and setuid enabled. However, that didn't fix the issue, and I noticed after the fact that the find commands were operating on /, so changing the properties on unrelated datasets might have been pointless. These changes also probably wouldn't have any effect on the third "mystery" command.I noticed in
top -o res that the commands growing in memory were often in a zfs-related state. To dig a little deeper, I waited for one of these commands to get into this bad state, then looked at what files it was trying to access with fstat -p. It seemed like the results were all related to my iocage jail that hosts the Plex plugin:Code:
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W nobody find 1727 text /mnt/data/iocage/jails/plex-plexpass/root/usr/bin 13286 -r-xr-xr-x 57744 r nobody find 1727 wd /mnt/data/iocage/jails/plex-plexpass/root 106877 drwxr-xr-x 62112981 r nobody find 1727 root /mnt/data/iocage/jails/plex-plexpass/root 4 drwxr-xr-x 24 r nobody find 1727 jail /mnt/data/iocage/jails/plex-plexpass/root 4 drwxr-xr-x 24 r nobody find 1727 0* pipe fffff800246ef5f0 <-> fffff800246ef758 0 rw nobody find 1727 1* pipe fffff80108ace460 <-> fffff80108ace2f8 0 rw nobody find 1727 2 /mnt/data/iocage/jails/plex-plexpass/root/dev 15 crw-rw-rw- null w nobody find 1727 3 /mnt/data/iocage/jails/plex-plexpass/root 4 drwxr-xr-x 24 r nobody find 1727 4 /mnt/data/iocage/jails/plex-plexpass/root 4 drwxr-xr-x 24 r nobody find 1727 5 /mnt/data/iocage/jails/plex-plexpass/root 106877 drwxr-xr-x 62112981 r
I also checked the output of
procstat -kk, which seemed to confirm that something zfs-related was afoot — but I couldn't glean any more information from the output than that.Nevertheless, this info correlates roughly with the timing of when I started noticing the issue. I never had problems with the Plex plugin as a warden jail, but after reinstalling it in an iocage jail shortly after updating to 11.2, this problem cropped up. It persists in 11.3-U2, which I'm running now.
Does anybody know what's going on here? I'm about at the end of my useful diagnostic knowledge, and I'd be incredibly grateful for any ideas about next steps I can take.
