Cron jobs not running concurrently

[websmith]

Hi,
I have an issue with my cron jobs not running if there is an existing cron job already running.

FreeNAS-11.3-U5, Xeon D-1528, 128GB RAM

I have a cron task running every 15 minutes that pings healthchecks.io

@ 04.00/04.11 in the night my rsync kicks in that rsyncs some compressed files to rsync.net. This takes a long time - usually several hours.

And it seems like when those two rsync jobs run FreeNAS stop calling the rsync job that is supposed to be running ever 15 minutes.

Looking at the cron log file I see the following:

Code:

Feb 21 04:00:00 nas cron: (root) CMD (/mnt/tank3/system/tasks/rsync.net.sh /mnt/tank2/backup/veeam/VeeamConfigBackup veeam)
.....
Feb 21 04:11:00 nas cron: (root) CMD (/mnt/tank3/system/tasks/rsync.net.sh /mnt/tank2/backup/veeam/critical_servers veeam)

My two rsync jobs kicking in 11 minutes apart
Then when the two rsync job finally finishes @ around 08.30 I see a range of "pings"

Code:

Feb 21 08:31:09 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:12 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:15 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:17 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:20 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:22 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba- > /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:25 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:27 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:30 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:32 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:35 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:37 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:40 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba- > /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:42 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:45 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:47 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:50 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)
Feb 21 08:31:52 nas cron: (root) CMD (curl -fsS --retry 3 https://hc-ping.com/91518b85-f9e7-4f8c-a5ba-> /dev/null > /dev/null 2> /dev/null)

A total of 18 requests being made, which mathces the number of request that should have been done in the 4.5 hours the rsync tasks took.

So it seems like freenas simply queued up all the 15 minute pings while the two slow cron jobs were running and when they finally finished it released the queue and triggered them all in almost one go.

This causes me to receive false negative alerts from healthchecks.io every night my rsync cron jobs run.

The issue started appearing more often when I expanded my rsync jobs to run more concurrent jobs instead of just one even slower job.

Is this a known issue that there are limit to parallelization of cron jobs? i.e. the number of "threads" FreeNAS assigns to cron jobs=

Thanks in advance.

EDIT: I will try to "upgrade" to truenas 12 and see if that makes any difference.

 

[websmith]

Ok,
It seems like upgrading to Truenas 12 resolved the issue.
At least I haven't seen it yet.
I will report back after the upcoming weekend where my backups take 4.5 hours to upload to see if the issue comes back.