Scrub schedule not running properly

[echelon5]

Hi,

I have 2 pools and I have a problem with my scrub schedule. I had it setup to run monthly:

• first pool (tank1) on the 1st, treshold days 20
• second pool (tank2) on the 3rd, treshold days 20

On the 14th a scrub started automatically for tank1:

• E-mail subject: Cron <root@FreeNAS> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 20 tank1"
• body: starting scrub of pool 'tank1'"

I thought that's weird so I've setup the scrub schedules according to cyberjock's guide, tank1 and tank2: every 1st and 15th, threshold days 10.

Last night, on the 16th, a scrub started automatically for tank2:

• E-mail subject: Cron <root@FreeNAS> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 10 tank2
• body: starting first scrub (since reboot) of pool 'tank2'

So, from what I undestrand the scrubs started outside the schedule. On the 13th I tried to update FreeNAS from 9.10.1-U2 to 9.10.1-U4 and it failed, so I upgraded manually with a USB stick.

 

[Robert Trevellyan]

Please post screenshots of all your scrub schedule settings.

 

[echelon5]

sorry, the email notification got stuck in the junk filter.

I'm away so i made screenshots to tank1 aka DATA from my phone.

Another scrub for this pool started last night on the 25th.

 

[joeschmuck]

So what I see above is you have your pool DATA scheduled to run a scrub on the 1st and 15th of each month at 1 AM with a threshold of 10 days. This doesn't match what you had indicated in your first posting but regardless this doesn't make sense that a scrub kicked off again on the 25th. The only thing I can think of right now is because of the upgrade you made, maybe the scrub schedule got hosed up. You might want to create a bug report on this.

 

[echelon5]

I thought that's weird so I've setup the scrub schedules according to cyberjock's guide, tank1 and tank2: every 1st and 15th, threshold days 10.

Doesn't this match the screenshots?

I think that I have another unit that did this. I'll check properly when I get home. The schedules seem to follow everything except for days of month.

I'll see if I have enough data file a bug.

 

[Sakuru]

Yeah, 9.10.2 messed up the schedules. Scrubs, SMART tests, snapshots, etc. I think someone said 9.10.2-U1 is due out around the 9th, even though the bug tracker says 2/6 https://bugs.freenas.org/projects/freenas/roadmap#9.10.2-U1.

 

[echelon5]

So, I've deleted and recreated the scheduled scrub jobs and they are still running erratically. Even though they are scheduled to run at the same time, they run on different dates and didn't run at all on the scheduled dates. They are running every 10 days, the treshold days. Since then I've upgraded the install to 9.10.2.

Excerpt from crontab:

Code:

00	  01	  01,15   *	   1,3,4,5,6,7	 root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 10 DATA

00	  01	  01,15   *	   1,3,4,5,6,7	 root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 10 Greuceanu

 

[joeschmuck]

So, I've deleted and recreated the scheduled scrub jobs and they are still running erratically. Even though they are scheduled to run at the same time, they run on different dates and didn't run at all on the scheduled dates. They are running every 10 days, the treshold days. Since then I've upgraded the install to 9.10.2.

Excerpt from crontab:

Code:

00	  01	  01,15   *	   1,3,4,5,6,7	 root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 10 DATA
 
00	  01	  01,15   *	   1,3,4,5,6,7	 root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/libexec/nas/scrub -t 10 Greuceanu

Submit a bug report.

 

[echelon5]

Bug reported here.

 

[joeschmuck]

Glad you did this, it is the only way to effectively troubleshoot the problem if this is a software issue.

 

[joeschmuck]

I've updated the bug report and there is more understanding of what is going on and a recommended solution. Also see this posting.

 

[zoomzoom]

Just wanted to clarify something, as I think I have the wrong understanding of threshold days.... Does that value mean x job will run every 10 days, or that x job will run 10 days after the scheduled job if that scheduled job didn't run?

 

[joeschmuck]

I really didn't want to reopen this thread, just update the the folks subscribed to it but if you look in the linked thread I added for the bug report, it will explain it some.

There are two things at work here, one is CRONTAB. CRONTAB will check once a minute to see if the conditions have been met and if that is true then it will send out the command, in this case it's a scrub. The second thing is that scrub command, the -t value is the threshold. If the value is "-t 10" then if it's been greater than 10 days since the last scrub, the scrub will commence, if not then is will exit immediately. That is how I understand it. The issue we are having is the "day of the week" value, I don't think it works as the developers think or it definitely is not clear in the GUI how it works.

Please refer to the bug report listed above, there is more details there.

 

[echelon5]

I've updated the bug report and there is more understanding of what is going on and a recommended solution. Also see this posting.

After reading your post I've enabled my scrub jobs, but I've checked all days of the week. So, the schedule was setup with threshold days 10, every 1st and 15th of the month. The second scrub started last night so it seems to be working right. I've enabled all scheduled tasks and I'll be keeping an eye on them.

From my tests and your analysis it seems that cron turns the schedule to an OR when setting it to run only on specific days.

 

[danb35]

The second thing is that scrub command, the -t value is the threshold. If the value is "-t 10" then if it's been greater than 10 days since the last scrub, the scrub will commence, if not then is will exit immediately.

...and more specifically, the threshold is the number of days since the last scrub completed. If you use -t 10, and it's been 9 days, 23 hours, 50 minutes since the last scrub completed, that's under the threshold. So, if I want my scrubs to run every other Sunday, I set them for Sundays with a threshold of 12 days.