SOLVED How to fix scrub schedule?

[joeschmuck]

I believe it's embeded in the coding and/or part of freenas-db. Here is what I've found which I think runs the smartctl test, look in /etc/local/periodic/daily and you will find a file called 'smart' but there are variables for the disk identifications and while some of this looks familiar from many years ago, I never got the full swing of this language. I think it will take someone familiar with programming this to answer your question.

Since the SMART tests do not look like they use CRON to validate the start day/date, this problem may not exist for smart tests.

 

[Ericloewe]

Maybe smartd is in charge of running SMART tests?

 

[HolyK]

Ok found it ... SMART tasks are handled via smartmontools and whatever you do in the GUI is reflected in /etc/local/smartd.conf

 

[joeschmuck]

Yup, that is what I get for using a test VM for looking. There are no actual tests setup to locate with it being a VM drive and smartd turned off. Lesson learned.

 

[JayG30]

I added it via GUI but i just found that it will not work because GUI is messing with the command string

If i use the "workaround" directly in console it is working:
[root@HolyNAS] ~# sh
# test `echo "$(date +%U) % 2" | bc` -eq 1 && echo "OK"
OK

But if i add the same via CRON it escapes the " % " automatically
[root@HolyNAS] ~# cat /etc/crontab | grep OK
00 02 * * 3 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" test `echo "$(date +\%U) \% 2" | bc` -eq 1 && echo "OK"
See the extra " \ " .... "$(date +\%U) \% 2"

Sadly this one is not working anymore
[root@HolyNAS] ~# sh
# test `echo "$(date +\%U) \% 2" | bc` -eq 1 && echo "OK"
test: bc:: unexpected operator

It seems like the GUI related to CRON jobs requires some polishing.


Thanks!

Code:

#!/bin/sh
test `echo "$(date +%U) % 2" | bc` -eq 1 && /usr/local/libexec/nas/scrub -t 14 st0rage

Make sure to chmod +x to the script and then call it.

Do either of you know, if you have entered your scrub into a cron job (whatever way you do it), how did you completely disable the default scrub?

When I disable the only scrub command through the web GUI and I check /etc/crontab I see it list the following;

Code:

45	 3	 *	 *	 *	 root	/usr/local/libexec/nas/scrub -t 35 freenas-boot

 

[JayG30]

Ok so here it comes ... simple workaround for the bi-weekly schedule ...

0 2 * * 1 root test `echo "$(date +%U) % 2" | bc` -eq 1 && /usr/local/libexec/nas/scrub -t 14 st0rage
0 2 * * 2 root test `echo "$(date +%U) % 2" | bc` -eq 1 && /usr/local/libexec/nas/scrub -t 14 redmirror

It basically checks if current week is odd or even. If it is odd then run scrub. This will ensure bi-weekly scrub execution no matter how much days each month has. Only issue is if the year has odd numbers of weeks so it will run scrub last week in the year and again at the first week in next year. But i guess it is OK.

BTW: I was about to use the same logic for smartctl long tests but for even weeks ... but they're not in the crontab ?! Where the schedule of smartct is actually stored?

Currently i have:
View attachment 18447

Which is again wrong due to the day-of-week vs day-of-month thing. But where i can find this?

Not sure if you discovered this but, I don't believe SMART schedules have the same limitation that scrubs do since they aren't handled through cron. I have a long SMART test scheduled like this;

• Hour: 02
• Day of Month: 23-31
• Month: All
• Day of week: Saturday

If you run smartd -q showtests it will show you the schedule. Here is my output (removing anything that isn't a long test) for my RAIDZ2 of 6 disks. And from what I can tell that schedule looks like what I expected, which is Day of Month and Day of Week use a logical AND operator.

Code:

Next scheduled self tests (at most 5 of each type per device):
Device: /dev/da0 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da1 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da2 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da3 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da4 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da5 [SAT], will do test 1 of type L at Sat Jun 24 02:19:15 2017 EDT
Device: /dev/da0 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da1 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da2 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da3 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da4 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da5 [SAT], will do test 2 of type L at Sat Jul 29 02:19:15 2017 EDT
Device: /dev/da0 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT
Device: /dev/da1 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT
Device: /dev/da2 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT
Device: /dev/da3 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT
Device: /dev/da4 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT
Device: /dev/da5 [SAT], will do test 3 of type L at Sat Aug 26 02:19:15 2017 EDT

Totals [Mon Jun 19 12:49:15 2017 EDT - Sun Sep 17 12:49:15 2017 EDT]:
Device: /dev/da0 [SAT], will do   3 tests of type L
Device: /dev/da1 [SAT], will do   3 tests of type L
Device: /dev/da2 [SAT], will do   3 tests of type L
Device: /dev/da3 [SAT], will do   3 tests of type L
Device: /dev/da4 [SAT], will do   3 tests of type L
Device: /dev/da5 [SAT], will do   3 tests of type L

 

[HolyK]

Do either of you know, if you have entered your scrub into a cron job (whatever way you do it), how did you completely disable the default scrub?
When I disable the only scrub command through the web GUI and I check /etc/crontab I see it list the following;

Code:

45	 3	 *	 *	 *	 root	/usr/local/libexec/nas/scrub -t 35 freenas-boot

Well i would say the scrub of the boot-device is an internal schedule aka "must-have" so this one will be always there. I personally don't care as it takes couple of minutes to complete - my boot device is a USB stick.
Rest of them i just disabled/deleted from the SCRUB section of GUI and scheduled custom CRON job which is executing the wrapper script.

Code:

[root@HolyNAS] /# cat /etc/crontab | grep scrub_
00  02  *  *  1  root  PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/misc/scripts/scrub_st0rage.sh
00  02  *  *  2  root  PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/misc/scripts/scrub_redmirror.sh

[root@HolyNAS] /# cat  /mnt/misc/scripts/scrub_st0rage.sh
#!/bin/sh
test `echo "$(date +%U) % 2" | bc` -eq 1 && /usr/local/libexec/nas/scrub -t 14 st0rage

Not sure if you discovered this but, I don't believe SMART schedules have the same limitation that scrubs do since they aren't handled through cron. I have a long SMART test scheduled like this;

Yes, well, partially. I found that the SMART tasks are handled via smartmontools and whatever you do in the GUI is reflected in /etc/local/smartd.conf
Anyway i was not aware about the smartd -q showtests so thanks for that one.

 

[JayG30]

I actually didn't realize the scrub that was being listed was for the boot drive, lol. I thought it was still referencing my data pool. That makes more sense. :)

 

[JayG30]

Maybe one of you can help me on this. I've never added a cron job via the GUI. I want to have my scrub run on the first Saturday of every Month. I found that this command supposedly will do this cron job will supposedly do that for me.

Code:

00 02 * * 6 [ $(date +\%d) -le 07 ] && /run/your/script

In the web GUI for cron jobs should I only enter the [ $(date +\%d) -le 07 ] && /run/your/script into the Command field, while setting the 00 02 * * 6 using the Min/Hr/DayofMonth/Month/DayofWeek options in the GUI?

It looks like my command only has a single escape character so I think it should work entered through the GUI (looks like it adds the escape character itself so you don't put it into the command field).

This is what my crontab looks like;

Code:

00	  02	  *	   *	   6	   root	PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" [ $(date +\%d) -le 07 ] && /usr/local/libexec/nas/scrub -t 21 store > /dev/null

 

[HolyK]

Well the automatically escaped "%" ( \% ) did not worked for me so i had to wrap a .sh script around and call that one from cron (see my last post above). You can easily test it by opening your command line, call a sh (instead of csh) and use the command (replace the scrub part via something like ... echo "OK" ... If it will go through then the call from cron should be OK as well.

And yes the "date" check is OK. Cron job will be called every Saturday. If the day of month is 1-7 then /your/script is called. Otherwise the date check fails thus nothing is executed.

EDIT: It looks like it works even with the \% ...
# [ $(date +\%d) -le 77 ] && echo "OK"
OK

# [ $(date +\%d) -le 7 ] && echo "OK" || echo "Nope, today is `date +%d` so fu*k off!"
Nope, today is 20 so fu*k off!

 

[JayG30]

Yea I think your problem was because you had/need 2 escapes.

 

[HolyK]

I just found why i had issues ...

My original entry is:
test `echo "$(date +%U) % 2" | bc` -eq 1 && echo "OK"
OK

When stored as cron entry via GUI all of the "%" gets escaped, so it looks like this:
test `echo "$(date +\%U) \% 2" | bc` -eq 1 && echo "OK"
test: bc:: unexpected operator

The problem is that the second "%" is a "modulo" so it can not be escaped otherwise it is interpreted as a plain symbol "%" rather than a operator. If only the first one gets escaped and the second not, it works fine.
test `echo "$(date +\%U) % 2" | bc` -eq 1 && echo "OK"
OK

So basically FreeNAS GUI is messing with the CRON standards. In normal crontab you need to handle all escapes properly by yourself. FreeNAS GUI is trying to help and covers these missing escapes automatically which is breaking other (modulo in this case) stuff.

I suppose this change caused it which was based on this discussion. Well no-one realized that before :/

EDIT: Bug #24084 updated

 

[JayG30]

Per cronjob man page HERE.

The entire command portion of the line, up to a newline or % character, will be executed by /bin/sh or by the shell specified in the SHELL variable of the cronfile. Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.

They give an example showing a cron job where the % sign should be treated as newline;

Code:

0 22 * * 1-5	mail -s "It's 10pm" joe%Joe,%%Where are your kids?%

And if you enter that into freenas it escapes every % trying to "help" and would result in the WRONG cron job;

Code:

00 22 * * 1-5 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" mail -s "It's 10pm" joe\%Joe,\%\%Where are your kids?\% > /dev/null

I believe all % you write using date need to be escaped however, and I believe you are even told so in documentation.

When you enter these manually on any other OS nothing automatically adds escapes for you. So personally I don't think that freenas should be inserting these escapes on it's own either. And it certainly shouldn't do it without even communicating that to the user through the GUI.

I'd simply remove this behavior entirely and leave it to the user to enter the CORRECT cron job as he would need to do if he was manually adding a cron job.

 

[JayG30]

@HolyK

Can you try the following;

Code:

expr `date +\%W` \% 2 >/dev/null ||  /usr/local/libexec/nas/scrub -t 14 st0rage

Or

Code:

expr `date +\%s` / 604800 \% 2 >/dev/null || /usr/local/libexec/nas/scrub -t 14 st0rage

Obviously if adding it through the web GUI don't include the escape characters since it will add them itself.
Reference HERE.

I've come across a bunch of different examples of how to do what you are doing. Some escape the modulo and others don't. But reading the man page it sounds like it should be escaped. Doing some testing of my own the above seems to work with the escaped modulo. Which is leading me to think the issue is something else related to the constructed cron job you have.

FYI, pick whichever fits your need best.
%W Week number of year, with Monday as first day of week (00..53).
%U Week number of year, with Sunday as first day of week (00..53).
%V ISO week number, with Monday as first day of week (01..53).

 

[HolyK]

Okay so basically the escaped % works (should work?) via cron (it should be actually escaped as per man page) but it does not work via CLI (here it can not be escaped). I haven't tested it yet but my assumption is as follows (Correct me if i am wrong).

test `echo "$(date +%U) % 2" | bc` -eq 1 && echo "OK"
// Works from CLI
// Should not work via CRON (not tested)

test `echo "$(date +\%U) % 2" | bc` -eq 1 && echo "OK"
// Works from CLI
// Should not work via CRON (not tested)

test `echo "$(date +\%U) \% 2" | bc` -eq 1 && echo "OK"
// Does NOT work from CLI
// Should work from CRON

So one mess is that "%" has different meaning in CLI (modulo) and CRON (new line).
Another mess is that GUI is escaping every single occurrence of '%' so it can not be used as "new line"

Anyway i will stick with the wrapper script. It works flawlessly. :]