scheduled scrub not running

[dominator99]

Hi All

I have 4No. mirrors (Mirror-1 to Mirror-4) each scheduled to run a scrub on the same day but at different times. Today was the first time the scrubs were scheduled to run; only 3 of the 4 scrubs ran as scheduled, the 4th had to be started manually using the shell command zpool scrub Mirror-2 which completed successfully without errors.
Any suggestions as to why the scheduled Mirror-2 didn't run?

 

[rs225]

Most likely you already had the maximum number of scrubs running when the 4th tried to start.

 

[dominator99]

Thanks but not so; other scrubs had already finished. I even changed the schedule start time to see if it would run on its own but no joy
Incidentally, how many scrubs can run simultaneously?

 

[rs225]

It's tunable I think, and I think the default is 1 scrub or resilver at any time.

 

[dominator99]

Thanks for that.
If tunable how would I go about changing the default scrub to 4 scrubs?

 

[MrToddsFriends]

It's tunable I think, and I think the default is 1 scrub or resilver at any time.

My scrub schedule comprises of three simultaneous scrubs (that is: three scrubs on different pools) and I have never seen a problem with that.

There is a "scrub max active" sysctl value that should be tunable, but --- according to it's hierarchical name --- I think that this apples to the number of simultaneous scrubs on different vdevs within a pool.

Code:

~ # sysctl vfs.zfs | grep scrub
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.scrub_delay: 4

Can anyone confirm or revoke my assumption?

 

[rs225]

My information is definitely out of date(one of which is the zpool man page). I just started 4 scrubs on one machine, and there was no problem.

 

[dominator99]

Just started manual scrubs on 4 mirrors (each start was confirmed in GUI) but running zpool status in shell showed only 3 scrubs running.
When I manually stopped the scrubs using GUI (volumes > mirror-x > scrub volume) only 3 of the 4 mirror scrubs were running.
How can I change the number of scrubs running simultaneously to 4?

 

[dominator99]

My scrub schedule comprises of three simultaneous scrubs (that is: three scrubs on different pools) and I have never seen a problem with that.

There is a "scrub max active" sysctl value that should be tunable, but --- according to it's hierarchical name --- I think that this apples to the number of simultaneous scrubs on different vdevs within a pool.

Code:

~ # sysctl vfs.zfs | grep scrub
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.scrub_delay: 4

Can anyone confirm or revoke my assumption?

I get the same as above so I agree the most likely explanation is as you say.
Any suggestions on how to run 4 simultaneous scrubs?

 

[MrToddsFriends]

Any suggestions on how to run 4 simultaneous scrubs?

I'm still not sure if there's a limit in FreeNAS that allows three simultaneous scrubs but not four or if the symptom you are seeing is caused by another reason. Is it the same scrub job that refuses to run each time you try to start them regardless of the order in which you are starting them? What's the output of zpool status -v?

 

[dominator99]

I reversed the order of the scrubs on my 4 mirrors & it is the last (4th) one chosen that doesn't run even though the GUI said the 4th scrub had started (see attached file) which shows scrub had completed but the run time of the scrub was only a few seconds

 

[MrToddsFriends]

Is the amount of data stored on Mirror-4 small? If so it's not surprising that a scrub takes not much time.

 

[dominator99]

This is a shell print after starting a 4 mirror scrub. As can be seen the last (Mirror-4) scrub hasn't started even though the GUI said it had!

 

[dominator99]

Is the amount of data stored on Mirror-4 small? If so it's not surprising that a scrub takes not much time.

Just missed your response.
Yes this could be the answer as I only have a small script folder on mirror-4. I'll copy a large file to mirror-4 & restart the scrub while the other 3 scrubs are still running

 

[MrToddsFriends]

I have a small SSD pool in my system besides two HDD pools and this is what zpool status -v tells me (there is no evidence that the scrub on Tue Aug 15 did not run, it just takes almost no time):

Code:

~ # zpool status -v volvar
  pool: volvar
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
  still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
  the pool may no longer be accessible by software that does not support
  the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Aug 15 23:00:07 2017
config:

  NAME  STATE  READ WRITE CKSUM
  volvar  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  gptid/343521f2-a9a6-11e4-b9d3-0cc47a317e3c  ONLINE  0  0  0
  gptid/34949893-a9a6-11e4-b9d3-0cc47a317e3c  ONLINE  0  0  0

errors: No known data errors

 

[dominator99]

Just copied a 9GB file to mirror-4 & ran a scrub as detailed in last post & can confirm that all 4 scrubs ran simultaneously (although the mirror-4 scrub didn't run for long due to the relatively small mirror size.
Good suggestion MrToddsFriends & thanks:)