ZFS Scrub Happening Everyday (instead of every 30)

[david31262]

Build: FreeNAS-8.0-RELEASE-amd64

For some reason, my ZFS is scrubbing every day (I thought it's supposed to be every 30 days).

I'm aware of 800.scrub-zfs, but I'm not enough of a linux person to understand what is going on there.

Here's my zpool history:

Media1_0-Z# zpool history
History for 'Media1_0-Z':
2011-06-01.20:23:03 zpool create -o cachefile=/data/zfs/zpool.cache -fm /mnt/Media1_0-Z Media1_0-Z raidz /dev/gpt/ada0 /dev/gpt/ada1 /dev/gpt/ada10 /dev/gpt/ada11 /dev/gpt/ada2 /dev/gpt/ada3 /dev/gpt/ada4 /dev/gpt/ada5 /dev/gpt/ada6 /dev/gpt/ada7 /dev/gpt/ada8 /dev/gpt/ada9
2011-06-01.20:25:19 zfs set compression=off Media1_0-Z
2011-06-01.20:25:19 zfs set atime=off Media1_0-Z
2011-06-01.20:25:20 zfs set refreservation=0 Media1_0-Z
2011-06-01.20:25:20 zfs set refquota=none Media1_0-Z
2011-06-01.21:27:02 zfs create -o atime=on -o compression=lzjb Media1_0-Z/David
2011-06-01.21:27:38 zfs create -o atime=off -o compression=off Media1_0-Z/TV_0-Z
2011-06-01.21:36:27 zpool import -o cachefile=none -f Media1_0-Z
2011-06-01.22:02:15 zpool import -o cachefile=none -f Media1_0-Z
2011-06-01.22:50:55 zfs destroy Media1_0-Z/David
2011-06-01.22:51:05 zfs destroy Media1_0-Z/TV_0-Z
2011-06-01.22:51:41 zfs create -o atime=on -o compression=lzjb Media1_0-Z/David
2011-06-01.22:52:03 zfs create -o atime=off -o compression=off Media1_0-Z/TV_0-Z
2011-06-01.22:56:38 zpool import -o cachefile=none -f Media1_0-Z
2011-06-01.23:08:38 zfs destroy Media1_0-Z/TV_0-Z
2011-06-01.23:10:19 zpool import -o cachefile=none -f Media1_0-Z
2011-06-01.23:11:24 zfs create -o atime=off -o compression=off Media1_0-Z/TV_0-Z
2011-06-01.23:15:24 zpool import -o cachefile=none -f Media1_0-Z
2011-06-09.21:51:39 zpool import -o cachefile=none -f Media1_0-Z
2011-06-10.03:02:35 zpool scrub Media1_0-Z
2011-06-11.03:02:05 zpool scrub Media1_0-Z
2011-06-12.03:02:05 zpool scrub Media1_0-Z

Perhaps I'm missing something.

 

[William Grzybowski]

Who told you it is supposed to work every 30 days? Is it written somewhere?
I am just trying to track where this information came from... to fix it..

The periodic is set to run everyday for periodic, that should probably be longer, i'll double check

 

[david31262]

I'm not home right now, so I'm working from memory. But, if you look at the script that periodic calls, there is a field with an assignment to a days between scrubbing. It's set to 30. It then reads the "zpool history" (that I originally included) to get the time/date since the last scrub. It then only runs the scrub if the appropriate time has passed. We really don't need to scrub daily. For my large box it's taking 4-6 hours (from memory). The array takes a terrible performance hit while scrub is running & it's a waste of power.

Also, I noticed this today in my Daily Status E-Mail... Looks like some kind of error is being reported by the script doing the "days between" check:

-----------------------------------------------------
Scrubbing of zfs pools:
[: Z_threshold: bad number
starting scrub of pool 'Media1_0-Z':
consult 'zpool status Media1_0-Z' for the result

 

[William Grzybowski]

Yeah, you're right... so the script has some kind of error in the parser

 

[James]

The script that does the scrubbing is a shell script, and it's not handling the - in the pool name properly. It's getting handed to a test and parsed as an option flag, which is then blowing up the script. We are working on a fix now.

 

[jpaetzel]

A fix has been committed to both FreeNAS and FreeBSD HEAD. In the process we upgraded the script to the version in FreeBSD HEAD since it's slightly better than the version in FreeBSD 8.2.

A caveat. Collisions are still possible. Because ZFS allows multiple characters in pool names that are not valid in shell variables they have to be normalized to a character that is allowed. Since there is only one valid character for a shell variable that isn't valid in a ZFS pool name, all 3 of the characters that can be in a pool name that can't be in a shell variable get mapped to that one character.

It's a rare but possible edge case, that you have two pools with names like this.-collides and this:.collides. They will both end up as this__collides and it will not be possible to set different scrub intervals for them.

 

[sjieke]

You could normalize using an escape character in combination with the ASCII value of the character to escape. That way you can always revert back to the original string. If the escape character can be in the original string, you need to normalize it also.
Take your example:
this.-collides becomes this_46__45_collides
this:.collides becomes this_58__46_collides
this._collides becomes this_46__95_collides

 

[William Grzybowski]

Ok,

what if i have a "this.-collides" pool AND a "this_46__45_collides" pool ?

 

[sjieke]

That's easy. You always need to escape the escape character also.

So the algorithm I use is the following:
* choose an escape character => in the example we choose '_'
* for each character that is not allowed you replace it with the ASCII value and add the escape character as prefix and postfix
our characters to escape in the example are
1. '.' becomes '_46_'
2. ':' becomes '_58_'
3. '-' becomes '_45_'
* the escape character also needs to be escaped so
'_' becomes '_95_'

If we use this algorithm on your last example we get
* "this.-collides" becomes "this_46__45_collides"
* "this_46__45_collides" becomes "this_95_46_95__95_45_95_collides"

Now, if you want to parse them back, you search for the escape character '_' followed by 1 or more digits and then again the escape character.

* "this_46__45_collides" resulting in "this.:collides"
* "this_95_46_95__95_45_95_collides" resulting in this_46__45_collides

I haven't found any situation yet were this algorithm doesn't work. If the algorithm isn't entirely clear yet I can post the code in C# by tomorrow. I implemented this recently in a Windows project for a client. Some code will probably be easier to understand, as I
find it hard to explain it in written text.

 

[sjieke]

Sometimes code says more than a thousand words :)

The code to escape the provided charachters from an input string. We use '_' as the special escape character.

Code:

public static string EscapeChars(string input, char[] charsToEscape)
{
   Debug.Assert(!charsToEscape.Contains('_'), "The characters to escape cannot contain the char '_' as we already use it as our escape character");
   //First escape our escape char
   input = input.Replace("_", "%" + (int)'_' + "_");
   foreach (char c in charsToEscape)
   {
      input = input.Replace(c.ToString(), "_" + (int)c + "_");
   }
   return input;
}

The code to revert escaped characters back to there original

Code:

public static string RevertEscapedChars(string input)
{
   string output = "";
   string escaped = "";
   bool detecting = false;
   foreach (char c in input)
   {
      if (c == '_' && detecting)
      {
         //the second '_', so we finished detecting an escaped character
         //we need to convert the found ASCII value back to the character and add it to our output
         output += (char)int.Parse(escaped);
         //we aren't detecting anymore, so reset the flags and escaped sequence 
         detecting = false;
         escaped = "";
      }
      else if (c == '_' && !detecting)
      {
         //the first '_' so we start detecting an escaped character
         detecting = true;
      }
      else if (char.IsDigit(c) && detecting)
      {
         //if we are in detecting state (we found a '_') and the next character is a digit we add it to our escaped sequence
         escaped += c;
      }
      else if (detecting)
      {
         //if we are in detecting state and encounter a character that is not a digit before we encounter the closing '_'
         //than we don't have an escaped character and just add what we found to the output
         output += escaped;
         //we also need to reset the flag and escaped sequence
         escaped = "";
         detecting = false;
      }
      else
      {
         //we aren't detecting an escaped character so we just add it to the output
         output += c;
      }
   }
   return output;
}

I hope this makes it more clear. If you want I could try to convert it to another language, but this I had already lying around :)

If you do find a situation where this doesn't work and can result in collisions, please let me know. It isn't because I couldn't find any that there aren't any edge cases I missed :)

 

[William Grzybowski]

Yes, i think you are missing something...

Does not matter to what you convert :- there will _ALWAYS_ be a possible zpool name to collide, because we are _stuck_ with shell script, which does not allow anything else besides "_" present in a variable name, which is what freebsd uses for configuration of periodic

Using "_" is a common case, you don't want to obligate the system admin to use an unusual escape sequence for it...
It is an edge case anyway

 

[david31262]

Is it best to avoid special characters in pool names ? Would that be hypens? Is Uncerscore OK ?

Thanks all.

I'm very new to Freenas, so haven't been messing with the betas. I'm not sure of the best way to upgrade w/o losing the pool (there's probably a thread here somewhere -> I'll go look (upgrade vs clean install).

And, thanks for the rapid response. Looks like the new free nas is a winner. I couldn't use the last version because I have a Sans Digital tower that requires Multi-LUN SATA support. Time for a donation.

 

[jpaetzel]

No, we are discussing very unusual edge cases, where pools have names that differ only in their use of a few special characters.

 

[david31262]

Is this fix ready for (beta) prime time ?
Would it be in beta 3 ?

 

[sjieke]

Yes, i think you are missing something...

Does not matter to what you convert :- there will _ALWAYS_ be a possible zpool name to collide, because we are _stuck_ with shell script, which does not allow anything else besides "_" present in a variable name, which is what freebsd uses for configuration of periodic

Using "_" is a common case, you don't want to obligate the system admin to use an unusual escape sequence for it...
It is an edge case anyway

The user doesn't need to know of the escape sequence. He has an input field, the system converts it for internal use and as it is displayed back to the user the escape sequence is reverted back. So the user doesn't notice anything.

I don't know anything about FreeNAS and the systems used behind it. Shell scripts are indeed limiting the possibilities. So I agree with you that in 'this situation' you can always have a colliding name.

It is indeed an edge cage and probably not worth all the hassle. I just wanted to make clear that (according to me) you can always escape special characters and revert them back without any collisions.

 

[William Grzybowski]

Actually the user needs to know about it, since periodic.conf is an user interface and people will eventually use it, in freenas or freebsd itself (remember this is a limitation of freebsd and the workaround was pushed back to the freebsd HEAD)

But I get your point, you're right about escaping, unfortunately is not doable in this case unless we drastically change how things work/behave, which is not worth/desirable.

thank you