Scrub Performance Tuning

[Terry Wilson]

AKA How to speed up the process of tuning of your Freenas box. This one is for 9.10 - FreeNAS-9.10.2-U2

Before I begin let me get the standard disclaimers out of the way, Try at your own risk, backup first, this may cause the moon to fall from the sky and make Trump and Hillary besties....

Credit needs to be given to Shawn Webb the purported creator of this code.


So there are 4 lines of code that we want to change to potentially decrease the amount of time it takes to scrub and or resilver. In my case on a 6 disc vdev running z2 in my pool (before this tuning) it would take 28 hours to scrub, after these settings it went down to 15 hours, that's almost 2x faster !

So while you can do this from the command line its frowned upon by the forum, the developers, etc.. so I am just going to cover this approach from the gui.

What we need to do is logon, head over to system and then choose Tunables:

Next we are going to choose Add Tunable

Note the Add Tunable box that pops up is where we are going go enter the following variables one at a time

vfs.zfs.scrub_delay=0
Note how I removed the = and told the Tunable diaglog box it was a Sysctl (as there are 3 values in the dropdown to choose from, were are just interested in Sysctl)

Click OK to add,

Now once again click Add Tunable and then put this one in:
vfs.zfs.top_maxinflight=128

Click OK to add, then once again click Add Tunable and then put this one in:
vfs.zfs.resilver_min_time_ms=5000

And Finally Click OK to add, then once again click Add Tunable and then put this one in:
vfs.zfs.resilver_delay=0

Now you are done, these values will be saved thru a reboot and you should see significant savings in time when you do your next Scrub!

Thanks and enjoy...


Terry

 

[joeschmuck]

It might be good to describe what is being changed.

The Default Values for the above parameters are:
vfs.zfs.scrub_delay=4
vfs.zfs.top_maxinflight=32
vfs.zfs.resilver_min_time_ms=3000
vfs.zfs.resilver_delay=2

A description of each tunable is listed below: (from the FreeBSD Handbook)

vfs.zfs.scrub_delay - Number of ticks to delay between each I/O during a scrub. To ensure that a scrub does not interfere with the normal operation of the pool, if any other I/O is happening the scrub will delay between each command. This value controls the limit on the total IOPS (I/Os Per Second) generated by the scrub. The granularity of the setting is determined by the value of kern.hz which defaults to 1000 ticks per second. This setting may be changed, resulting in a different effective IOPS limit. The default value is 4, resulting in a limit of: 1000 ticks/sec / 4 = 250 IOPS. Using a value of 20 would give a limit of: 1000 ticks/sec / 20 = 50 IOPS. The speed of scrub is only limited when there has been recent activity on the pool, as determined by vfs.zfs.scan_idle. This value can be adjusted at any time with sysctl(8).

vfs.zfs.top_maxinflight - Maxmimum number of outstanding I/Os per top-level vdev. Limits the depth of the command queue to prevent high latency. The limit is per top-level vdev, meaning the limit applies to each mirror, RAID-Z, or other vdev independently. This value can be adjusted at any time with sysctl(8).

vfs.zfs.resilver_min_time_ms -Minimum time allocated to the resilver process.

vfs.zfs.resilver_delay - Number of milliseconds of delay inserted between each I/O during a resilver. To ensure that a resilver does not interfere with the normal operation of the pool, if any other I/O is happening the resilver will delay between each command. This value controls the limit of total IOPS (I/Os Per Second) generated by the resilver. The granularity of the setting is determined by the value of kern.hz which defaults to 1000 ticks per second. This setting may be changed, resulting in a different effective IOPS limit. The default value is 2, resulting in a limit of: 1000 ticks/sec / 2 = 500 IOPS. Returning the pool to an Online state may be more important if another device failing could Fault the pool, causing data loss. A value of 0 will give the resilver operation the same priority as other operations, speeding the healing process. The speed of resilver is only limited when there has been other recent activity on the pool, as determined by vfs.zfs.scan_idle. This value can be adjusted at any time with sysctl(8).

So by making these changes as indicated you will be reducing the responsiveness of the FreeNAS server to outside requests during a Scrub or Resilver operation. And this is not a bad change if you have to resilver a large hard drive back into your system. You could also adjust those settings to half the differences and see how that affects things.

 

[frush]

It appears that for FreeNAS 11.2 the following defaults are true:

vfs.zfs.resilver_min_time_ms: 9000
vfs.zfs.resilver_delay: 0

 

[jdabb]

can this be tuned while re silvering is in process?