Scrubbing a very large pool - Strange system load pattern

[jallis]

I have a very large pool of 63 * 8TB disks across 7 raidz2 VDEVs. The server has 384GB RAM and 24 CPU cores.

When running scrub on this pool I see a very strange system load where the system is completely idle for about two hours, and then running at 100% capacity for about two hours, and then repeating this on/off cycle. See attached screenshot.

Is this normal system load for a scrub? Why is it doing this? Will adding *even more* memory help anything at all?

When the system load is at 100% the system is barely usable, can I use the "Resilver Priority" setting to avoid 100% system load during work hours?

 

[Ericloewe]

What version of FreeNAS?

 

[jallis]

What version of FreeNAS?

FreeNAS 11.1-RELEASE

 

[Ericloewe]

Okay, first of all, 11.1-U0 has several extremely nasty bugs. Definitely update to 11.1-U1 or later ASAP.

I believe you may be hitting one of them in a not-as-serious condition. Some people's pools would cause a near deadlock when scrubbing and effectively halt the system. It seems fairly likely that in your case the deadlock gets solved somehow (your system is rather grunty, brute force might be enough to overcome it).

 

[jallis]

Okay, first of all, 11.1-U0 has several extremely nasty bugs. Definitely update to 11.1-U1 or later ASAP.

I believe you may be hitting one of them in a not-as-serious condition. Some people's pools would cause a near deadlock when scrubbing and effectively halt the system. It seems fairly likely that in your case the deadlock gets solved somehow (your system is rather grunty, brute force might be enough to overcome it).

Thank you for feedback. I'll update as soon as this scrub is finished :)

Are you able to answer my question regarding usage of "Resilver Priority" setting also?

 

[Ericloewe]

I expect that it will be an academic question after the update. Performance limitations for your applications would be caused by IOPS limitations, in which case you can try the resilver priority option. It's fairly recent, so there's not much data around it yet.