How do you change the Scrub & Resilvering Priority?

[joeschmuck]

So after reading something today I started thinking about the scrub & resilvering priority. I possibly think changing the values could help out some people with certain speed requirements. I was able to find out how to speed up scrubbing and resilvering but not able to find out how to slow them down. One might think just do the opposite of the speed up settings but I'm not that brave or stupid to thing that is true.

Here is the link to how to speed it up and I am not saying anyone should do this, it's only to support my question. https://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/

So for my own use I'd like to know if I can slow down a scrub. It's not so my system can process more I/O while a scrub is going on but rather I'm looking to reduce the heat generated from the hard drives by maybe a few degrees (if I'm lucky). During a scrub two of my drives hit 44C, the rest are 40 or less. Normal operating temps are about 8C less.

I have no idea if changing the priority of a scrub when nothing is accessing the NAS would do anything at all, meaning since there is no competition due to I/O operations, the scrub would get the job anyway. So if a delay or pause isn't forced then I don't see things working my way.

In the link above there is a sysctl called zfs_scrub_delay in which the default value is 4. I understand a value of 0 (zero) is a higher priority. I don't believe this will solve my problem by setting it to say a value of 8. I believe there must be a time limit associated to limit the minimum and maximum time allowed for the process to take control. Again even if this is true just because there are no competing processes might be a reason this wouldn't work.

Any help would be appreciated or a pointer to another link that discusses this.

As for resilvering priority, for someone who really wants to resilver fast like if you have a RAIDZ1 (not recommended now days for data safety) you might want resilvering to occur as the highest priority over I/O. Again I'm not saying anyone should use the example in the link above, it's there just as a reference for my question but if someone knew what settings would work in FreeNAS 9.2.x, that might be helpful. It might even be nice if this became an on-the-fly option in the GUI.

 

[jgreco]

Not in a position to look at the code right now, but I thought this had been discussed previously. There are several tunables that might be able to affect this, the one I think I'd look at is whatever the FreeBSD equivalent of zfs_scrub_limit is ...

 

[joeschmuck]

@jgreco, Thanks for that other value to search for.

I did a search of the FreeNAS forum last night and didn't find anything for "scrub priority" or "zfs_scrub_delay". Doing a Google search for these items hasn't come up with anything promising with respect to just slowing down the speed of a scrub. Did the same thing this morning using "zfs_scrub_limit" and same results. Everything is about priority and only slows the scrub down or speeds it up with respect to I/O operations. So none of this meets my objective unfortunately. My current scrub takes about 3hours and 45 minutes (430M/s), if I could stretch that out to 6 hours I'd be fine with that to see if that has any impact on the temperature.

My drives are all cooled passively, absolutely no forced air flow. I know one way I could improve the air flow but it means cutting out one side of the drive cage and replacing it with a thin solid metal strip and thus will improve the air flow. Right now the drive cage almost covers the entire length of the drives so this change will help out. I guess the other solution is to use my nice new Cooler Master HAF 912 case but I'll try the cutting of the drive cage first.

-Mark

 

[jgreco]

vfs.zfs.scrub_limit is what i was thinking of...


https://forums.freebsd.org/viewtopic.php?&t=31628

 

[joeschmuck]

I saw that site this morning as well.

From what I can tell this value is defined like this:
zfs_scrub_limit = maximum number of scrub I/O per leaf vdev

I have no idea what "per leaf vdev" means but I think I will go ahead and first try this on a VM machine. If it seems to slow down the scrub process I will try it on my test machine. If that works then the real machine. I'll post my results. With the holidays this could take a while. Too much family fun going on.

 

[cyberjock]

If you are trying to prevent the hard drives from heating up by 4C you won't find tunables to do what you want. :(

You can change the priority of scrubs and you can change tunables that can kill scrub performance by killing pool performance. But I'm thinking that's not your ultimate goal.

At the end of the day, an idle pool is going to scrub as fast as it can. I/Os add heat and unless you can tell ZFS to force itself to be idle for several seconds at a time you aren't going to appreciably affect temperature.

I don't know if you remember but I had my own eat problems in April/May of this year, and there is no easy fix except to add more cooling or deliberately kill pool performance(and thereby affecting scrub performance).

 

[joeschmuck]

Yea, killing pool performance is not the right path for me. I'll work on the cooling and hopefully I can fix it passively.

 

[Dusan]

FreeNAS 9.2.0:
[PANEL][root@freenas] ~# sysctl vfs.zfs.scrub_limit
sysctl: unknown oid 'vfs.zfs.scrub_limit'[/PANEL]
Looking at the code I found vfs.zfs.scrub_delay (Number of ticks to delay scrub), but it seems to insert the delay only when there is other I/O:

Code:

TUNABLE_INT("vfs.zfs.scrub_delay", &zfs_scrub_delay);
SYSCTL_UINT(_vfs_zfs, OID_AUTO, scrub_delay, CTLFLAG_RW,
    &zfs_scrub_delay, 0, "Number of ticks to delay scrub");
...
    if (scn->scn_phys.scn_func == POOL_SCAN_SCRUB) {
        zio_flags |= ZIO_FLAG_SCRUB;
        needs_io = B_TRUE;
        scan_delay = zfs_scrub_delay;
    }
...
        /*
        * If we're seeing recent (zfs_scan_idle) "important" I/Os
        * then throttle our workload to limit the impact of a scan.
        */
        if (ddi_get_lbolt64() - spa->spa_last_io <= zfs_scan_idle)
            delay(MAX((int)scan_delay, 0));

You could probably set vfs.zfs.scan_idle to a larger value to force the delay condition to be true more often. However, I don't think that would help with a completely idle pool.

Another interesting sysctl is vfs.zfs.top_maxinflight (Maximum I/Os per top-level vdev). It doesn't have scrub anywhere in its name, but it's only used by the resilver/scrub code, so it should not impact anything else. The default is 32, setting it to 1 slows a scrub in an idle VM from ~120MB/s to ~30MB/s.

Code:

TUNABLE_INT("vfs.zfs.top_maxinflight", &zfs_top_maxinflight);
SYSCTL_UINT(_vfs_zfs, OID_AUTO, top_maxinflight, CTLFLAG_RW,
    &zfs_top_maxinflight, 0, "Maximum I/Os per top-level vdev");
...
        uint64_t maxinflight = rvd->vdev_children *
            MAX(zfs_top_maxinflight, 1);
...
        while (spa->spa_scrub_inflight >= maxinflight)
            cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock);

 

[cyberjock]

AFAIK maxinflight kills pool performance. joe wants to slow down scrub performance while leaving the pool at full performance.

 

[Dusan]

AFAIK maxinflight kills pool performance. joe wants to slow down scrub performance while leaving the pool at full performance.

Are you sure? I just searched the entire ZFS code again and top_maxinflight is referenced only in function dsl_scan_scrub_cb (dsl_scan.c). No other ZFS code uses that sysctl. A quick test with dd confirms that pool performance doesn't change when I wiggle that knob. On the other hand the scrub performance goes down/up when I decrease/increase the value.

 

[cyberjock]

Bah, I can't find my notes on maxinflight anymore. :(

But, I don't think it significantly affect I/Os in a way that is appropriate for this situation. The problem is if you want to see temps go down it means there MUST be idle time on the disks. That also means that for the temp curves I saw before(15 mins to go from idle temp to full I/O temp) means that if you wanted to significantly change the final temp you'd need something like 1 second of I/O followed by 5-10 seconds of idle time. You'll never get that with tunables as far as I know. You can cripple the throughput. You can even cripple the I/O numbers themselves by doing things such as disabling the NCQ. But this isn't a numbers game. This is an "I want idle time" game(or less accurately "I want to see that HDD LED go off for long periods of time during a scrub"). And the question is "how do you make ZFS go completely idle for seconds at a time... during a scrub?" I just don't think there's a knob for that. The scrub can be momentarily deferred by other load. It can even be slowed down in terms of throughput and total I/Os per second. But you aren't truly making the disk go completely idle for seconds at a time.

Most people want to do things to optimize I/Os, prioritize reads/writes over scrubs, and other such things. Most people don't want to cripple a scrub to the point that it runs at 5%(0.5seconds of user data read/writes, 0.5seconds of scrub, 9 seconds of idle time). So why make a knob for something like that? I just don't think there is one that is going to deliberately force lots of idle time on your pool.

 

[joeschmuck]

Well let me ask your opinion(s) of this... If my drives typically run at a high of 37 under normal use and hits a high of 44 summer or 42 winter only during a scrub, and I run a scrub once a month which lasts 3 to 4 hours (dropped to 3 hours after deleting some old data), do you think it would harm the life of the drives? I think not but if I can still get them to run a little cooler then it won't hurt either.

BTW, thank you for the participation in this thread.

 

[jgreco]

It is like asking if it is better to drive 55, 60, 65, or 70 in a 55 mile per hour zone. Some people will go years doing 70 without a ticket. Some will be pulled over for momentarily doing 57 because an officer was having a bad day.

The SMART money's on fixing the heat problem. Heh

 

[cyberjock]

When I had heat problems this summer I was almost in your boat. Drives would be 37C and a scrub would put them over 40C with some fan speed settings. It probably won't do any damage to go over the 40C recommendation just for scrubs that last a few hours. But I also viewed it as being far too close to for comfort. All I needed was a fan to decide to stop working and I'd be in deep trouble. I also set my warning temp to 39C, so that meant I could expect an email every scrub. :(

Any particular reason why you are avoiding adding fans, increasing their speed, or getting higher flow rate fans?

 

[joeschmuck]

The case I am using is very old (almost like me) and it doesn't have the provisional space for fans in the hard drive areas. Imagine this: An old sturdy half height tower ATX case from 10+ years ago. The old style mounting of the drives is two wide metal strips that almost are the length of a 3.5" drive and only had holes that accommodate mounting the drives, nothing for air flow. I was able to space apart each drive using every other drive mounting point and I also had to use the upper 5.25" drive bay using the 3.5" mount adapters. Again, no possible way to mount fans. And I have taken off both side panels and using a small support table sandwiched the computer between a pair of high flow (cheap) household air filters which allows for very good thermal flow. So my only real problem is the drive mounting just restricts thermal air flow since it's so damn wide. If I laid the computer on it's face then I think I'd have better thermal flow for the drives but it would be too unstable unless I created maybe a wooden mount/foot. Well I guess I could do one more thing... Cut out the front of the metal case and build up the area in front of the 5.25" bays to accept a few 80mm fans to blow some air over the drives. Wish I had a metal workshop to use, I could then just build a custom case with good open thermal design. I've even thought about mounting the drives on the outside of the case cover (right side of course) but haven't done that just yet. I'd like to devise a way to remove a drive from the outside of the case for when failure strikes.

You know I started my FreeNAS machine on the cheap, hence the case. It was old yet sturdy and worked great in the old days. It holds all my parts and it works well.

Did I answer the question? No place to mount the extra fans is the short answer.