Endless scrub on pool !

[blckhm]

Hi everone,

Our freenas system contains ~8TB image data on 4x 12 TB WD Gold drives which populated as striped 2way mirrors.

Our problem is, when we started to scrub the pool, It will probably take 11 days and It is not acceptable.
We checked also there was not any running smart test during scrub in progress.

status of the pool after 12 hours

Code:

 pool: tank2
 state: ONLINE
  scan: scrub in progress since Mon Apr  1 00:51:07 2019
    8.78G scanned at 254K/s, 8.77G issued at 254K/s, 8.31T total
    0 repaired, 0.10% done, no estimated completion time 
config:

    NAME                                            STATE     READ WRITE CKSUM
    tank2                                           ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/6ede3773-d150-11e8-8c16-000c2937eabd  ONLINE       0     0     0
        gptid/7073b925-d150-11e8-8c16-000c2937eabd  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        gptid/71faa011-d150-11e8-8c16-000c2937eabd  ONLINE       0     0     0
        gptid/7364fe0a-d150-11e8-8c16-000c2937eabd  ONLINE       0     0     0

errors: No known data errors

What's your suggestion?

 

[dlavigne]

Which FreeNAS version?

 

[blckhm]

Which FreeNAS version?

Code:

Build    FreeNAS-11.2-U2.1
Platform    Intel(R) Xeon(R) CPU E5-2630L v3 @ 1.80GHz
Memory    53207MB
System Time    Mon, 1 Apr 2019 13:44:45 +0300
Uptime    1:44PM up 5 days, 5:52, 0 users
Load Average    0.20, 0.24, 0.23

 

[SweetAndLow]

You only waited 2h to complain. What makes you think it is "endless". How about you wait a little and see how things progress.

My 109TB pool that is 50% full takes 8h-12h to scrub

 

[blckhm]

You only waited 2h to complain. What makes you think it is "endless". How about you wait a little and see how things progress.

My 109TB pool that is 50% full takes 8h-12h to scrub

No, I did not wait only 2 hours. Where did you understand that?

Like I said before, I started to scrub at ~00:51 AM and checked the status of zpool at ~ 11:00 AM. Almost 12 hours

 

[nemisisak]

Hi bkckhm,

Yeah something has gone wrong there, the scrub speed is crawling which you wouldn't expect for your setup. If one of the drives is bad this may be slowing down the process.

I would be inclined to stop the scrub and run a long smart test.

Is this a new or old system? Did you burn in the drives before creating your pool?

The scrub can sometimes pick up speed, but its unlikely after such a period of time at low speeds. Another reason for slow scrub timewould be if the server is under load from users?

 

[Ericloewe]

The pool is somewhat full, but not necessarily "crawling scrub" slow.

Another reason for slow scrub timewould be if the server is under load from users?

I think we're beyond that.

The most typical scenario that explains this is a failing disk, so SMART data for all disks would be valuable.

 

[Apollo]

I would use Netdata and look at each disk "busy" status and crosscheck against "throughput" per drive.
I had a failed drive running at about 10-20MB/s but "busy" was nearly 100%.
If that is the case you most likely have a failing drive. As soon as the drive was replaced, scrubs would return to normal speed.