Hey all! I had a disk that was showing some offline uncorrectable sectors. I ran the long smartctl test and it failed, so I RMA'd it and have replaced it in my pool. I used the GUI, and followed the instructions for my version (9.2.1.3 - I want to upgrade, but I want a healthy pool first).
Things started off reasonably well - scanning at around 100-120 MB/s, looking at about 31 hours to finish the resilver. However, in the ensuing hours, it has slowed to a near standstill (scanning a couple MB/s, looking at 130+ hours to complete).
So where should I start looking to troubleshoot this?
Here's the systat -vm output. You can see that each disk is reading <1MB/s.
Any pointers?
Things started off reasonably well - scanning at around 100-120 MB/s, looking at about 31 hours to finish the resilver. However, in the ensuing hours, it has slowed to a near standstill (scanning a couple MB/s, looking at 130+ hours to complete).
So where should I start looking to troubleshoot this?
Code:
[root@delta] ~# zpool status pool: tank state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Jul 15 16:21:21 2014 474G scanned out of 12.6T at 26.9M/s, 130h57m to go 78.8G resilvered, 3.68% done config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/f645c671-dfe0-11e2-b96b-50465d6afb74 ONLINE 0 0 0 gptid/f6959517-dfe0-11e2-b96b-50465d6afb74 ONLINE 0 0 0 gptid/f6e96295-dfe0-11e2-b96b-50465d6afb74 ONLINE 0 0 0 replacing-3 OFFLINE 0 0 0 14595285845210114515 OFFLINE 0 0 0 was /dev/gptid/f7547512-dfe0-11e2-b96b-50465d6afb74 gptid/95942e35-0c5d-11e4-934c-50465d6afb74 ONLINE 0 0 0 (resilvering) gptid/f7c2859e-dfe0-11e2-b96b-50465d6afb74 ONLINE 0 0 0 gptid/f8333b58-dfe0-11e2-b96b-50465d6afb74 ONLINE 0 0 0 cache gptid/7cddd7fe-bb49-11e3-b237-50465d6afb74 ONLINE 0 0 0 errors: No known data errors [root@delta] ~#
Here's the systat -vm output. You can see that each disk is reading <1MB/s.
Code:
2 users Load 0.03 0.07 0.07 Jul 15 21:28 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 302592 29992 2808616 48016 943592 count All 6187856 32516 1076965k 84876 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 1549 total 1 76 6202 80 1479 1549 1031 zfod attimer0 0 ozfod 35 ata1 15 0.4%Sys 0.0%Intr 0.0%User 0.0%Nice 99.6%Idle %ozfod 2 ohci0 ohci | | | | | | | | | | | daefr ehci0 17 prcfr 4 atapci0+++ dtbuf 2 totfr 432 em0 20 Namei Name-cache Dir-cache 202416 desvn react 67 ahci1 22 Calls hits % hits % 17586 numvn pdwak 1009 hpet0:t0 4431 4430 100 7995 frevn pdpgs intrn Disks md0 md1 md2 ada0 ada1 ada2 ada3 6369864 wire KB/t 0.00 0.00 0.00 75.61 35.57 35.24 35.70 298812 act tps 0 0 3 4 19 19 19 228628 inact MB/s 0.00 0.00 0.00 0.28 0.65 0.65 0.65 cache %busy 0 0 0 0 13 13 13 943592 free 155396 buf
Any pointers?