SOLVED Resilvering reading all disks in multi VDEV Mirror set

[sretalla]

I was wondering if anybody can help to explain the results I'm seeing when resilvering a disk in only one of the mirrored VDEVs in my Zpool.

If I understand Mirrors properly, resilvering one should only need to read the other drive in that same mirrored VDEV.

What I am observing is that the drives in the other 2 VDEVs are also reading at the same data rate as the one in the "expected" VDEV.

I should explain that I'm building up a new zpool and have deliberately taken one drive in each mirrored VDEV offline during the initial seed copy as the copy speed seemed to be unreasonably slow to me initially and it improved about 30% when I essentially moved back to a stripe of 3 drives (3 single-disk VDEVs) to take the file copy. Now the copy is over, I am putting back the first drive in Mirror-0, but I see in the reporting that the other Mirrors are also reading at exactly the same speed.

Can anybody confirm that they have seen this? (I guess in any case of resilvering in a grouping of mirrored VDEVs)

Can anyone explain why it's happening? (how can it be useful or "by-design" to read data that has nothing to do with the target of the resilver?)

What I see:
ZpoolX
Mirror-0 Disk1 reading at about 13 MB/s
Mirror-0 Disk2 writing at about 13 MB/s

Mirror-1 Disk1 reading at about 13 MB/s
Mirror-1 Disk2 (still offline, but I assume would also be reading)

Mirror-2 Disk1 reading at about 13 MB/s
Mirror-2 Disk2 (still offline, but I assume would also be reading)


What I expect:
ZpoolX
Mirror-0 Disk1 reading at about 13 MB/s
Mirror-0 Disk2 writing at about 13 MB/s

Mirror-1 Disk1 online - idle
Mirror-1 Disk2 offline

Mirror-2 Disk1 online - idle
Mirror-2 Disk2 offline


I don't need to discuss the relatively slow reading/writing speeds here, I'll address that separately myself unless somebody can see how it would be related.

CASE: Fractal Node 804
MB: ASUS x-99M WS
CPU: Xeon E5-2620v4 + Corsair H60 Water Cooler
RAM: CRUCIAL 64GB DDR4-2133 ECC Registered
HDD: WD RED 6TBx8
OS: 4x Sandisk Ultra Flair 32GB (mirror)
PSU: Corsair RM1000
FreeNAS-11.1-U1
3xFractal R3 120mm - 2 Front, 1 Rear
1xCorsair H60 CPU Radiator - Front

 

[MrToddsFriends]

Please show us the output of zpool status -v in code tags.

I should explain that I'm building up a new zpool and have deliberately taken one drive in each mirrored VDEV offline during the initial seed copy as the copy speed seemed to be unreasonably slow to me initially and it improved about 30% when I essentially moved back to a stripe of 3 drives (3 single-disk VDEVs) to take the file copy.

Sounds weird to me. Did you perform a disk burnin before creating the pool?
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[sretalla]

Pretty much exactly what I already indicated above, but here you go:

Code:

  pool: ZpoolX
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Feb 26 18:31:03 2018
   3.65T scanned at 48.8M/s, 2.72T issued at 36.4M/s, 6.73T total
   949G resilvered, 40.47% done, 1 days 08:02:44 to go
config:

   NAME											STATE	 READ WRITE CKSUM
   ZpoolX										  DEGRADED	 0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/e9d76d74-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   gptid/eb7f543f-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0  (resilvering)
	 mirror-1									  DEGRADED	 0	 0	 0
	   gptid/ed8c012c-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   11554275818972966987						OFFLINE	  0	 0	 0  was /dev/gptid/ef7be38c-195f-11e8-90e5-38d547c91e29
	 mirror-2									  DEGRADED	 0	 0	 0
	   gptid/f18fda99-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   8354969272770818364						 OFFLINE	  0	 0	 0  was /dev/gptid/f38437b4-195f-11e8-90e5-38d547c91e29

No burn-in as I was planning to use the copy as a stress test (keeping the old pool alive until I'm satisfied)... this is only for home use in any case.

Reports of disk activity as you can see even from the thumbnail below. (first disk resilvering, 3 others reading, 2 offline)

 

[rs225]

I think this is normal behavior. A resilver and scrub use the same code.

Even though you have a mirror, ZFS does not just image one drive to the other. Using the scrub code, it walks the entire metadata tree and then replicates blocks it finds in that mirror to the other disk. It would seem like a logical optimization to not bother reading the other blocks, but as I said, a resilver is a scrub. So, you get two for one, and put your next scrub farther into the future.

 

[sretalla]

A resilver and scrub use the same code.

Perfect explanation! Thanks for the information, it makes sense (at least as to why it happens if not actually that it happens as per your comment).

 

[sretalla]

As a PS to the thread, some more unexpected outcomes when I onlined the other 2 disks:


This time, the first VDEV remains inactive while the other 2 VDEVS are resilvering (at about 9MB/s... slower than when I did the first one).

For the pedantic:

Code:

# zpool status -v ZpoolX
  pool: ZpoolX
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Feb 28 23:15:53 2018
   4.97T scanned at 80.9M/s, 2.76T issued at 44.9M/s, 6.79T total
   1.10T resilvered, 40.63% done, 1 days 02:09:16 to go
config:

   NAME											STATE	 READ WRITE CKSUM
   ZpoolX										  ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/e9d76d74-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   gptid/eb7f543f-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	 mirror-1									  ONLINE	   0	 0	 0
	   gptid/ed8c012c-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   gptid/ef7be38c-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0  (resilvering)
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/f18fda99-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0
	   gptid/f38437b4-195f-11e8-90e5-38d547c91e29  ONLINE	   0	 0	 0  (resilvering)

errors: No known data errors

No need for further explanation, but interesting to note the behavior for those interested in the topic.

 

[Chris Moore]

Although a resilver uses the same code as a scrub for part of the work, it does not count as a scrub, and a pool that has had a disk resilvered does not show as having had a scrub. To test this, for the sake of my own curiosity, I setup a new pool, and found that after doing a resilver, the last scrub date was undefined, because there had not yet been a scrub, even after replacing a disk. The resilver took about half an hour because of the amount of test data I had loaded onto the pool. When I did run a scrub, it completed in under a minute and sowed a mechanically unattainable speed, especially at the end, so that must have something to do with caching of the relatively small amount of test data that was in the pool. Still, I thought it was an interesting thing to share.