Replacing a mirrored vdev instead of individual disks?

[Maxlink]

Hi all,

I am trying to replace a bunch of mirror vdev's.
I have the option to connect extra disks to the system, so no need to offline disks first before replacing them.
This maintains the redundancy too. Even though the system is not under load while replacing disks.

However I can only find the option to replace a single disk, not the whole mirror.
Am I overlooking something?

My thought was that replacing a mirror as a whole with a new mirror, would be faster than replacing both drives on their own.
Because now both replace actions start reading the same data twice. Maybe I got this part wrong?

 

[sretalla]

You didn't miss anything... you can replace more than one disk simultaneously though (not waiting for each one you start to finish, to do them one at a time... but of course with a 2-way mirror, you can't just replace both disks simultaneously as that would leave nothing to resilver from).

As you say, if you have the available slots, you can certainly do it before detaching the older disks from the mirror. (effectively, what you are able to do is just add 2 drives to the mirror... making a 4-way mirror... then detach the old 2 from it when the resilvering is done).

 

[Maxlink]

You didn't miss anything... you can replace more than one disk simultaneously though (not waiting for each one you start to finish, to do them one at a time... but of course with a 2-way mirror, you can't just replace both disks simultaneously as that would leave nothing to resilver from).

As you say, if you have the available slots, you can certainly do it before detaching the older disks from the mirror. (effectively, what you are able to do is just add 2 drives to the mirror... making a 4-way mirror... then detach the old 2 from it when the resilvering is done).

Thanks for your reply.

Just out of curiosity and performance wise: wouldn't it be faster to first make a new mirror vdev and then replace the old mirror vdev with the new mirror vdev?

Somehow I have the idea that it would require less reads to do it this way versus replacing both disks in a mirror simultaneously.

I have that idea because I figure that replacing both disks simultaneously requires truenas to read 100% from disk A and write 100% to the new disk A and simultaneously read 100% from disk B and write 100% to the new disk B.
Where as replacing a mirror as a whole could potentially read 50% from disk A and read the other 50% from disk B and write both halves as 100% to the new mirror. Therefore reducing reads on the existing mirror?
Maybe I am fully mistaken here. I also don't see an option to create a new mirror vdev and use it to replace an existing mirror vdev.

 

[jgreco]

ZFS doesn't let you micromanage how it handles resilvers in this manner. You can attach your two new drives to the existing mirror and it will resilver both of them, presumably getting benefits from ARC caching. Then you disconnect the two old drives when the new ones are done. Being overly concerned about the exact manner in which this happens is pointless, since it's going to do what it wants to do, and the code is written to handle many different resilvering cases, not just optimize for this one unusual one you've presented.

 

[Patrick M. Hausen]

@jgreco ever worked with Amanda? The backup software from Maryland?

Same thing: you cannot micromanage your father-son-whatever tapes. You tell Amanda how many tapes you have, how many days you want to keep, how many full backups in a full cycle of all tapes should be done, and that's it. Amanda will figure it out.

Great product, never let us down even once. All in the age before ZFS and snapshots and send|receive, of course.

 

[jgreco]

Yes! Amanda! Ran it in the early 90's with a DAT autochanger. It rocked.

 

[Maxlink]

I think I see a valid point here: not replacing both disks but adding two new disks to the existing vdev, then removing the initial two disks.

Is that indeed a big difference?

 

[sretalla]

Is that indeed a big difference?

Yes, It should be much faster.

 

[Maxlink]

I did a test using two spare disks, I have added the two disks (extend) to the existing mirror vdev.
It started resilvering. I noticed reads from both existing drives and writes to the two new drives. However, after finishing the resilvering it started again with resilvering just a single (3th) disk?
This is quite unexpected.

I have added a screenshot from the read/write speeds.
The first two drives are the original two disks in the mirror vdev that is being extended.
It finished resilvering and then started resilvering the 3th drive.
Can someone explain what is going on here? I did not expect this to happen.