Anyone done multiple drive replacement on Mirrored Volume/Pool?

[Mirfster]

Contemplating increasing an iSCSI Pool/Volume consisting of 5x Mirror vDevs, 1x Hot Spare and 1x Cold Spare.

Was thinking that I would:

• Replace the Hot and Cold Spares first
  
• Replace one drive in each Mirror all at the same time; so basically "replace" 5 drives at once but only one drive out of each mirror
  
• Let the resilver complete
  
• Do the same for the remaining drives 5 drives

Just curious if anyone else has been through similar? Technically, I don't see anything incorrect with it but figured it wouldn't hurt to toss it out there in-case I am overlooking something...

TIA

 

[Arwen]

If you have an un-used disk slot(s), I would add the replacement disk(s) first, and then perform the replacement. It's safer since an error on the disk not being replaced won't be fatal, as the disk being replaced is still available. Basically ZFS will create a 3 disk mirror, and when the re-silver is complete, break off the disk to be replaced.

This is some what unique to ZFS, and something I have wanted since last century .

 

[Zredwire]

If you have an un-used disk slot(s), I would add the replacement disk(s) first, and then perform the replacement. It's safer since an error on the disk not being replaced won't be fatal, as the disk being replaced is still available. Basically ZFS will create a 3 disk mirror, and when the re-silver is complete, break off the disk to be replaced.

I agree with this. If you do not have enough un-used disk slots I would certainly not replace more than one at a time (like if you have to break a mirror and offline the disk leaving only 1 disk in the mirror until the new one is resilvered).

 

[Mirfster]

Good point, while that system doesn't have free slots it does give me an idea to possibly use another system that is a JBOD.

Thanks

 

[Stux]

You could do it, but you are fully removing all redundancy.

If you do, run a scrub first.

 

[Arwen]

Silly me, I forgot to mention you could detach your hot spare and use it's slot for the replacements. Whence you have replaced them all, you simply re-add the hot spare using the new size disk.


And while a scrub before replacement is a good idea, there is a tiny chance that the scrub caused a hardware failure that is not yet detected. (Okay, extremely tiny :)

However, on one job when my boss got back from a sales trip, I un-packed the workstation he took, boot tested it and ran a full backup to test the disk drive and tape drive. Log showed no SCSI bus errors during the backup. An hour later, the system died. Lost both the hard drive and system board. Disk was totally un-recoverable by us. I hurriedly dug out the test tape from my recycle pile, write protected it and labeled it. My boss was known to make notes on the workstation, including code changes he wanted to keep.

Long story cut short. Paranoia can be a good thing.