SOLVED How to replace disks without losing redundancy

[willglynn]

Suppose I have a zpool with a RAID-Z2 vdev where drives da0-da5 are ONLINE and healthy. Now, suppose I want to replace da0. I know I can pull it out, install its replacement, and resilver – but what if I have extra drive bays? Can I install the replacement as da6, resilver, and then remove da0?

My research indicates that ZFS handles this properly with a simple zpool replace: Oracle docs say so (scroll to "zpool replace tank c1t3d0 c4t3d0"), and ServerFault agrees about ZFS on FreeBSD. On the other hand, I know that FreeNAS defaults to adding swap space on each drive, and zpool status shows gptid/… instead of da0, so I expect the solution isn't just zpool replace da0 da6.

So: is this possible in FreeNAS? If so, what is the procedure? Is there some reason I shouldn't do this?

 

[SirMaster]

You would have to do it in the command line as the GUI does not provide the option to do this.

But there are some people on here who don't believe in using the command line so they will say don't do it. I do it though and it works fine. The choice is yours whether you personally want to go against the forum and do it from the command line like a real server admin.

You are on the correct track it's not just "zpool replace da0 da6". You would want to create the partitions on da6 like FreeNAS does before you run the replace specifying the new partition rather than whole disk.

Then I would export and import the pool in the UI so it gets updated of the new disk.

 

[mjws00]

Yep. The gotcha is the swap partition set up and using gtpid's. Plus the config db won't know what you've done. I suppose you could set up the new drive as per the FN defaults, then manually do the replace using the second partition as the target and referenced via gptid. You might have to export and then re-import at that point to ensure consistency between pool and config db.

All in all sounds like more trouble than it's worth. Of course zfs from the CLI will happily let you replace da(x) with almost anything.

 

[willglynn]

Ah, it looks like the FreeNAS GUI can do this after all, so I can replace without losing redundancy and without de-synchronizing the higher and lower layers of the system. Volume Status > Replace… lets you select ONLINE drives, resulting in exactly the behavior I wanted.

Code:

pegasus# zpool status
  pool: tank
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Nov 13 18:34:05 2014
        34.1G scanned out of 20.6T at 2.27G/s, 2h34m to go
        3.37M resilvered, 0.16% done
config:

    NAME                                              STATE     READ WRITE CKSUM
    tank                                              ONLINE       0     0     0
      raidz2-0                                        ONLINE       0     0     0
        gptid/16307ba2-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
        gptid/169e6cc3-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
        gptid/170ea37f-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
        gptid/177ccfd3-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
        gptid/17e9f568-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
        gptid/1858fc11-67b5-11e4-b529-d8d3855efa4e    ONLINE       0     0     0
      raidz2-1                                        ONLINE       0     0     0
        gptid/946f1195-6a1d-11e4-9f64-d8d3855efa4e    ONLINE       0     0     0
        gptid/956818f8-6a1d-11e4-9f64-d8d3855efa4e    ONLINE       0     0     0
        gptid/9603f1f1-6a1d-11e4-9f64-d8d3855efa4e    ONLINE       0     0     0
        gptid/96a738a2-6a1d-11e4-9f64-d8d3855efa4e    ONLINE       0     0     0
        replacing-4                                   ONLINE       0     0     0
          gptid/979de4d5-6a1d-11e4-9f64-d8d3855efa4e  ONLINE       0     0     0
          gptid/efbf69dc-6b95-11e4-b184-d8d3855efa4e  ONLINE       0     0     0  (resilvering)
        gptid/989b2489-6a1d-11e4-9f64-d8d3855efa4e    ONLINE       0     0     0

errors: No known data errors

 

[NASbox]

In your example above, once you have resilvered can you move da6 to da0? (This would be great for the using an e-sata port to build the replacement outside the box, then move it inside. In my system I would have to power down to make the swap.)

 

[SirMaster]

When the resilver successfully completes, the old disk is automatically removed from the zpool.

Also, ZFS does not care what ports the disks are connected to. So after the resilver, simply shut down, stick the new disk into the old slot and boot up and everything will be normal.

 

[willglynn]

Yep, that's fine. FreeNAS doesn't care where the drives are plugged in, and you're free to change that across boots.