Please help me vet this comprehensive guide to re-balancing data on striped mirror pools

[SnoppyFloppy]

Hi

I have a striped pool of mirrored vdevs that I extended some time ago without considering the distribution of data on the disks and so I want to re-balance the pool to improve performance and distribute disk-wear better.

I'm only somewhat familiar with zfs and though I do have off-side backup of the pool, I would much prefer to not have to replicate everything back in case of a screw-up.

Therefore I pieced together this detailed guide for myself, from info on stachexchange, reddit and here, for that I would very much appreciate if any of you veterans could help me vet. And when the guide are all good, maybe it can help someone else.

NB. I know there are other ways around this - like local copy/paste and even bash scripts that do this but I see this as a zfs learning opportunity

Thanks in advance.

---

1. Get to the TrueNAS shell either locally or over ssh

2. Get a root shell and start a tmux session

Code:

$ sudo -i
$ tmux

3. Get an overview of your existing pool (oldpool) and copy the output to a text file somewhere

Code:

   $ zpool list -v oldpool | column -t
   NAME               SIZE   ALLOC  FREE   CKPOINT  EXPANDSZ  FRAG  CAP    DEDUP  HEALTH  ALTROOT
   oldpool            30T    20.2T   9.8T  -        -         0%    67%    1.00x  ONLINE  -
      mirror-0        10.9T  9.04T  1.86T  -        -         0%    82.9%  -      ONLINE
         diskID1      -      -      -      -        -         -     -      -      ONLINE
         diskID2      -      -      -      -        -         -     -      -      ONLINE
      mirror-1        10.9T  9.03T  1.88T  -        -         0%    82.8%  -      ONLINE
         diskID3      -      -      -      -        -         -     -      -      ONLINE
         diskID4      -      -      -      -        -         -     -      -      ONLINE
      mirror-2        10.9T  2.1T   8.8T   -        -         0%    19.3%  -      ONLINE
         diskIDn-1    -      -      -      -        -         -     -      -      ONLINE
         diskIDn      -      -      -      -        -         -     -      -      ONLINE

4. Scrub the old pool

Code:

   $ zfs scrub oldpool

5. Stop all services using the old pool, (e.g. shares, k8s, VMs, Etc.)

6. If applicable: move the 'system dataset' to another pool

7. Make a temporary recursive snapshot of the entire pool including all local snapshots

Code:

$ zfs snap -r oldpool@migration

8. Detach one disk from each mirror in the old pool

Code:

$ zpool detach oldpool DiskID1
$ zpool detach oldpool DiskID3
$ zpool detach oldpool DiskIDn-1

9. Create a new pool with single-disk vdevs using the detached disks from the old pool

Code:

$ zpool create \
   -o ashift=12 \
   -o atime=off \
   -o compression=lz4 \
   -o xattr=sa \
   -o encryption=on \
   -o keylocation=file:///path/to/encryption-key.json \
   -o keyformat=hex \
   newpool \
   mirror DiskID1 \
   mirror DiskID3 \
   mirror DiskIDn-1

11. Replicate everything from oldpool to newpool

Code:

$ zfs send -R oldpool@migration | zfs recv -Fuv newpool

11. Scrub the new pool

Code:

$ zfs scrub newpool

11. Destroy the old pool

Code:

$ zfs destroy oldpool

12. Attach a disk from the now destroyed oldpool to each single-disk vdev in newpool

Code:

$ zpool attach newpool DiskID1   DiskID2
$ zpool attach newpool DiskID3   DiskID4
$ zpool attach newpool DiskIDn-1 DiskIDn

13. Rename the pool back to oldpool

Code:

$ zpool export newpool
$ zpool import newpool oldpool

14. Move back system dataset and restart services

 

[sretalla]

That will work assuming you're happy with the risk of having no redundancy for much of the process.

You could also get the same result (much quicker) without that part of the risk by using this:

GitHub - markusressel/zfs-inplace-rebalancing: Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool.

Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool. - markusressel/zfs-inplace-rebalancing

github.com

 

[SnoppyFloppy]

That will work assuming you're happy with the risk of having no redundancy for much of the process.

You could also get the same result (much quicker) without that part of the risk by using this:

GitHub - markusressel/zfs-inplace-rebalancing: Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool.

Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool. - markusressel/zfs-inplace-rebalancing

github.com

Yeah I am aware of that script and lacking redundancy in my method.

EDIT: On the flip-side, it doesn't look like the script checks the integrity of the copied file before removing the original.

My concern is that I have a lot of local snapshots on that pool and it seems like I would have to delete all those in order not to having redundant snapshots, which I fear would screw up my remote replication.

 

[sretalla]

On the flip-side, it doesn't look like the script checks the integrity of the copied file before removing the original.

ZFS does do that for you though...

I get your point about the snapshots.

Anyway, whatever works for you is fine.

 

[sretalla]

Oh, I should also have suggested the other way... I find it annoying, but it would seem to meet your criteria.

For each dataset, create a new copy with snapshot/zfs send|recv, (you can then check the integrity of the copy if you wish, again, ZFS already does that), then once happy, remove the original and rename the new to the original using zfs rename.

That should land you with the same content on the same pool with the same snapshots, only balanced. (with redundancy maintained throughout)

Assumes you have the free space for it. (one dataset and all its snapshots at a time).

 

[SnoppyFloppy]

Oh, I should also have suggested the other way... I find it annoying, but it would seem to meet your criteria.

For each dataset, create a new copy with snapshot/zfs send|recv, (you can then check the integrity of the copy if you wish, again, ZFS already does that), then once happy, remove the original and rename the new to the original using zfs rename.

That should land you with the same content on the same pool with the same snapshots, only balanced. (with redundancy maintained throughout)

Assumes you have the free space for it. (one dataset and all its snapshots at a time).

That's a good approach and certainly have many advantages over my suggested approach. I do fear though that my movie dataset is larger than the free space though.

 

[sretalla]

Some rebalancing is better than none... if that pool is only/mostly media anyway you won't see a lot of difference after balancing. ... other than in zpool list -v of course