Moving data to a new pool, rebalancing?

[Andybluejay]

My current pool has 3 12-wide vdevs, 2x8TB and 1x18TB. In hindsight I should have added the 18TB drives as a second pool. Ultimately what I'd like is a pool with 2 12-wide vdevs of 18TB disks, and then either repurpose or retire the 8TB drives.

The original pool has around ~150TB of data on it that I'd like to move to the new pool. I created a second pool with 12 new 18TB drives in raidz2 which gives me 149TB usable space... you can probably see where I'm going with this. If I pare down the data on the original pool so it will fit on the new pool can I just snapshot and send + receive? Is there a better method to transfer the data? I realize that will more or less completely fill the new pool, which seems to be a ZFS no-no but how concerned should I be about 'over' filling the new pool? i.e. what is an acceptable minimum amount of free space?

I would be destroying the old pool then adding the other 12x18TB drives as a 2nd vdev, which would leave me with 1 full vdev and 1 empty vdev. Can I simply rebalance the data using this script? https://github.com/markusressel/zfs-inplace-rebalancing

The alternative to this scheme would be to backup the original pool in its entirety, destroy it, create my new 2x12wideZ2 pool, and move the data back over. Is this a better plan or does moving/rebalancing get me to the same end result?

 

[Arwen]

You do want to keep the space used on the initial single 12 x 18TB vDev / pool down. Otherwise you end up with somewhat permanent fragmentation of the last files to be copied. I've heard that ZFS begins space saving at 90%, meaning it will use smaller chunks of disk space to satisfy a write transaction. That generally means fragmentation.

Yes, in theory copying the datasets after adding a 2nd vDev would reduce the fragmentation and re-balance the data.

But, if it were me, and only doing the copy to a single 12 disk x 18TB vDev / pool, I would keep the copy below 90%. Even down to 80%. That might mean having to off-load, (with backups of course), 10s of TBs that only get copied after adding the 2nd vDev.

 

[Andybluejay]

You do want to keep the space used on the initial single 12 x 18TB vDev / pool down. Otherwise you end up with somewhat permanent fragmentation of the last files to be copied. I've heard that ZFS begins space saving at 90%, meaning it will use smaller chunks of disk space to satisfy a write transaction. That generally means fragmentation.

Yes, in theory copying the datasets after adding a 2nd vDev would reduce the fragmentation and re-balance the data.

But, if it were me, and only doing the copy to a single 12 disk x 18TB vDev / pool, I would keep the copy below 90%. Even down to 80%. That might mean having to off-load, (with backups of course), 10s of TBs that only get copied after adding the 2nd vDev.

Appreciate the response. Yeah I've read mixed things, seems like 20% free space used to be a common recommendation with some anecdotal reports of people going to 90-95% without issue. Like most things with ZFS I expect the actual answer is a bit more complicated and depends on things like the type of data stored, the record size, disk configuration, etc...

I'm just going to backup as much as I can to local servers and give it a shot.

 

[Baenwort]

I'd do the backup and restore method as it's also good to test your backup method occasionally and I view moves like this as a chance to do a live validation of my backup methods.

 

[dxun]

@Andybluejay Not sure if I am late with this but I did use the script to successfully rebalance my own (much smaller) pool. What did you end up doing in the end?

 

[ZhangYue]

My current pool has 3 12-wide vdevs, 2x8TB and 1x18TB. In hindsight I should have added the 18TB drives as a second pool. Ultimately what I'd like is a pool with 2 12-wide vdevs of 18TB disks, and then either repurpose or retire the 8TB drives.

The original pool has around ~150TB of data on it that I'd like to move to the new pool. I created a second pool with 12 new 18TB drives in raidz2 which gives me 149TB usable space... you can probably see where I'm going with this. If I pare down the data on the original pool so it will fit on the new pool can I just snapshot and send + receive? Is there a better method to transfer the data? I realize that will more or less completely fill the new pool, which seems to be a ZFS no-no but how concerned should I be about 'over' filling the new pool? i.e. what is an acceptable minimum amount of free space?

I would be destroying the old pool then adding the other 12x18TB drives as a 2nd vdev, which would leave me with 1 full vdev and 1 empty vdev. Can I simply rebalance the data using this script? https://github.com/markusressel/zfs-inplace-rebalancingwe become what we behold

The alternative to this scheme would be to backup the original pool in its entirety, destroy it, create my new 2x12wideZ2 pool, and move the data back over. Is this a better plan or does moving/rebalancing get me to the same end result?

This is a great choice.