How to free space on a replication target?

[FrankWard]

This should be dead simple, but alas.. I currently replicate from NAS1 source to NAS2 target (details in sig) and everything was fine until I started to run out of space on the target. I assumed I could delete/exclude from the source, and this would replicate over the the target and free up space. I made some snapshot deletions, dataset exclusions, and took a new snapshot for them on source. I then tried to replicate these changes over but it didn't seem to work.

The replication task properly reported that 11.18TB was the size, but it exceeded this and threw an out of space error. I have 300GB reserved on the target, so the disk is not full, but I need to figure out how to clear up some space and wanted to check with the wizards before I went any further.

Does anyone have tips for how I should be approaching this? Is what I did the wrong approach?

Thanks!

 

[winnielinnie]

How did you configure the "retention" policy?

Keep in mind that under-the-hood, you need to use the -R flag (sender) and -F flag (receiver) to destroy any extinct datasets/snapshots on the destination, which were deleted on the source's side. Not sure how to expose that in the GUI, sorry. It might even veer from iX's intended use of the Replication Task.

 

[chuck32]

Does anyone have tips for how I should be approaching this? Is what I did the wrong approach?

Fitting 28 TB into 16 TB is ambitious ;)

Your signature may vary over time, please include the details in the post.

What do you mean by 300 GB reserved? I hope that is not the free space left, you shouldn't really exceed 80 % usage.

Where did you exclude the datasets? In your snapshots or in your replication? The replication task will add the changes via snapshots and deleted snapshots on source should be deleted on the target as well. I haven't that figured out 100% yet, sometimes they will not be deleted.
However this will not automatically delete datasets.

I would say you need to delete the datasets on the target that you wish to exclude manually. You can also delete older snapshots there but make sure you don't destroy an incremental base, i.e. that there is no common snapshot left to increment the changes from.

Edit: I'm assuming the GUI Replication task was used.

 

[FrankWard]

Fitting 28 TB into 16 TB is ambitious ;)

I keep my sigs up to date to avoid confusion and to avoid retyping. We're working with 11.8TB. ;)

What do you mean by 300 GB reserved? I hope that is not the free space left, you shouldn't really exceed 80 % usage.

I reserved space on the target dataset so it would never actually consume 100% and prevent delete operations. It's now reporting 100% usage, but 300GB is still available. If the replication works properly, there will only be 11TB replicated to the 16TB target.

Where did you exclude the datasets? In your snapshots or in your replication? The replication task will add the changes via snapshots and deleted snapshots on source should be deleted on the target as well. I haven't that figured out 100% yet, sometimes they will not be deleted.
However this will not automatically delete datasets.

Datasets were excluded on the source snapshot task and also in the source replication task. I assumed that whatever the source snapshot contains will be replicated which would include removing any blocks of data excluded from the snapshot thereby freeing up space on the target.

I would say you need to delete the datasets on the target that you wish to exclude manually. You can also delete older snapshots there but make sure you don't destroy an incremental base, i.e. that there is no common snapshot left to increment the changes from.

GUI task was used. I actually tried this before, but it gave me the 'no incremental base found' issue. I then removed all snapshots from the source except for the new ones that were snapshotted with the datasets excluded. The replication task went beyond the 11TB it was supposed to replicate. This is where my confusion lies. If there is only one snapshot of the current source data, how can that snapshot replicate over larger than it is on the source?

 

[FrankWard]

How did you configure the "retention" policy?

Keep in mind that under-the-hood, you need to use the -R flag (sender) and -F flag (receiver) to destroy any extinct datasets/snapshots on the destination, which were deleted on the source's side. Not sure how to expose that in the GUI, sorry. It might even veer from iX's intended use of the Replication Task.

The retention is default, same as source. I'll look into this more once I figure out what is actually going on.

 

[FrankWard]

To confirm my findings for future users..

1. Yes, snapshots work exactly the way I thought they did.
2. I did have to delete the remote dataset and let it re-replicate back to the target.
3. I still am not sure if I could have located a base snapshot and copied it over via shell to solve the issue.
4. Reserving space in a dataset on the target prevented the disk from filling during replication and avoiding filling the pool to 100% usage.
5. Excluded child datasets were properly excluded from the snapshots reducing the total usage of the target.

 

[chuck32]

To confirm my findings for future users..

1. Yes, snapshots work exactly the way I thought they did.
2. I did have to delete the remote dataset and let it re-replicate back to the target.

Are you sure with 1)?

You did have to remove the datasets first as I suggested.

Datasets were excluded on the source snapshot task and also in the source replication task. I assumed that whatever the source snapshot contains will be replicated which would include removing any blocks of data excluded from the snapshot thereby freeing up space on the target.

Talking only about the GUI options, I'm not sure if I get you right, but I assume you have setup something like this:

Code:

└───parentdataset
    ├───child1
    ├───child2
    └───child3

And have setup a recursive snapshot task for parentdataset. Feel free to correct me but to my knowledge this will result in:

Code:

└───parentdataset@snapshot1
    ├───child1@snapshot1
    ├───child2@snapshot1
    └───child3@snapshot1

There will be independent snapshots for each child and I assume if you either start excluding them from the snapshot task or the replication task those will not get send over. As long as you don't delete them on the source they will not get deleted on the target.

I read it somewhere here and made my own experiences: A single recursive snapshot task for a dataset with many child datasets is not what I would recommend, this makes it fairly easy to break the replication task when you start messing around with deleting individual snapshots because you will break the incremental structure. Personally I prefer 1 snapshot task and a corresponding replication task per individual dataset.

 

[FrankWard]

Are you sure with 1)?

You did have to remove the datasets first as I suggested.

Thanks for the insight. I did have to remove the child dataset from the target before it would replicate properly, but this did not affect the replication of the parent or other child datasets after. Excluding datasets from the snapshot was successful in freeing up the space on the target.

POOL
└───parentdataset
├───child1
├───child2
└───child3

└───parentdataset@snapshot1
├───child1@snapshot1
├───child2@snapshot1
└───child3@snapshot1

That's essentially the same hierarchy.

There will be independent snapshots for each child and I assume if you either start excluding them from the snapshot task or the replication task those will not get send over. As long as you don't delete them on the source they will not get deleted on the target.

I read it somewhere here and made my own experiences: A single recursive snapshot task for a dataset with many child datasets is not what I would recommend, this makes it fairly easy to break the replication task when you start messing around with deleting individual snapshots because you will break the incremental structure. Personally I prefer 1 snapshot task and a corresponding replication task per individual dataset.

I see where you're coming from. I did have the parent set to recursive. My mistake was accidentally removing the base for one of the child datasets. It appears to be the same as having independent snapshots as the parent and other children were unaffected by this one child dataset not being able to replicate. That's the one I had to delete. In fact, I only deleted it remotely, and just excluded it from the new snapshot/replication tasks which resulted in freeing up the space on the target but keeping the dataset and snapshots on the source.

Eventually I will avoid this by upgrading the pool with larger disks, but for now, I don't need to replicate the entire pool, just the important datasets. I've backed the excluded dataset offsite, so if I were to lose it on the source, I could recover it. In the meantime, I have a few extra TB for important data on the target.