Re-Claim space after Replication

[Doug183]

1) I have a large data set that lives on 10x18TB in RAIDZ2 and last year, replicated it to another exact drive set which I then off-lined the 2nd set by pulling the drives and putting them on a shelf.
2) This year, I have removed 10 TB on the source set.
3) I am pretty sure that if I just send /receive that Snapshot again, I will have a perfect copy again. (not yet done yet)

The Snapshot is working perfectly as designed by preserving that 10TB of data. However, I am clear and want to permanently delete that 10TB of data on both sets of drives, and I am good with that.

Question:
So with Replication, how do I recover that 10TB of space to use again without doing a full, new complete replication job?
Can I just create a new snapshot, delete the old snapshot, and re-run the replication job? My gut says that's too simplistic but maybe not?

 

[winnielinnie]

It depends how you set up your snapshots and replication task.

Without a (practical) "expiration" life, old and intermediary snapshots will continue to take up the space used by delete files.

If you know exactly which snapshot(s) contains 10TB of unneeded data, you can specifically destroy it on both ends; source and destination.

Just make sure this snapshot is not needed as a "base" to do future incremental replications.

 

[Doug183]

Ok, so what you are saying is.

1) Create New Snapshot on the source.
2) Replicate the new Snapshot. (This will not force the entire stream to be sent, just changes in a sense.)
3) Delete "old" Snapshot on both the source and destination.

 

[winnielinnie]

3) Delete "old" Snapshot on both the source and destination.

If the "expiration" date set on the snapshot is too far away (usually the same on the source and destination, which is technically handled by a zettarepl "parse").

The next subsequent replication should be able to send an incremental stream using the most recent "base" snapshot (shared on both ends).

Just double- and triple-check your steps. Whether it's files, datasets, or snapshots, destruction is destruction. (It's especially important that you do not destroy a "base" snapshot on either side before you are 100% certain you've already created and sent a new snapshot that will be the latest "base" snapshot for subsequent replications.)

Are you sure that only a single snapshot, on its own, is responsible for the 10TB of deleted data? This implies that earlier snapshots do not share this deleted data, and neither does the immediate snapshot after it.


EDIT: Should have asked earlier:

• Are you doing this via the GUI or command-line? (Snapshots and replications.)
• Are these automatic snapshots, or manually created?
• If using the GUI, is this Replication Task married to a Periodic Snapshot Task?

 

[Doug183]

First thanks for the explanations. They clearly explain the relationships and flow of snapshots.

Are you sure that only a single snapshot, on its own, is responsible for the 10TB of deleted data? This implies that earlier snapshots do not hold this deleted data, and neither does the immediate snapshot after it.

Yes, there is currently only one snapshot, so I am sure the 10TB is only "preserved" there.

• Are you doing this via the GUI or command-line? (Snapshots and replications.)
• Are these automatic snapshots, or manually created?
• If using the GUI, is this Replication Task married to a Periodic Snapshot Task?

- Snapshot is manual but done in the GUI.
- Send / Receive is done via command line.
- As for your point #3, I don't think applies.


Overall I am either leveraging or abusing how Snapshots and replications works. Most of my pools (datasets) do not change over time, so they are in a sense fixed storage. (Big video archive where nothing changes.) Those are easily replicated and updated. However, there are two pools/datasets that grow by leaps and bounds - (10TBs) gets copied into one pool/dataset every month or so. Then that data gets sorted and labeled, and a portion moves to the other growing dataset. The copied data gets deleted eventually. On an empty pool/dataset, I can be lazy and let it just grow and delete data as long as the free space is bigger than the 10 TB transfer a month. However as either the pool gets close to filling up, there is a lot of storage managing to maxamize space, which I then need to deal with replication. I am sure there is an easier and more expensive method of just having many extra open drive slots and extra sets of empty 10 drives on hand, but we haven't made that commitment yet. Ok maybe TMI.

Thanks
Doug

 

[winnielinnie]

- Send / Receive is done via command line.

What is the full command you are using? (You can obfuscate the names of your pools, datasets, and snapshots if they contain private/personal info.)

 

[Doug183]

I used:

zfs send Storage8_Live/Endevour@Endevour_041022 | zfs recv Storage8_bak/Endevour_Clone


I got this from this Oracle document.

 

[winnielinnie]

That's not an incremental send.

To do an incremental send you need to use either the -I or -i flag, and specify a "base" snapshot and "latest" snapshot, such as:

Code:

zfs send -i Storage8_Live/Endevour@Endevour_041022 Storage8_Live/Endevour@Endevour_060822 | zfs recv Storage8_bak/Endevour_Clone

This will send Endevour@Endevour_060822 to the destination, but rather than sending the entire stream (of the entire snapshot's filesystem/contents in its entirety), it will only need to transfer the differences between @Endevour_041022 and @Endevour_060822; assuming that the destination already has @Endevour_041022.

 

[Doug183]

Great. Then "060822" becomes my new "base" Snapshot and I can delete "041022"? And then into the future, use later dates to do additional incrementals? I'm just concerned that if I delete the original base (041022), all hell breaks loose.

 

[winnielinnie]

I'm just concerned that if I delete the original base (041022), all hell breaks loose.

Only if...

• You later realize that it contains a (deleted) file that you actually didn't want to lose
• You destroy it before creating and (incrementally) sending a new snapshot from source to destination

If the current state of your dataset's "live filesystem" is what you are happy with, then the new snapshot you create will be as such. If your destination pool also has this new snapshot (because you sent it over), then you're essentially at a point where you don't need the older snapshot (according to your routine and use-case.)

But I can't speak for you...


EDIT: You might want to reconsider how you're doing things. 10TB of data regularly being created and deleted? It seems inefficient and results in needlessly massive transfers.

 

[Doug183]

Yep.

EDIT: You might want to reconsider how you're doing things. 10TB of data regularly being created and deleted? It seems inefficient and results in needlessly massive transfers.

But its a video archive with a server storing High Definition footage and another with 4K sized footage. (4Ks are not used as frequently so that server stays shut down more often. Once a month or so, 10 TBs (Ok maybe its 8TB.) comes into the 4K server. 6 TB's stays on it (QuickTimes of 4K film transfers) and 2 TB (QuickTime of High Definition versions) are transferred to the High Definition server. The 2TB's gets moved into a temporary trash folder which is deleted as space becomes needed. In this way, the 4K server hardly needs to switched on and I don't have to run as many servers simultaneously to find footage i.e. if the footage wasn't split up this way. Also the HD server fills up slowly.