Replicate large pool or zvol

[Steve B.]

I've got three storage servers:

• The original 100TB storage server running TrueNAS Core 12.0-U2.1.
• Two new 300TB storage servers running TrueNAS 13 Core, but I could switch to TrueNAS Scale if it would help.

On the original storage server, I've got a 60TB zvol in the main pool and I need to replicate it to one of the new storage servers. I can't use the normal replication method because there's not enough free space on the original storage server to snapshot the 60TB zvol.

I've already migrated the data in the other zvols from the original storage server to one of the new ones by doing a VMware storage migration. But, a storage migration of the 60TB volume keeps dying at about 36% so it doesn't seem like that's an option. 'StorageServer01' has the migrated data on it, minus the 60TB zvol. 'StorageServer02' doesn't have anything on it other than a basic storage pool. Once the 60TB zvol is migrated/copied/replicated, it will be either extended or have another volume added so the total storage is about 120TB. If I keep each volume below 60TB, I think I'll have enough storage to snapshot them but is that really what I want to do? Snapshot 60TB? Also, eventually, I want all the volumes on one of the storage servers and replicating to the other one.

How do I need to replicate without using snapshots? I know I can use rsync but I'm concerned about doing that because I'm worried I might over-write the current zvols. Any help on this is greatly appreciated.

 

[Ericloewe]

I can't use the normal replication method because there's not enough free space on the original storage server to snapshot the 60TB zvol.

That's not how this works. Snapshots don't take up extra space. They only take up space if you "overwrite"/delete something. If you never actually delete anything, snapshots take only the minimal amount of storage needed for their metadata.
For your specific case, that means that you can take a snapshot, send it, and delete it from the origin if it's inconvenient. But it's not optional, and for a very good reason: Replication could not be made atomic on a live filesystem. Snapshots are atomic, and thus can be replicated safely.

 

[Steve B.]

That's not how this works. Snapshots don't take up extra space. They only take up space if you "overwrite"/delete something. If you never actually delete anything, snapshots take only the minimal amount of storage needed for their metadata.
For your specific case, that means that you can take a snapshot, send it, and delete it from the origin if it's inconvenient. But it's not optional, and for a very good reason: Replication could not be made atomic on a live filesystem. Snapshots are atomic, and thus can be replicated safely.

Thank you for your reply. When I've taken snapshots at the SAN level, the initial snapshot was sizable, although usually nowhere near as large as the volume. Subsequent snapshots were just delta changes so they were small. It's good to know TrueNAS does it differently. The OS drive I've got to replicate is a full 50TB of rotating video files (the volume is 60TB) so I was assuming the snapshot was going to be huge as well. Once again, TrueNAS crushes it!

BTW. Cool avatar. I miss playing that.