freenas transfer at block level help

Status
Not open for further replies.

greavette

Cadet
Joined
Jul 13, 2016
Messages
5
Hello Forum,

We have two FreeNAS setup over the Internet (two different locations - Location A and Location B. Both FreeNAS are running version::
FreeNAS-9.10-STABLE-201606270534 (dd17351)

  • We connected both FreeNAS over SSH and confirmed that the connection is made between both FreeNAS is good in that we can both see each others disk in Storage > Volumes.
  • At Location A a replication task was successfully setup.
  • At Location A a large virtual machine backup was placed into FreeNAS of roughly 20 GB large. Over 2 hours we could see that the entire file was successfully transferred to Location B FreeNAS.
  • A second backup was taken of the same virtual machine. Nothing really had changed on the virtual machine but the filename of that second backup would of course be different.

What we are looking for at this point is block level replication to occur between the two FreeNAS boxes where our expectation would be that the transfer of the second virtual machine backup would be done quick (since at the block level not all that much has changed). What we discovered though was that the replication task took over 4 hours to complete to move the second file. Basically it took twice as long to move the second backup using replication task.

How do we know that replication at the block level was being used? I'm guessing here but based on the time it took for the replication to finish..it seems more like freenas is acting like rsync where it has double check each block?

Is replication at the block level in FreeNAS expected to be slower or should our experience have been much quicker?

Any advice or pointers you can give would be greatly appreciated.

Thank you.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
So you took a snapshot, replicated, changed nothing in the file, took another snapshot, replicated again? You confused me when you said "took over 4 hours to complete to move the second file". What second file?

What backups are you talking about? Are you making backups? Or do you mean snapshots?

You are asking about block level replication, but your ZFS terminology is confusing me. I want to make sure we are talking about the same things.

Replication copies a snapshot or it can copy the difference between 2 snapshots. So how did you kick off the replication? Manually? Did you use the option to only send the deltas?

I've taken a snapshot of a dataset with 10TB (initial replication took hours), made a couple file changes, then replicated again and it only took a minute. So what you are trying to achieve is possible.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Zfs replication will transfer that changes that occur between two snapshots.
 

greavette

Cadet
Joined
Jul 13, 2016
Messages
5
Hello depasseg and SweetAndLo, I appreciate you replying to my question.

I apologize for not using the correct terminology...still learning the right words to say in providing detail for my question.

Yes I'm referring to ZFS replication within FreeNAS.

I use Proxmox as my virtual hypervisor..doesn't really matter for the question I suppose what I use. I've taken a full backup of a virtual machine which created one 20GB file with a filename that contains a time date stamp. I put this file into my Location A FreeNAS and waited for the replication task to copy it to the Location B FreeNAS. I turned back on the virtual machine and added a couple of text files to the desktop. Once the copy was done I turned off the virtual machine and took a second backup of the virtual machine. The second backup creates another 20 GB file with a filename that contains the time date stamp of the second backup. So now I have two files...but the virtual machine in each of the backup files is for the most part the same with the exception of the few files I created (and I'm sure other files created during the running of the O/S..but for the most part not a lot of differences). I placed this second full backup file into my Location A FreeNAS and waited for the replication task to complete again. I was hoping to see this replication task take a very short time but it appeared to take over 4 hours before my Location B FreeNAS had the second full backup file.

Hopefully I've described the process a little better now...onto your question depasseng.

"Replication copies a snapshot or it can copy the difference between 2 snapshots. So how did you kick off the replication? Manually? Did you use the option to only send the deltas?"

Yes! This is exactly what I need to do...only send the deltas for my replication. I added a replication task in the Volumes section over SSH to be run every 5 minutes. I don't see an option on only sending deltas though? Where can I enable the setting to only send deltas? It would appear that I've setup replication to send a fully copy instead of sending the difference between my two full backup files. I setup my replication as per section 8.3 Replication Tasks from here - http://doc.freenas.org/9.3/freenas_storage.html#replication-tasks but I'm not seeing how to setup just the delta transfer of my two full backup files to make the replication that much quicker?

Or have I misunderstood what replication is doing...does my filename need to be the same for both my backup files for FreeNAS to only look at the deltas between each file and only replicate the unique blocks?

Thank you.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Are you taking snapshots of your datasets? Usually you replicate snapshots not your dataset to get the Delta copy. I have not played with this much so I could be wrong but I bet if you used snapshots and replication you get the behavior you want.
 

greavette

Cadet
Joined
Jul 13, 2016
Messages
5
Ahhh, I think I understand what you are saying. Let me try to explain better...sorry about this confusion.

FreeNAS A has a dataset called rep_test. It is made up of 2 100 GB drives in mirror mode (ZFS) so I have 100 GB of space on this dataset.
FreeNAS B has a dataset called reptest. It is also made up of 2 100 GB drives in mirror mode (ZFS) so again I have 100 GB of space on this dataset.
I've connected both datasets beteen the two FreeNAS servers so when I look in volumes I can see under Volumes the connected datasets: rep_test/reptest.

On my FreeNAS Location A I drop my full virtual machine backup (20 GB large) with a filename of 'vzump-qemu_100-2016_07_11_09_01_20_02.vma.lzo" to onto rep_test. Replication occurs to copy (not sure if that is the correct term to use) my full virtual machine backup Location B FreeNAS reptest dataset.

I then take another backup of the same virtual machine which has a filename 'vzdump-qemu-100-2016_07_11_11_00_03.vma.lzo' and I put this into my FreeNAS Location A rep_test dataset. Replication happens again but for this replication task my hope was that at a new file gets created on the FreeNAS Location B dataset reptest dataset with the new virtual machine backup name, but that the replication would happen extremely fast since at the block level both full backup files are very much the same.

So...you mention replicate your snapshots. Are you talking about replicating a snapshot of the rep_test dataset to reptest dataset? From what I saw...it almost looked like rsync was being used to simply 'copy' my second full backup file to my reptest dataset. I did get both full backups into my FreeNAS Location A reptest dataset..but the process took far longer than I'd hoped it would.

I will read up on replicating snapshots and what means to make sure I've setup my replication test properly.

I appreciate you taking the time to understand my problem.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Huh? Do you have 2 vms on freenas A. This is why the replication took so long it backed up the second vm.

I think your idea of replication and how things work could use some clarification. If you would have used the original vm and modified that then only the Delta would be copied. But since you created a new vm you essentially have twice as many blocks to backup.
 

greavette

Cadet
Joined
Jul 13, 2016
Messages
5
Yes...I think that is my problem. In that I did not understand what would happen.

I have one virtual machine but two backups for this one virtual machine. The two backups were taken hours apart. So the two backups are very much the same virtual machine with very few changes between them. But yes I have two 20 GB virtual machine backups in my FreeNAS A dataset.

My hope was that FreeNAS ZFS replication would see the ZFS blocks on the FreeNAS A dataset (rep_test) and recognize that even though I have two files there...at the block level a lot of the blocks are the exact same. So my hope was that the repliation of the second virtual machine backup would have been lightening fast because at the block level only the changed blocks would be send to my FreeNAS B reptest dataset...and still the second full virtual machine backup file would then exist on my FreeNAS B reptest dataset.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
The GUI Replication will use delta snapshots, so you are all set there. The problem is that you are creating a second 20GB file. If you were to use the snapshots function in FreeNAS to control the "backup", instead of actually creating a backup in ProxMox (or elsewhere), then only the changed blocks will replicate.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
My hope was that FreeNAS ZFS replication would see the ZFS blocks on the FreeNAS A dataset (rep_test) and recognize that even though I have two files there...at the block level a lot of the blocks are the exact same.
In the eye of freenas, those are 2 separate files. What you are looking at is called deduplication and has a host of considerations.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,420
If you were to use the snapshots function in FreeNAS to control the "backup", instead of actually creating a backup in ProxMox (or elsewhere), then only the changed blocks will replicate.
It's worth pointing out that Proxmox itself can run on ZFS, and when you're doing so, it can do the backups using ZFS replication using the pve-zsync script. I'm using this on my system, and it works quite well. Not sure if this is relevant to OP's question, though.
 

greavette

Cadet
Joined
Jul 13, 2016
Messages
5
Good point danb35, and we do use ZFS with our Proxmox hosts..as well as Ceph Clusters with our larger Proxmox Cluster. But for offsite/disaster recovery it would be good to have an almost realtime replication of our virtual infra to Location B through FreeNAS.

I think Dedupe is closer to what we are looking for.

Thanks all for your helpful suggestions in this thread.
 
Status
Not open for further replies.
Top