Replication not replicating dataset quotas and other options

Status
Not open for further replies.

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
I have three FreeNAS servers. Master has replication setup to both a backup FreeNAS and a disaster recovery (DR) FreeNAS. Before setting up replication on master I transferred the main 'tank' pool's latest snapshot to backup and DR manually using zfs send recv piped through netcat--replication otherwise uses SSH and transferring the initial 35TB was going way too low. Replication is keeping up with files being written and deleted, new datasets, and destroyed datasets, but both backup and DR have discrepancies otherwise. I have verified this in both the web UI and console using zfs list. For example:

Master:tank/dataset1 has a quota of 9T and backup shows this but DR has no quota
Master:tank/dataset1 has the share type of Windows and backup shows this but DR shows UNIX
Master:tank/dataset2 has a quota of 2T and DR shows this but backup has no quota

This is a minor issue because if I needed to restore from backup the files are all there, but it would be inconvenient to have to remember and redo quotas and share types/permissions.
 
Joined
Jul 3, 2015
Messages
926
This can sometimes be the case if the replication is still running and hasn't finished. Also might be worth if/when the replication has finished and you still have the problem given the replica a reboot to see if that helps.
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Yes replication finished quickly using netcat over 10 Gb/s and although I haven't restarted master FreeNAS, backup and DR FreeNAS have been rebooted since then.
 
Joined
Jul 3, 2015
Messages
926
Ah ok, Im not sure then. I guess you did the replication outside of the UI as you used netcat?
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Yes but I'm not sure why a manual zfs send recv would affect replication. Replication had no errors picking up where the initial manual zfs send recv ended.
 
Joined
Jul 3, 2015
Messages
926
Sometimes you get unexpected results when operating outside of the UI. The issues you are seeing I have not and I have multiple boxes replicating daily on various versions of the appliance however all configured using the UI for snapshots and replication. Im not saying for sure it is that however that's the only difference I can see between you and me.
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
I couldn't stomach replication's 150-300 Mb/s speed slowly pumping out 35 TB while netcat was able to do zfs send recv at 2.5 Gb/s with much lower CPU overhead.

Unfortunately my disaster recovery FreeNAS is off-site behind a ~60 Mb/s connection. I had to plug its drives into my backup server to do the initial netcat zfs send recv and then walk it over sneaknet style to the DR server and import the pool. Importing pools: one of the best features of ZFS.
 
Joined
Jul 3, 2015
Messages
926
Perhaps you could have done a local replication first setup via the UI if the drives were physically attached to speed up the process then move it?
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Does web UI replication not use SSH if it is local (all drives plugged up to the same server)? Because the bottleneck is the overhead of SSH.

My initial manual zfs send recv's were local in the sense that they were in the same rack, but it was actually from master --> backup and master --> backup (second array for DR) over directly connected 10 Gb.
 
Joined
Jul 3, 2015
Messages
926
Im not sure tbh however Im sure I've done it before and it was much faster. My default speeds when replicating across 10Gb infrastructure is about 1Gb/s which agreed isn't great but it works, however this does vary a lot.

Ah when you said local I thought you meant the drives physically connected so you could import the pool into the same head.
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Hmm that makes me think my 150-300 Mb/s replication speeds over 10 Gb/s is because of a bottleneck in my processor's age not SSH, in general. I'm using R610s with Xeon E5540s, ugh.
 
Joined
Jul 3, 2015
Messages
926
Yeah that's why I mentioned it as I thought your speeds were slightly on the low side. You can see Im running a very similar setup. Don't get me wrong I don't get a solid 1Gb/s all the time as it fluctuates quick a bit but on the whole as an average I'd say its approx 1Gb/s.
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
I just moved the disks that make up my DR array from off-site to the same NetApp DS4243 that my primary array is in and setup replication for localhost. Unfortunately I am still getting ~120 Mb/s running replication from local array to local array. The CPU graph is only showing ~20% usage too.
 
Last edited:

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Replication is actually slower from one array to another array in the same server--makes sense since a single server has to do both read and writes including compression and decompression. But even when I moved my DR array to my backup server which is connected to the master via 10 Gb replication never got over ~200 Mb/s. After some thinking I suspect the issue I was originally having was because I didn't use the proper flags during the initial netcat zfs send recv backup. I simply did zfs send pool@snapshot | nc pipe | zfs recv pool. I'm going to resend using zfs send -R pool@snapshot | nc pipe | zfs recv -Fdu pool. I'm getting ~2.3 Gb/s using netcat again.
 

eexodus

Dabbler
Joined
Aug 31, 2016
Messages
39
Unfortunately I am still experiencing loss of dataset options (Windows and Mac datasets are defaulting to UNIX still) even after using additional flags (send -R and recv -Fdu) on the initial zfs send recv over netcat.
 
Status
Not open for further replies.
Top