ZFS Replication Failed... wait no? umm maybe sorta kinda?

[depasseg]

Yes but I think I have to add the key to that file which I do not know how to do. I guess I need to google how to edit a file via CLI in FreeBSD. Once I get in the PULL system I need to paste the PUSH systems SSH key there.

RTFM, edit the user in freenas account settings.

"Go to PULL and click Account ‣ Users ‣ View Users. Click the “Modify User” button for the user account you will be using for replication (by default this is the root user). Paste the copied key into the “SSH Public Key” field and click “OK”. If a key already exists, append the new text after the existing key."


I don't know how to make this section of the directions any clearer.

 

[Fuganater]

Ya it still fails.

Hello,
The replication failed for the local ZFS Vol1 while attempting to
send snapshot auto-20151224.1845-2w to 192.168.2.11

 

[depasseg]

I feel like we are having the same conversation over and over. Try the cli ssh test.

Sent from my Nexus 6P using Tapatalk

 

[Fuganater]

Here is what I put in my Root User's SSH Public Key

I figured out how to add the other key to the authorized_keys file. Had to use 'ee' to edit it.

The SSH CLI test goes through now without asking me for a PW but the replication task still fails.


auth.log on PULL shows this so I think it is authenticating ok.

Dec 31 00:17:02 R2D2 sshd[28480]: Accepted publickey for root from 192.168.2.10 port 53511 ssh2: RSA 70:06:b6:26:a9:6a:b8:f9:8c:5a:18:87:84:e6:54:9c
Dec 31 00:17:02 R2D2 sshd[28480]: Received disconnect from 192.168.2.10: 11: disconnected by user

No idea if this is causing problems.

Code:

ec 30 01:08:09 C3PO nmbd[3413]: [2015/12/30 01:08:09.818850,  0] ../source3/nmbd/nmbd_become_lmb.c:397(become_local_master_stage2)
Dec 30 01:08:09 C3PO nmbd[3413]:   *****
Dec 30 01:08:09 C3PO nmbd[3413]:
Dec 30 01:08:09 C3PO nmbd[3413]:   Samba name server C3PO is now a local master browser for workgroup WORKGROUP on subnet 192.168.2.10
Dec 30 01:08:09 C3PO nmbd[3413]:
Dec 30 01:08:09 C3PO nmbd[3413]:   *****
Dec 30 01:08:09 C3PO nmbd[3413]: [2015/12/30 01:08:09.819009,  0] ../source3/nmbd/nmbd_incomingdgrams.c:311(process_local_master_announce)
Dec 30 01:08:09 C3PO nmbd[3413]:   process_local_master_announce: Server R2D2 at IP 192.168.2.11 is announcing itself as a local master browser for workgroup WORKGROUP and we think we are master. Forcin
Dec 30 01:08:09 C3PO nmbd[3413]: [2015/12/30 01:08:09.819096,  0] ../source3/nmbd/nmbd_become_lmb.c:150(unbecome_local_master_success)
Dec 30 01:08:09 C3PO nmbd[3413]:   *****
Dec 30 01:08:09 C3PO nmbd[3413]:
Dec 30 01:08:09 C3PO nmbd[3413]:   Samba name server C3PO has stopped being a local master browser for workgroup WORKGROUP on subnet 192.168.2.10
Dec 30 01:08:09 C3PO nmbd[3413]:
Dec 30 01:08:09 C3PO nmbd[3413]:   *****
Dec 30 01:08:25 C3PO nmbd[3413]: [2015/12/30 01:08:25.836985,  0] ../source3/nmbd/nmbd_become_lmb.c:418(become_local_master_fail2)
Dec 30 01:08:25 C3PO nmbd[3413]:   become_local_master_fail2: failed to register name WORKGROUP<1d> on subnet 192.168.2.10. Failed to become a local master browser.

I looked in the messages log on PUSH and I don't see any errors authenticating persay.

 

[Fuganater]

I rebooted both machines for shits and giggles and I am now getting this error.

Replication Vol1 -> 192.168.2.11:Backup failed: Please move system dataset of remote side to another pool.

The 'Backup' dataset is empty with 15TB free space.

 

[depasseg]

You cannot replicate into a dataset that has a .system dataset in it.

Sent from my Nexus 6P using Tapatalk

 

[toadman]

I rebooted both machines for shits and giggles and I am now getting this error.

Replication Vol1 -> 192.168.2.11:Backup failed: Please move system dataset of remote side to another pool.

The 'Backup' dataset is empty with 15TB free space.

On the backup server when you go to System > System Dataset and look at "System dataset pool" what does it say?

As depasseg says you should have the system dataset somewhere other than the pool you are replicating to. Either put it on the boot pool or some other pool you have on the server.

 

[depasseg]

Or create a replication pull dataset. You just can't replicate into the root dataset that contains the system.

 

[Fuganater]

Finally got it working! Ok I didn't know the system couldn't be in that dataset. I had just created a RAIDZ2 pool called Backup and the dataset was called Backup. I switched it and called the pool Vol1 and created another dataset called Backup. The task successfully completed.

Now it looks like it only replicates the snapshot. Will it eventually be a exact copy of the primary server?

 

[Ericloewe]

Finally got it working! Ok I didn't know the system couldn't be in that dataset. I had just created a RAIDZ2 pool called Backup and the dataset was called Backup. I switched it and called the pool Vol1 and created another dataset called Backup. The task successfully completed.

Now it looks like it only replicates the snapshot. Will it eventually be a exact copy of the primary server?

Replication transfers snapshots - that is, the metadata and any data referenced.

Replication can only be as up-to-date as your latest snapshot.

 

[Fuganater]

Is there anway to make to snapshot the entire Dataset? The whole point in me doing this was to have 2 copies of my data on the network.

 

[depasseg]

Is there anway to make to snapshot the entire Dataset? The whole point in me doing this was to have 2 copies of my data on the network.

If you used the gui to create the replication task, then the entire dataset will be exactly the same including all snapshots. And every time a new snapshot is taken, replication (assuming it's enabled) will push the snapshot to the pull system.

 

[depasseg]

Keep an eye on the network and dataset utilization graphs and totals. It's possible that it is still running.

 

[Fuganater]

Keep an eye on the network and dataset utilization graphs and totals. It's possible that it is still running.

Took a few hours but it is now moving data. The replication task said it was successful but nothing was moving.

 

[Fuganater]

Sorry to beat this horse dead but it looked like data was moving but it is not.

On the PUSH system

On the PULL system

Pull system drive activity

Storage Snapshot

So looking at what is on the PULL system there is only 33GB of data from the PUSH system... There should be 10TB.

 

[depasseg]

You are only pushing 200Mbit/sec. If you are on the same LAN, and you notice that your CPU usage is fully pegged, I'd recommend disabling SSH encryption and compression.

Post screenshots of the CPU usage and the volume/dataset usage.

Just give it time.

 

[Fuganater]

CPU on PUSH

CPU on PULL

Oh snap... I think I see what is going on. I have it snapping Vol1 which apparently does not include any of the sub datasets. Is that right??

On PUSH

I have several datasets under Vol1

On PULL

 

[depasseg]

Oh snap... I think I see what is going on. I have it snapping Vol1 which apparently does not include any of the sub datasets. Is that right??

That is certainly possible. Replication only replicates snapshots. So you need to make sure that the snapshots on Push are set up correctly. And then ensure that the replication tasks contain the proper dataset tasks. The simplest way is to enable the child datasets options. But that really only is feasible if you want to back up everything.