ZFS Replication

[Mark Rodgers]

Hi FreeNAS,

I spent some time setting up ZFS replication from our FreeNAS to our backup FreeNAS. The files (/mnt/share - ~470GB) started transferring yesterday at noon. Judging by the network report graph in the UI on the backup system the files stopped transferring around 1:13am.

The only concern I have is that /mnt/share on FreeNAS1 is ~470GB while the backup FreeNAS transfered the data into /mnt/share but is only ~330GB. Is this normal or does it seem that the transfer didn't complete? Snapshots are now showing in the web UI on the backup, which according to the FreeNAS docs happens after all the data has been transferred.

EDIT: currently running rsync to diff both directories recursively

Code:

rsync -n -avrc /mnt/share server2@x.x.x.x:/mnt/share > diff.txt

 

[Mlovelace]

You should get a "succeeded" message on the replication task window of the push box. You can always clone the replicated snapshot on the pull box and check the files manually to make sure it worked.

 

[Mark Rodgers]

There is thousands of files on my nas1 box that are having changes to them as we speak. making it very difficult to create a clone to compare the nas2. As for the succeeded message, I don't see this anywhere? Are you talking about in the shell because I used the web ui.

 

[Mlovelace]

 

[Mark Rodgers]

My push box Replication Tasks tab looks completely different, possibly due to it being an outdated version.

push box is on 8.3.0.
pull box is on 9.3.

push box (ZFS REPLICATION):

I also noticed my pull box has a compression ratio of 1.31x, which seems to be pretty accurate after doing the math. If possible I'd still like to be able to confirm these files have all moved over successfully as this is A LOT of important data.

push box (VOLUMES):

pull box (VOLUMES):

EDIT: inb4 we plan to temporarly take push box offline so files can't be changed, do one final zfs replication push for backup, then update the box to 9.3. After the update is (hopefully) successful, it will come back online and we will continue using ZFS Replication as an off-site backup. I just want to confirm all of our data has been backed up incase something did happen during the update process.

 

[Mlovelace]

Yes the difference you're seeing is the 8.3.0 version discrepancy. Are you motivated to upgrade the 8.3.0 box? The 9.3 gives you % complete on the replication task and succeeded notifications. If I were in your shoes I would clone the pull snapshot, mount it as a cifs share and compare the files.

 

[Mark Rodgers]

I setup a cifs share and will try to browse around to look for anything that is possibly missing, as there is a great deal of data on here.

 

[Mlovelace]

I setup a cifs share and will try to browse around to look for anything that is possibly missing, as there is a great deal of data on here.

What OS are you running? Windows you can do the FC \pathtoshare \pathtoshare command and linux has the diff command. I wouldn't try doing a visual comparison.

 

[Mark Rodgers]

Smart. I am on Windows 7.
FC doesn't seem to want to cooperate with my shared folders though.

EDIT: I must be using this wrong because it will not work locally either.

 

[Mlovelace]

create a batch file with this script and change the folder paths to what ever your needs are.

Code:

@echo off
setlocal
set "folder1=c:\path\To\Folder1\Root"
set "folder2=d:\path\To\Folder2\Root"
set "fileMask=*"

for /f "delims=" %%F in (
'echo "."^&forfiles /s /p "%folder1%" /m "%fileMask%" /c "cmd /c if @isdir==TRUE echo @relpath"'
) do fc "%folder1%\%%~F\%fileMask%" "%folder2%\%%~F\*"

edit: I forgot to mention the script will look in subfolders too.

 

[Mark Rodgers]

UNC paths not supported so I mapped them to drives g:\ and t:\ then received another error (screenshot #2).

Note: we also have a Linux machine on the network that I can access and try to diff from there.

screenshot #1

screenshot #2

 

[Mlovelace]

UNC paths do work... I don't know why you have that error.

Your batch script should look like this (I think I got your UNCs right)

Code:

@echo off
setlocal
set "folder1=\\nas1\files-am"
set "folder2=\\192.0.0.31\files-am"
set "fileMask=*"

for /f "delims=" %%F in (
'echo "."^&forfiles /s /p "%folder1%" /m "%fileMask%" /c "cmd /c if @isdir==TRUE echo @relpath"'
) do fc "%folder1%\%%~F\%fileMask%" "%folder2%\%%~F\*"

 

[Mlovelace]

Also, when you upgrade that box from 8.3.0 @cyberjock had a recommended upgrade path which he spelled out in this post:
https://forums.freenas.org/index.ph...over-afp-in-yosemite-10-10.25619/#post-161323

 

[Mark Rodgers]

UNC paths do work... I don't know why you have that error.

Your batch script should look like this (I think I got your UNCs right)

Code:

@echo off
setlocal
set "folder1=\\nas1\files-am"
set "folder2=\\192.0.0.31\files-am"
set "fileMask=*"

for /f "delims=" %%F in (
'echo "."^&forfiles /s /p "%folder1%" /m "%fileMask%" /c "cmd /c if @isdir==TRUE echo @relpath"'
) do fc "%folder1%\%%~F\%fileMask%" "%folder2%\%%~F\*"

correct. the script just wont take it. lol.
Thanks for the upgrading tip btw ^

 

[Mark Rodgers]

Decided to run this snippet of code on both NAS boxes and then run a diff on the two lists. Hopefully this works out.

Code:

ls -R /mnt/share | awk '
/:$/&&f{s=$0;f=0}
/:$/&&!f{sub(/:$/,"");s=$0;f=1;next}
NF&&f{ print s"/"$0 }'

Source: http://stackoverflow.com/questions/1767384/ls-command-how-can-i-get-a-recursive-full-path-listing-one-line-per-file

 

[JayG30]

Can you compare how many snapshots show on each box?

My guess is it might be related to the number of snapshots and the size of those snapshots.
Consider running these commands to check http://docs.oracle.com/cd/E19253-01/819-5461/gbcxc/index.html

Code:

zfs list -o space
zfs list -ro space tank/home

 

[Mark Rodgers]

I ran diff on both the file lists generated, which ended up being millions of lines, 1 file per line. There was only a few differences between the live and backup NAS, due to users currently working on files on the FreeNAS push box. Obviously this doesn't check file integrity but at this point I've tested a few files manually and have trust in FreeNAS' replication task that the files all transferred w/o corruption.

Next goal is to take the live FreeNAS box offline to update to the latest stable release. Thanks for all the help Mlovelace!