13TB Snapshot Replication Failure

[Sneffets]

Hello,
I have an existing FreeNAS server running the latest 9.4 software (96GB RAM, 6 - 4TB SAS Constellation Drives, 2 - 128GB Enterprise SSD for caching) that is getting full...so I upgraded my existing local BackUp FreeNAS server with 4 - 8TB Seagate Pro drives with 2 - 128GB Enterprise SSD's for caching...and I am trying to complete the initial replication task between the two...it is approximately 13TB, it starts running no problem, sync'ing for hours at about 500Mbps (via a direct connect Mellanox 10Gbps link) then all of the sudden it appears to stop/time out for some reason...the boxes and connections appear to be "ok" meaning the interfaces are intact (I can ping and SSH across them, etc), I don't see any services crashing/failing...but as most are aware, it does not support resume...so it starts over every time. I have read several threads on using rsync with resume, and/or just copying the data manually...but I was hoping that someone could point out where I could possibly see what is actually happening, and fix the issue if one exists?

 

[kdragon75]

Are you using the built in replication or ZFS send/recv from the CLI?

 

[Sneffets]

The built in Replication via the GUI...also the "new" target system in running FreeNAS-11.1-U5...and by accident, the existing box will be upgraded to FreeNAS-11.1-U5 as well...I was going to wait until I had a backup copy of the data prior to upgrading...but at some point I must have clicked update...and it was waiting to for a reboot...so I backed up the config and rebooted...LOL

 

[kdragon75]

The built in Replication via the GUI...also the "new" target system in running FreeNAS-11.1-U5...and by accident, the existing box will be upgraded to FreeNAS-11.1-U5 as well...I was going to wait until I had a backup copy of the data prior to upgrading...but at some point I must have clicked update...and it was waiting to for a reboot...so I backed up the config and rebooted...LOL

Try running plain jain zfs send and receive from CLI. https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html

 

[Sneffets]

The Server is back up and running, I will allow the Replication task to start again, but I assume I am going to have the same issue(s)...trying to see if I can locate where in various logs the error will actually show...

 

[Sneffets]

Try running plain jain zfs send and receive from CLI. https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html

Thank you, I will try this and let you know!

 

[Sneffets]

Try running plain jain zfs send and receive from CLI. https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html

As I stated the Server is backup and running on 11.1-U5 and the GUI based replication task started and ran overnight, so that is good...BUT, the sync is now running at 50% of the original speed...250Mbps verses the 500Mbps...so it would appear that something when going from 9.4 to 11.1 has "cut" the speed in half. This is odd, in initial load testing I could read/write at close to 2Gbps...and when copying to/from my Mac Pro with a EIDE based NAND Flash I can easily max out the Gigabit link (980+Mbps). I haven't really tried copying to/from my Mac while the large replication task is running, since I really want to go back to having a complete copy of my data...but instead of 2-3 days, at this rate it will take almost a week :( It is odd that it is almost exactly 50%...which would normally lead you to some type of duplex setting...BUT I checked and the Mellanox is functioning at 10Gbps with an MTU of 9000 (on both sides)...so it isn't a networking "issue", but something is slowing the send down...and the Replication limits are set to zero...very strange indeed.

 

[Sneffets]

This "could" be the problem...it is showing the SSD cache drives as having a non-native block size, they are showing 512B verses 4096B...which is something I can change after the sync prior to upgrading the drives, add some large data files and initiate a replication task and see if it goes back to "normal"...it would make sense, except for the fact that it was set to 512B before and was sending at almost 500Mbps, but at the same time, it wasn't stable either...

root@freenas:~ # zpool status
pool: DataVault
state: ONLINE
status: One or more devices are configured to use a non-native block size.

Expect reduced performance.
action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool.

scan: scrub repaired 0 in 0 days 10:35:08 with 0 errors on Sun May 27 10:35:09 2018

config:
NAME STATE READ WRITE CKSUM

DataVault ONLINE 0 0 0

raidz2-0 ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

logs

mirror-1 ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0 block size: 512B configured, 4096B native

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0 block size: 512B configured, 4096B native

cache

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0

gptid/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ONLINE 0 0 0


errors: No known data errors

 

[dir_d]

Just to help you not feel crazy when i did a zfs send from my 9.3 FN to my new 11.1 FN my speed was cut in half as well. For some reason that i dont understand the Nas boxes on their own would get full speed to clients but when they talk to each other their speeds were automatically halved.

 

[Sneffets]

Just to help you not feel crazy when i did a zfs send from my 9.3 FN to my new 11.1 FN my speed was cut in half as well. For some reason that i don't understand the Nas boxes on their own would get full speed to clients but when they talk to each other their speeds were automatically halved.

The odd thing was, when I was sending from 9.4 to 11.1 I was getting 500Mbps, BUT after X amount of hours it would drop and/or time out with some error...but the 11.1 to 11.1 seems to be running smooth, but at 50% of the previous rate, despite knowing that the boxes are capable of quite a bit higher send/receive rates....just not sure why..., but thank you...at least I know I am not the only one seeing these oddities :)

 

[PhilipS]

Here are commands I've used from the command line to handle resume if you need it. Change the dataset, snapshot and enter in the receiving IP address.

These are all started from the source machine - I removed the compression, throttle, and pipewatcher commands since you probably won't need them.

Initial send:

Code:

/sbin/zfs send -v -p tank/dataset@backup | /usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "/sbin/zfs receive -F -s 'tank/targetdataset'"

If the send has a failure, to resume (I put this in a script):

Code:

#!/bin/sh
RESUME_TOKEN="$(/usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "zfs get -H -o value receive_resume_token 'tank/targetdataset'")"
/sbin/zfs send -t ${RESUME_TOKEN} -v | /usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "/sbin/zfs receive -F -s 'tank/targetdataset'"

 

[Sneffets]

Here are commands I've used from the command line to handle resume if you need it. Change the dataset, snapshot and enter in the receiving IP address.

These are all started from the source machine - I removed the compression, throttle, and pipewatcher commands since you probably won't need them.

Initial send:

Code:

/sbin/zfs send -v -p tank/dataset@backup | /usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "/sbin/zfs receive -F -s 'tank/targetdataset'"

If the send has a failure, to resume (I put this in a script):

Code:

#!/bin/sh
RESUME_TOKEN="$(/usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "zfs get -H -o value receive_resume_token 'tank/targetdataset'")"
/sbin/zfs send -t ${RESUME_TOKEN} -v | /usr/local/bin/ssh -i /data/ssh/replication -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=7 -p 22 x.x.x.x "/sbin/zfs receive -F -s 'tank/targetdataset'"

Thank you, I will certianly test this out!

 

[Sneffets]

Update:
After spending much time reviewing everyone's information above (Thank you) and combining some existing information found on the Web (thank you google)...data is currently flowing between boxes averaging 1.7Gbps (verses 250Mbps) using nothing more than a combination of zfs send/receive and netcat (nc)...here are the commands I used...next I would love to "edit" the GUI based Replication commands to "perform" these actions...and see if I can get the "resume" feature above incorporated...worst case cron job with a script...but hey might as well shoot for the stars! I will let you know when/IF I get the entire 31.1TB moved over using this method...but it should only take about 21 hours verses 5.5 days (at the 250Mbps)...so I should know sooner than later...thank you again for everyones assistance!

Run this first on the Destination: nc -w 60 -l 3333 | zfs receive -F BackupData/Replication/Vault1
Run this second on the Source (within 60 seconds or increase the wait time above): zfs send DataVault/Vault1@auto-20180627.1301-5d | nc -w 20 172.19.2.2 3333

 

[PhilipS]

If your transfer fails and you want to allow resume with your above commands - simply add a -s option to the zfs receive command.

If it fails, then restart by obtaining the resume token off the destination zfs get -H -o value receive_resume_token 'BackupData/Replication/Vault1'
and use it on the source: zfs send -t [token from above]

 

[Sneffets]

If your transfer fails and you want to allow resume with your above commands - simply add a -s option to the zfs receive command.

If it fails, then restart by obtaining the resume token off the destination zfs get -H -o value receive_resume_token 'BackupData/Replication/Vault1'
and use it on the source: zfs send -t [token from above]

Awesome, thank you!!!

 

[Sneffets]

It is interesting...it did slow down to 250-300Mbps...and I "know" why...just don't know enough about it to see if there is anything I can do to speed it back up...right when I saw the network throughput drop...I checked the various disk usage, cache, swap, etc...and nothing was really "maxed" out...but the ARC Requests dropped at the exact same time...and sped back up when the requests increased....I am assuming there might be something that I can "tweak" to optimize this? Googling and searching the forum(s) right now...

 

[Sneffets]

From what I have read, there might not be much I can do to "increase" this...as it is my understanding from reading, this is "simply" based on what metadata/data is being transferred...IF it is already in the ARC Cache...it will be a lot faster than the data being copied that is not (meaning it is pulling from the spinning hard drives)??????

 

[Sneffets]

Just a quick update...the above zfs and nc combination worked...and I was able to sync all 13.1TB in just under two days. I reconfigured and enabled the old GUI based Replication tasks, they detected the existing snapshots and are functioning as expected moving forward...so if you are looking to move a lot of data between two local FreeNAS boxes...

Run this first on the Destination: nc -w 60 -l 3333 | zfs receive -F BackupData/Replication/Vault1 'replace with yout destination file path this was mine' (the -w is the amount of time it will listen before closing the connection)
Run this second on the Source (within 60 seconds or increase the wait time above): zfs send DataVault/Vault1@auto-20180627.1301-5d | nc -w 20 172.19.2.2 3333

You can find the list of existing snapshots by running 'zfs list -t snapshot' (or the GUI of course)

For anyone newer to FreeNAS (use my actual paths above and IP's for an example above):
zfs send 'FilePath and full name of snapshot' | nc -w 20 'IP of destination FreeNAS' 'port number' (same as you set to listen on the destination box, I just chose 3333, but it could be any non-active port)

I ended up not need to resume, but PhillipS kindly shared above...using the info below (again Thank you PhillipS)...you can retrieve the resume token from the destination host, and use it to restart/resume the transfer

On Destination Host: zfs get -H -o value receive_resume_token 'full path to partial file'
On Source Host: zfs send -t [token from above]

Thank you to everyone for their help, on this Forum as well as on the Internet and other Forums.