Replication abnormally slooow

[Verea1975]

Hi guys!

I have two servers with a 12 core xeon CPU, 64Gb RAM and 24x 4TB hard drive. They are divided into 4 vdevs of 6 hdd in raidz1 on both servers.
On both I installed freenas. The first is in 11.0 and has a dataset of 50TB that I want to replicate to the second server freshly installed in 11.2.

The two servers are connected in 10GB by fiber optic and with iperf I observe many flows approaching 10Gb / s in sending and receiving.

I created a snapshot job, then a replication job to the new server, but the data only goes to 1GB / minute ....

With scp I have 93MB / s

How should I transfer the 50TB? How to optimize these flows (it would take me more than a month at this rate ... it's impossible).

 

[Johnny Fartpants]

Can we see your replication config and also what are the system processes like on both send and receive machines i.e. is any process maxing out?

 

[Mlovelace]

Hi guys!

I have two servers with a 12 core xeon CPU, 64Gb RAM and 24x 4TB hard drive. They are divided into 4 vdevs of 6 hdd in raidz1 on both servers.
On both I installed freenas. The first is in 11.1 and has a dataset of 50TB that I want to replicate to the second server freshly installed in 11.2.

The two servers are connected in 10GB by fiber optic and with iperf I observe many flows approaching 10Gb / s in sending and receiving.

I created a snapshot job, then a replication job to the new server, but the data only goes to 1GB / minute ....

With scp I have 93MB / s

How should I transfer the 50TB? How to optimize these flows (it would take me more than a month at this rate ... it's impossible).

If the replication is not going across a WAN connection, which it sounds like it's not in your case, you should send the replication stream through netcat. Take a look at an earlier post I made about using netcat for transferring datasets. https://www.ixsystems.com/community/threads/replication-stream-compression.73563/post-509985

 

[Verea1975]

Great! Thank you for your quick answers :)

Sorry for my english.... ;)

I've do this:

zfs send pool/dataset@relocate | pv -b | nc -w 20 XX.XX.XX.XX 8023

nc -w 20 -l 8023 | mbuffer -q -s 1024k -m 1G | pv -rtab | zfs receive -vF pool/dataset

On sender I did not put "mbuffer -q -s 1024k -m 1G" because with this version of freenas (11.0) mbuffer is not available.

I did a test and transfers are between 300 and 400MB / s, it's great! On the other hand, the transfer is cut before the end :( For example, I launched it Friday night, and this morning when I arrive I see that it stopped at about 40TB. Here are the messages I see on the machine sending datas (Freenas 11.1)

[timestamp xxxxxx] - Aggregation plugin unable to read the current rate of "xxxxxxxx.xxx.xxx/cpu-1/cpu-user"
[timestamp xxxxxx] - Utils-vl-lookup The user object callback failed with status 2


[timestamp xxxxxx] - Aggregation plugin unable to read the current rate of "xxxxxxxx.xxx.xxx/cpu-4/cpu-user"
[timestamp xxxxxx] - Utils-vl-lookup The user object callback failed with status 2


[timestamp xxxxxx] - Aggregation plugin unable to read the current rate of "xxxxxxxx.xxx.xxx/cpu-0/cpu-user"
[timestamp xxxxxx] - Utils-vl-lookup The user object callback failed with status 2


[timestamp xxxxxx] - Aggregation plugin unable to read the current rate of "xxxxxxxx.xxx.xxx/cpu-2/cpu-user"
[timestamp xxxxxx] - Utils-vl-lookup The user object callback failed with status 2

etc...

And the transfert is terminated on the receiver server and transfered datas are deleted (because incomplete?)


do you have an idea? or the ability to keep the data transferred to complete with a second try etc. ?

Thank you so much!!!