FN11, 10 & 40Gbe and dismal performance with replication

Status
Not open for further replies.

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
So, out of curiosity I wanted to see how replication performance was with the 9.10.2-U5 version. It appears to be better than what I was getting with 11.0-U1. The transfer rate was about 3Gbit/sec and the CPU usage was a bit lower. I'll leave these filers on the freeNAS 9.10 version for a couple weeks and see how it looks over time. One bonus is I don't get the false "replication failed" errors with the older build of freeNAS.
CPU.PNG
Network.PNG
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
@Mlovelace do you know why you are getting false "replication failed" errors? You shouldn't be getting any false replication errors, and that can be a problem for people that are using the new alert system in FN11...
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
@Mlovelace do you know why you are getting false "replication failed" errors? You shouldn't be getting any false replication errors, and that can be a problem for people that are using the new alert system in FN11...
I summited a ticket and uploaded logs, but I haven't seen progress on the ticket in... 22 days now. The replication does in fact work, it reports as failed and emails as such, and a short time later changes to "up to date". I confirmed the files match on both the source and target systems.

https://bugs.freenas.org/issues/23837#change-132161
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
@Mlovelace I'm sending you a PM so we don't clutter this thread with your issue.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
OK, so more testing and more weirdness.

So this is what my speed is running replication between my two servers:

screen 2017-07-20 at 2.52.15 PM.jpg

[FreeNAS1]-------------10Gbe-------------[FreeNAS2]


No matter what I do, no matter which cards I use (I have tested Intel 10Gbe, Chelsio 10Gbe and Chelsio 40Gbe cards) this is the best I can get utilizing replication.


So then I decided, HEY - why not just use rsync locally (so no ssh). So I mounted an NFS share from the backup server to the primary server and ran this command:

rsync -aAX --numeric-ids --progress --no-whole-file /mnt/vol1/media/ /mnt/plexnas-ii/

and was NOT impressed:

screen 2017-07-20 at 2.56.44 PM.jpg

I went from 900Mbit/s to about 600Mbits/s. YIKES.

[FreeNAS1]-------------10Gbe-------------[FreeNAS2]


OK, so what next....?

So I decided to try something else altogether. I have a media management server that is connected via Chelsio 10Gbe cards to both FreeNAS servers. On that server, I mount an NFS share from each FreeNAS server and ran the following command:

rsync -aAX --numeric-ids --progress --no-whole-file /mount/media/ /mount/plexnas-ii/


[FreeNAS1]--------10Gbe--------[Media Server]----------10Gbe---------[FreeNAS2]


screen 2017-07-20 at 3.44.08 PM.jpg

Almost 2GBits/s!!


OK, so why in the world would running rsync running through a third machine be faster than running rsync on one of the actual FreeNAS servers? The CPU load on the FreeNAS servers while the rsync is running on them is not very high so I do not think that is it?

screen 2017-07-20 at 3.45.11 PM.jpg

screen 2017-07-20 at 3.45.26 PM.jpg


I am looking for ANY ideas to try to try and figure this issue out.
 

Attachments

  • screen 2017-07-20 at 3.45.26 PM.jpg
    screen 2017-07-20 at 3.45.26 PM.jpg
    108.1 KB · Views: 292
Status
Not open for further replies.
Top