OK, so more testing and more weirdness.
So this is what my speed is running replication between my two servers:
[FreeNAS1]-------------10Gbe-------------[FreeNAS2]
No matter what I do, no matter which cards I use (I have tested Intel 10Gbe, Chelsio 10Gbe and Chelsio 40Gbe cards) this is the best I can get utilizing replication.
So then I decided, HEY - why not just use rsync locally (so no ssh). So I mounted an NFS share from the backup server to the primary server and ran this command:
rsync -aAX --numeric-ids --progress --no-whole-file /mnt/vol1/media/ /mnt/plexnas-ii/
and was NOT impressed:
I went from 900Mbit/s to about 600Mbits/s. YIKES.
[FreeNAS1]-------------10Gbe-------------[FreeNAS2]
OK, so what next....?
So I decided to try something else altogether. I have a media management server that is connected via Chelsio 10Gbe cards to both FreeNAS servers. On that server, I mount an NFS share from each FreeNAS server and ran the following command:
rsync -aAX --numeric-ids --progress --no-whole-file /mount/media/ /mount/plexnas-ii/
[FreeNAS1]--------10Gbe--------[Media Server]----------10Gbe---------[FreeNAS2]
Almost 2GBits/s!!
OK, so why in the world would running rsync running through a third machine be faster than running rsync on one of the actual FreeNAS servers? The CPU load on the FreeNAS servers while the rsync is running on them is not very high so I do not think that is it?
I am looking for ANY ideas to try to try and figure this issue out.