Replication bottleneck?

[DEMONmachine]

Hello,
I'am trying to figure out why is my replication task not fast enough .
And I don't see any problem in my setup.

Setup:
Primary DS: FN 11.2u6 ; 2x E5-2623 v4; 128GB RAM; 24x 1TB Crucial mx300
Backup DS: FN 11.2u7 ; 1x E5-2623 v4; 64GB RAM; 12x 6TB WDred HDD

Network: 10Gb SFP+DAC (dedicated - replication link is not used by anything else)

Configuration:
Pool Sync disabled
Stream compression disabled
Cipher disabled

Measurements:
iperf / netperf hitting almost full network capacity. (up to 9.65Gbps with payload aroud 8.8 Gbps)
Primary DS load: cpu usage (avg 10min) - 20% ; disk - 22% ; network 5 / 10 Gbps
Backup DS load: cpu 40%; disk - 57% ; network ~4.2 / 10 Gbps ; writes 200-450MBps

In this scenario, as I see, the weakest poit is backup HDDs, but there are still a lot of space to speed up.

So what is real bottleneck? What prevent to utilize maximum resources?

Thank you in advance.

 

[dlavigne]

Did you make any progress with this?

 

[SMnasMAN]

what is the layout of your pool on the backup DS? (ie # vDevs / z1 z2 ect).

also this command will help you to see disk IO / disks bottlenecks:
gstat -s -p -o -d

(btw, 4.2gbit is pretty good for replication speed, when using spinning hdds + FN/zfs, in my experience)