Using iperf I can show that 10GB at the network layer works fine. However with real data it is only about half of what 10GB can do. Some of this also lies in the process of how we test. IE, mounting an NFS share and running 10 instances of cp is more realistic than showing the iperf theoretical throughput.
I do get some pretty decent numbers though. Nothing wrong at my network layer, but that wasn't without tuning.
iperf -c 10.254.254.253 -i 1 -t 10 -P 8
------------------------------------------------------------
Client connecting to 10.254.254.253, TCP port 5001
TCP window size: 131 KByte (default)
------------------------------------------------------------
[ 8] local 10.254.254.254 port 58032 connected with 10.254.254.253 port 5001
[ 10] local 10.254.254.254 port 58034 connected with 10.254.254.253 port 5001
[ 4] local 10.254.254.254 port 58028 connected with 10.254.254.253 port 5001
[ 5] local 10.254.254.254 port 58029 connected with 10.254.254.253 port 5001
[ 9] local 10.254.254.254 port 58033 connected with 10.254.254.253 port 5001
[ 7] local 10.254.254.254 port 58031 connected with 10.254.254.253 port 5001
[ 3] local 10.254.254.254 port 58027 connected with 10.254.254.253 port 5001
[ 6] local 10.254.254.254 port 58030 connected with 10.254.254.253 port 5001
[ ID] Interval Transfer Bandwidth
[ 8] 0.0- 1.0 sec 193 MBytes 1.62 Gbits/sec
[ 10] 0.0- 1.0 sec 202 MBytes 1.69 Gbits/sec
[ 5] 0.0- 1.0 sec 205 MBytes 1.72 Gbits/sec
[ 9] 0.0- 1.0 sec 203 MBytes 1.70 Gbits/sec
[ 7] 0.0- 1.0 sec 195 MBytes 1.63 Gbits/sec
[ 3] 0.0- 1.0 sec 129 MBytes 1.08 Gbits/sec
[ 6] 0.0- 1.0 sec 199 MBytes 1.67 Gbits/sec
[ 4] 0.0- 1.0 sec 72.0 MBytes 604 Mbits/sec
[SUM] 0.0- 1.0 sec 1.36 GBytes 11.7 Gbits/sec
[ 8] 1.0- 2.0 sec 190 MBytes 1.59 Gbits/sec
[ 10] 1.0- 2.0 sec 204 MBytes 1.71 Gbits/sec
[ 4] 1.0- 2.0 sec 67.6 MBytes 567 Mbits/sec
[ 5] 1.0- 2.0 sec 200 MBytes 1.68 Gbits/sec
[ 9] 1.0- 2.0 sec 202 MBytes 1.70 Gbits/sec
[ 7] 1.0- 2.0 sec 201 MBytes 1.69 Gbits/sec
[ 6] 1.0- 2.0 sec 202 MBytes 1.69 Gbits/sec
[ 3] 1.0- 2.0 sec 124 MBytes 1.04 Gbits/sec
[SUM] 1.0- 2.0 sec 1.36 GBytes 11.7 Gbits/sec
[ 8] 2.0- 3.0 sec 198 MBytes 1.66 Gbits/sec
[ 10] 2.0- 3.0 sec 201 MBytes 1.68 Gbits/sec
[ 5] 2.0- 3.0 sec 197 MBytes 1.65 Gbits/sec
[ 9] 2.0- 3.0 sec 204 MBytes 1.71 Gbits/sec
[ 7] 2.0- 3.0 sec 197 MBytes 1.65 Gbits/sec
[ 3] 2.0- 3.0 sec 126 MBytes 1.06 Gbits/sec
[ 6] 2.0- 3.0 sec 199 MBytes 1.67 Gbits/sec
[ 4] 2.0- 3.0 sec 69.5 MBytes 583 Mbits/sec
[SUM] 2.0- 3.0 sec 1.36 GBytes 11.7 Gbits/sec
[ 8] 3.0- 4.0 sec 192 MBytes 1.61 Gbits/sec
[ 10] 3.0- 4.0 sec 204 MBytes 1.71 Gbits/sec
[ 4] 3.0- 4.0 sec 67.8 MBytes 568 Mbits/sec
[ 5] 3.0- 4.0 sec 204 MBytes 1.71 Gbits/sec
[ 9] 3.0- 4.0 sec 206 MBytes 1.73 Gbits/sec
[ 7] 3.0- 4.0 sec 199 MBytes 1.67 Gbits/sec
[ 3] 3.0- 4.0 sec 124 MBytes 1.04 Gbits/sec
[ 6] 3.0- 4.0 sec 201 MBytes 1.69 Gbits/sec
[SUM] 3.0- 4.0 sec 1.37 GBytes 11.7 Gbits/sec
[ 8] 4.0- 5.0 sec 205 MBytes 1.72 Gbits/sec
[ 10] 4.0- 5.0 sec 207 MBytes 1.74 Gbits/sec
[ 4] 4.0- 5.0 sec 68.6 MBytes 576 Mbits/sec
[ 5] 4.0- 5.0 sec 198 MBytes 1.66 Gbits/sec
[ 9] 4.0- 5.0 sec 211 MBytes 1.77 Gbits/sec
[ 7] 4.0- 5.0 sec 198 MBytes 1.66 Gbits/sec
[ 3] 4.0- 5.0 sec 125 MBytes 1.05 Gbits/sec
[ 6] 4.0- 5.0 sec 196 MBytes 1.64 Gbits/sec
[SUM] 4.0- 5.0 sec 1.38 GBytes 11.8 Gbits/sec
[ 8] 5.0- 6.0 sec 195 MBytes 1.63 Gbits/sec
[ 10] 5.0- 6.0 sec 208 MBytes 1.75 Gbits/sec
[ 4] 5.0- 6.0 sec 68.8 MBytes 577 Mbits/sec
[ 5] 5.0- 6.0 sec 202 MBytes 1.70 Gbits/sec
[ 9] 5.0- 6.0 sec 209 MBytes 1.75 Gbits/sec
[ 7] 5.0- 6.0 sec 202 MBytes 1.69 Gbits/sec
[ 3] 5.0- 6.0 sec 126 MBytes 1.06 Gbits/sec
[ 6] 5.0- 6.0 sec 204 MBytes 1.71 Gbits/sec
[SUM] 5.0- 6.0 sec 1.38 GBytes 11.9 Gbits/sec
[ 8] 6.0- 7.0 sec 193 MBytes 1.62 Gbits/sec
[ 10] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec
[ 4] 6.0- 7.0 sec 69.9 MBytes 586 Mbits/sec
[ 5] 6.0- 7.0 sec 204 MBytes 1.71 Gbits/sec
[ 9] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec
[ 7] 6.0- 7.0 sec 199 MBytes 1.67 Gbits/sec
[ 3] 6.0- 7.0 sec 128 MBytes 1.07 Gbits/sec
[ 6] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec
[SUM] 6.0- 7.0 sec 1.38 GBytes 11.8 Gbits/sec
[ 8] 7.0- 8.0 sec 198 MBytes 1.66 Gbits/sec
[ 10] 7.0- 8.0 sec 208 MBytes 1.74 Gbits/sec
[ 4] 7.0- 8.0 sec 70.2 MBytes 589 Mbits/sec
[ 5] 7.0- 8.0 sec 202 MBytes 1.69 Gbits/sec
[ 9] 7.0- 8.0 sec 203 MBytes 1.70 Gbits/sec
[ 7] 7.0- 8.0 sec 200 MBytes 1.68 Gbits/sec
[ 3] 7.0- 8.0 sec 129 MBytes 1.08 Gbits/sec
[ 6] 7.0- 8.0 sec 202 MBytes 1.70 Gbits/sec
[SUM] 7.0- 8.0 sec 1.38 GBytes 11.8 Gbits/sec
[ 8] 8.0- 9.0 sec 198 MBytes 1.66 Gbits/sec
[ 10] 8.0- 9.0 sec 206 MBytes 1.73 Gbits/sec
[ 4] 8.0- 9.0 sec 67.9 MBytes 569 Mbits/sec
[ 5] 8.0- 9.0 sec 204 MBytes 1.71 Gbits/sec
[ 9] 8.0- 9.0 sec 208 MBytes 1.74 Gbits/sec
[ 7] 8.0- 9.0 sec 201 MBytes 1.69 Gbits/sec
[ 3] 8.0- 9.0 sec 124 MBytes 1.04 Gbits/sec
[ 6] 8.0- 9.0 sec 204 MBytes 1.71 Gbits/sec
[SUM] 8.0- 9.0 sec 1.38 GBytes 11.9 Gbits/sec
[ 8] 9.0-10.0 sec 195 MBytes 1.64 Gbits/sec
[ 8] 0.0-10.0 sec 1.91 GBytes 1.64 Gbits/sec
[ 10] 9.0-10.0 sec 206 MBytes 1.72 Gbits/sec
[ 10] 0.0-10.0 sec 2.00 GBytes 1.72 Gbits/sec
[ 4] 9.0-10.0 sec 70.6 MBytes 592 Mbits/sec
[ 4] 0.0-10.0 sec 693 MBytes 581 Mbits/sec
[ 5] 9.0-10.0 sec 205 MBytes 1.72 Gbits/sec
[ 5] 0.0-10.0 sec 1.97 GBytes 1.69 Gbits/sec
[ 9] 9.0-10.0 sec 205 MBytes 1.72 Gbits/sec
[ 9] 0.0-10.0 sec 2.01 GBytes 1.73 Gbits/sec
[ 7] 9.0-10.0 sec 200 MBytes 1.68 Gbits/sec
[ 7] 0.0-10.0 sec 1.94 GBytes 1.67 Gbits/sec
[ 3] 9.0-10.0 sec 129 MBytes 1.08 Gbits/sec
[ 3] 0.0-10.0 sec 1.23 GBytes 1.06 Gbits/sec
[ 6] 9.0-10.0 sec 204 MBytes 1.71 Gbits/sec
[SUM] 9.0-10.0 sec 1.38 GBytes 11.9 Gbits/sec
[ 6] 0.0-10.0 sec 1.97 GBytes 1.69 Gbits/sec
[SUM] 0.0-10.0 sec 13.7 GBytes 11.8 Gbits/sec