Performance of 40Gb network with FreeNAS

[skyyxy]

Both, AFAIK.

Thank you so much,I will try that:)

 

[bigphil]

I'm curious what driver they have baked in to both versions. The latest is 1.7.12. You also need to make sure your firmware is up to 5.05 on the xl710. If you wouldn't mind posting the output of "sysctl -a | grep dev.ixl.0.%desc" and which version of FreeNAS you're running, I'd be curios to see.

 

[skyyxy]

I'm curious what driver they have baked in to both versions. The latest is 1.7.12. You also need to make sure your firmware is up to 5.05 on the xl710. If you wouldn't mind posting the output of "sysctl -a | grep dev.ixl.0.%desc" and which version of FreeNAS you're running, I'd be curios to see.

I desnt check the firmware yet, so I cant tell you about it.
The XL710 was back to my suppiler because not working good under freenas atleast 9.10.2 before u5, same speed like x520 10Gb., but I can buy it again :) and show the informations here.
Thanks alot.

 

[indy]

FreeNAS 11.0-RELEASE is shipping with driver version 1.6.6-k for the XL710 series.

My connection is far from running at full speed atm but from my experience the first thing you should look into is interrupt moderation.

 

[skyyxy]

FreeNAS 11.0-RELEASE is shipping with driver version 1.6.6-k for the XL710 series.

My connection is far from running at full speed atm but from my experience the first thing you should look into is interrupt moderation.

Thanks a lot. But how I can check the interrupt moderaton? thanks

 

[skyyxy]

FreeNAS 11.0-RELEASE is shipping with driver version 1.6.6-k for the XL710 series.

My connection is far from running at full speed atm but from my experience the first thing you should look into is interrupt moderation.

Also, Please let me know, how fast you ran? 3.xGB/s or 4.xGB/s ?? Thanks a lot

 

[micahtangelo]

hate to necro but 40gbe is something we're operating on, and seeing some similar numbers but with chelsio cards, so wanted to see if you got any further @skyyxy ?

 

[skyyxy]

hate to necro but 40gbe is something we're operating on, and seeing some similar numbers but with chelsio cards, so wanted to see if you got any further @skyyxy ?

I not tested yet, because too busy hahaha. If you have any idea or result with it ? Thanks .

 

[Donny Davis]

I can test with my mellanox cards tomorrow to see what I get.

 

[skyyxy]

I can test with my mellanox cards tomorrow to see what I get.

Thats cool .look forward with it.

 

[Donny Davis]

It would seem the network is not my bottleneck. I did have to do some tuning, but I see nowhere near 40GB. However like I said, I don't really have any equipment that can do 4GB per second reads or writes.
Here is my thread for 10/40GB networking
https://forums.freenas.org/index.php?threads/very-poor-10gbe-performance.60677/page-2

 

[skyyxy]

It would seem the network is not my bottleneck. I did have to do some tuning, but I see nowhere near 40GB. However like I said, I don't really have any equipment that can do 4GB per second reads or writes.
Here is my thread for 10/40GB networking
https://forums.freenas.org/index.php?threads/very-poor-10gbe-performance.60677/page-2

Looking great, may be you can try the real 40Gb network speed by AJA SYSTEM TEST. Thanks

 

[Donny Davis]

Using iperf I can show that 10GB at the network layer works fine. However with real data it is only about half of what 10GB can do. Some of this also lies in the process of how we test. IE, mounting an NFS share and running 10 instances of cp is more realistic than showing the iperf theoretical throughput.

I do get some pretty decent numbers though. Nothing wrong at my network layer, but that wasn't without tuning.

iperf -c 10.254.254.253 -i 1 -t 10 -P 8

------------------------------------------------------------

Client connecting to 10.254.254.253, TCP port 5001

TCP window size: 131 KByte (default)

------------------------------------------------------------

[ 8] local 10.254.254.254 port 58032 connected with 10.254.254.253 port 5001

[ 10] local 10.254.254.254 port 58034 connected with 10.254.254.253 port 5001

[ 4] local 10.254.254.254 port 58028 connected with 10.254.254.253 port 5001

[ 5] local 10.254.254.254 port 58029 connected with 10.254.254.253 port 5001

[ 9] local 10.254.254.254 port 58033 connected with 10.254.254.253 port 5001

[ 7] local 10.254.254.254 port 58031 connected with 10.254.254.253 port 5001

[ 3] local 10.254.254.254 port 58027 connected with 10.254.254.253 port 5001

[ 6] local 10.254.254.254 port 58030 connected with 10.254.254.253 port 5001

[ ID] Interval Transfer Bandwidth

[ 8] 0.0- 1.0 sec 193 MBytes 1.62 Gbits/sec

[ 10] 0.0- 1.0 sec 202 MBytes 1.69 Gbits/sec

[ 5] 0.0- 1.0 sec 205 MBytes 1.72 Gbits/sec

[ 9] 0.0- 1.0 sec 203 MBytes 1.70 Gbits/sec

[ 7] 0.0- 1.0 sec 195 MBytes 1.63 Gbits/sec

[ 3] 0.0- 1.0 sec 129 MBytes 1.08 Gbits/sec

[ 6] 0.0- 1.0 sec 199 MBytes 1.67 Gbits/sec

[ 4] 0.0- 1.0 sec 72.0 MBytes 604 Mbits/sec

[SUM] 0.0- 1.0 sec 1.36 GBytes 11.7 Gbits/sec

[ 8] 1.0- 2.0 sec 190 MBytes 1.59 Gbits/sec

[ 10] 1.0- 2.0 sec 204 MBytes 1.71 Gbits/sec

[ 4] 1.0- 2.0 sec 67.6 MBytes 567 Mbits/sec

[ 5] 1.0- 2.0 sec 200 MBytes 1.68 Gbits/sec

[ 9] 1.0- 2.0 sec 202 MBytes 1.70 Gbits/sec

[ 7] 1.0- 2.0 sec 201 MBytes 1.69 Gbits/sec

[ 6] 1.0- 2.0 sec 202 MBytes 1.69 Gbits/sec

[ 3] 1.0- 2.0 sec 124 MBytes 1.04 Gbits/sec

[SUM] 1.0- 2.0 sec 1.36 GBytes 11.7 Gbits/sec

[ 8] 2.0- 3.0 sec 198 MBytes 1.66 Gbits/sec

[ 10] 2.0- 3.0 sec 201 MBytes 1.68 Gbits/sec

[ 5] 2.0- 3.0 sec 197 MBytes 1.65 Gbits/sec

[ 9] 2.0- 3.0 sec 204 MBytes 1.71 Gbits/sec

[ 7] 2.0- 3.0 sec 197 MBytes 1.65 Gbits/sec

[ 3] 2.0- 3.0 sec 126 MBytes 1.06 Gbits/sec

[ 6] 2.0- 3.0 sec 199 MBytes 1.67 Gbits/sec

[ 4] 2.0- 3.0 sec 69.5 MBytes 583 Mbits/sec

[SUM] 2.0- 3.0 sec 1.36 GBytes 11.7 Gbits/sec

[ 8] 3.0- 4.0 sec 192 MBytes 1.61 Gbits/sec

[ 10] 3.0- 4.0 sec 204 MBytes 1.71 Gbits/sec

[ 4] 3.0- 4.0 sec 67.8 MBytes 568 Mbits/sec

[ 5] 3.0- 4.0 sec 204 MBytes 1.71 Gbits/sec

[ 9] 3.0- 4.0 sec 206 MBytes 1.73 Gbits/sec

[ 7] 3.0- 4.0 sec 199 MBytes 1.67 Gbits/sec

[ 3] 3.0- 4.0 sec 124 MBytes 1.04 Gbits/sec

[ 6] 3.0- 4.0 sec 201 MBytes 1.69 Gbits/sec

[SUM] 3.0- 4.0 sec 1.37 GBytes 11.7 Gbits/sec

[ 8] 4.0- 5.0 sec 205 MBytes 1.72 Gbits/sec

[ 10] 4.0- 5.0 sec 207 MBytes 1.74 Gbits/sec

[ 4] 4.0- 5.0 sec 68.6 MBytes 576 Mbits/sec

[ 5] 4.0- 5.0 sec 198 MBytes 1.66 Gbits/sec

[ 9] 4.0- 5.0 sec 211 MBytes 1.77 Gbits/sec

[ 7] 4.0- 5.0 sec 198 MBytes 1.66 Gbits/sec

[ 3] 4.0- 5.0 sec 125 MBytes 1.05 Gbits/sec

[ 6] 4.0- 5.0 sec 196 MBytes 1.64 Gbits/sec

[SUM] 4.0- 5.0 sec 1.38 GBytes 11.8 Gbits/sec

[ 8] 5.0- 6.0 sec 195 MBytes 1.63 Gbits/sec

[ 10] 5.0- 6.0 sec 208 MBytes 1.75 Gbits/sec

[ 4] 5.0- 6.0 sec 68.8 MBytes 577 Mbits/sec

[ 5] 5.0- 6.0 sec 202 MBytes 1.70 Gbits/sec

[ 9] 5.0- 6.0 sec 209 MBytes 1.75 Gbits/sec

[ 7] 5.0- 6.0 sec 202 MBytes 1.69 Gbits/sec

[ 3] 5.0- 6.0 sec 126 MBytes 1.06 Gbits/sec

[ 6] 5.0- 6.0 sec 204 MBytes 1.71 Gbits/sec

[SUM] 5.0- 6.0 sec 1.38 GBytes 11.9 Gbits/sec

[ 8] 6.0- 7.0 sec 193 MBytes 1.62 Gbits/sec

[ 10] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec

[ 4] 6.0- 7.0 sec 69.9 MBytes 586 Mbits/sec

[ 5] 6.0- 7.0 sec 204 MBytes 1.71 Gbits/sec

[ 9] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec

[ 7] 6.0- 7.0 sec 199 MBytes 1.67 Gbits/sec

[ 3] 6.0- 7.0 sec 128 MBytes 1.07 Gbits/sec

[ 6] 6.0- 7.0 sec 206 MBytes 1.73 Gbits/sec

[SUM] 6.0- 7.0 sec 1.38 GBytes 11.8 Gbits/sec

[ 8] 7.0- 8.0 sec 198 MBytes 1.66 Gbits/sec

[ 10] 7.0- 8.0 sec 208 MBytes 1.74 Gbits/sec

[ 4] 7.0- 8.0 sec 70.2 MBytes 589 Mbits/sec

[ 5] 7.0- 8.0 sec 202 MBytes 1.69 Gbits/sec

[ 9] 7.0- 8.0 sec 203 MBytes 1.70 Gbits/sec

[ 7] 7.0- 8.0 sec 200 MBytes 1.68 Gbits/sec

[ 3] 7.0- 8.0 sec 129 MBytes 1.08 Gbits/sec

[ 6] 7.0- 8.0 sec 202 MBytes 1.70 Gbits/sec

[SUM] 7.0- 8.0 sec 1.38 GBytes 11.8 Gbits/sec

[ 8] 8.0- 9.0 sec 198 MBytes 1.66 Gbits/sec

[ 10] 8.0- 9.0 sec 206 MBytes 1.73 Gbits/sec

[ 4] 8.0- 9.0 sec 67.9 MBytes 569 Mbits/sec

[ 5] 8.0- 9.0 sec 204 MBytes 1.71 Gbits/sec

[ 9] 8.0- 9.0 sec 208 MBytes 1.74 Gbits/sec

[ 7] 8.0- 9.0 sec 201 MBytes 1.69 Gbits/sec

[ 3] 8.0- 9.0 sec 124 MBytes 1.04 Gbits/sec

[ 6] 8.0- 9.0 sec 204 MBytes 1.71 Gbits/sec

[SUM] 8.0- 9.0 sec 1.38 GBytes 11.9 Gbits/sec

[ 8] 9.0-10.0 sec 195 MBytes 1.64 Gbits/sec

[ 8] 0.0-10.0 sec 1.91 GBytes 1.64 Gbits/sec

[ 10] 9.0-10.0 sec 206 MBytes 1.72 Gbits/sec

[ 10] 0.0-10.0 sec 2.00 GBytes 1.72 Gbits/sec

[ 4] 9.0-10.0 sec 70.6 MBytes 592 Mbits/sec

[ 4] 0.0-10.0 sec 693 MBytes 581 Mbits/sec

[ 5] 9.0-10.0 sec 205 MBytes 1.72 Gbits/sec

[ 5] 0.0-10.0 sec 1.97 GBytes 1.69 Gbits/sec

[ 9] 9.0-10.0 sec 205 MBytes 1.72 Gbits/sec

[ 9] 0.0-10.0 sec 2.01 GBytes 1.73 Gbits/sec

[ 7] 9.0-10.0 sec 200 MBytes 1.68 Gbits/sec

[ 7] 0.0-10.0 sec 1.94 GBytes 1.67 Gbits/sec

[ 3] 9.0-10.0 sec 129 MBytes 1.08 Gbits/sec

[ 3] 0.0-10.0 sec 1.23 GBytes 1.06 Gbits/sec

[ 6] 9.0-10.0 sec 204 MBytes 1.71 Gbits/sec

[SUM] 9.0-10.0 sec 1.38 GBytes 11.9 Gbits/sec

[ 6] 0.0-10.0 sec 1.97 GBytes 1.69 Gbits/sec

[SUM] 0.0-10.0 sec 13.7 GBytes 11.8 Gbits/sec