Weird Network Performance

[huehnerhose]

Hi!

I am really lost with figuring out what happens here.
My setup:

• Supermicro MBD-X10DRi
• Intel Xeon E5-2609v3
• 64GB DDR4 (ECC)
• Intel X710-DA2 / SFP+ SR Optics 10GBASE-SR (E10GSFPSR) (primary network connection)
  
• 12x WD Red 6TB SATA (2x raidz2 with 6 HDs, then striped)
• FreeNAS-9.3-STABLE-201506292130

Especially read performance via smb/scp/ftp is very poor. At best 8-10MB/s. So I started at the "bottom" and benchmarked the Filesystem via bonnie++. After deactivating compression and working with 100GB test file sizes bonnie postulates this speeds:

Code:

read
528820 Kb/s
write
620058 Kb/s
rewrite
305870 Kb/s

'zpool iostat' shows between 300M/800M while bonnie was running. So I think I am safe here.

I used 2 SSDs as mirror in that system. And tried copying to them from my main pool. Speeds around 300MB/s.

Next step was testing different network service (smb/scp/ftp), but they all cap at around 8-10MB/s. A raw benchmark of network connection via netperf showed 890MBit/s. Since my test target only has a 1GBit/s connection, that seems fine, too.
I was able to find one "glitch" in the network setup, the MTU was at 1500 but in my network only 1472 are working unfragmented. "Correcting" this, didn't change a thing.

Now I do not have any ideas how to find the bottle neck and would be gratefull for any advices!

Thanks a lot and best regards

 

[DifferentStrokes]

I'd start checking your network stuff. 8-10 MB/s put you right at 100Mb/s speeds. Maybe try a known good cable and connect a computer straight to the NAS.

 

[SweetAndLow]

Check cables and also I wouldn't mess with mtu at all. Just let things negotiate what they want. Most networks don't need manual configuration of mtu.

 

[huehnerhose]

Thanks for your answers. But I can't simply connect directly to the server, since it is connected via optical fibre.
Also netperf get's the full gigabit throughput. I really think there has to be some software fuckup in the network stack or something. Is there any way for me to test the different components between some process reading data and that what comes out oft the network port?

 

[DifferentStrokes]

You told us some about the server hardware and setup, can we get some info on the clients that you are seeing the slow speeds.

I know awhile back I had a crossover cable make its way into the network. Even though the switch and NICs auto-negotiated and connected I will still have weird problems. SSH transfers, iperf and replications would saturate the Gb NIC, but Samba and NFS would only run @ 15-16 MB/s. Replaces the cable and problem sloved. 90-95 MB/s on Samba share now.

 

[cyberjock]

I was able to find one "glitch" in the network setup, the MTU was at 1500 but in my network only 1472 are working unfragmented. "Correcting" this, didn't change a thing.

That's not a glitch. That's how networking works. The "missing" bits are because of CRC and other overhead. With an MTU of 1500 the maximum packet you should be able to send unfragmented is 1472 bytes of payload.

 

[huehnerhose]

Ah! Thanks @cyberjock. That part about MTU sizes I got wrong.
@DifferentStrokes: I really doubt the network components outside my server to be the problem. The greater picture: This server is should become a part of the infrastructure of an university department. Therefor it is in university housing. The network is completly under control by the universities NOC, I am not allowed to use any of my network cables etc. inside the housing facility.
As testclients I used my workstation, which is connected to the housing via the whole university network stuff. But primarly I tested with a second server inside housing. The connection between this two servers is completly under NOC-control. The FreeNAS server is connected via 10GBit/s optical fibre, my second server is connected by 1GBit/s copper cable. How these to physical networks are interconnected is outside my knowledge and influence. But this is a "normal" setup for NOC, nothing special about it.

But again: netperf between these to servers measures 890MBit/s, which I think is really ok. So network throughput per se seems to be ok.

Edit: Here are some netperf outputs. I ran the starndard tests compiled in. The streaming performance looks ok for me. I can't evaluate the request/response test:

Code:

user @ XXX.XXX.112.69: ~$ netperf -L XXX.XXX.112.69 -H XXX.XXX.112.90 -p 8080 -t TCP_RR
MIGRATED TCP REQUEST/RESPONSE TEST from XXX.XXX.112.69 (XXX.XXX.112.69) port 0 AF_INET to XXX.XXX.112.90 (XXX.XXX.112.90) port 0 AF_INET : demo : first burst 0
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate        
bytes  Bytes  bytes    bytes   secs.    per sec  

16384  87380  1        1       10.01    1593.38  
32768  65536
user @ XXX.XXX.112.69: ~$ netperf -L XXX.XXX.112.69 -H XXX.XXX.112.90 -p 8080 -t UDP_RR
MIGRATED UDP REQUEST/RESPONSE TEST from XXX.XXX.112.69 (XXX.XXX.112.69) port 0 AF_INET to XXX.XXX.112.90 (XXX.XXX.112.90) port 0 AF_INET : demo : first burst 0
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate        
bytes  Bytes  bytes    bytes   secs.    per sec  

212992 212992 1        1       10.00     139.04  
9216   42080
user @ XXX.XXX.112.69: ~$ netperf -L XXX.XXX.112.69 -H XXX.XXX.112.90 -p 8080 -t UDP_STREAM
MIGRATED UDP STREAM TEST from XXX.XXX.112.69 (XXX.XXX.112.69) port 0 AF_INET to XXX.XXX.112.90 (XXX.XXX.112.90) port 0 AF_INET : demo
Socket  Message  Elapsed      Messages               
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   65507   10.00       17910      0     938.36
42080           10.00           0              0.00

user @ XXX.XXX.112.69: ~$ netperf -L XXX.XXX.112.69 -H XXX.XXX.112.90 -p 8080 -t TCP_STREAM
MIGRATED TCP STREAM TEST from XXX.XXX.112.69 (XXX.XXX.112.69) port 0 AF_INET to XXX.XXX.112.90 (XXX.XXX.112.90) port 0 AF_INET : demo
Recv   Send    Send                         
Socket Socket  Message  Elapsed             
Size   Size    Size     Time     Throughput 
bytes  bytes   bytes    secs.    10^6bits/sec 

65536  16384  16384    10.00     880.19  

 

[huehnerhose]

Hi again,

further investigations finished.
Last hope was to change the NIC. We used an Intel X710-DA2 with an E10GSFPSR. The mainboard has 2 1GBit NICs onboard. We switched to one of those and volià: SCP worked with around 40MBit/s. (Which seemed like the possible maximum limited by the test vm-client).

So, all I need now is to find out why the hell this NIC can't send properly. I think/hope it is a problem with the driver. I will open a ticket and hope this can be solved by some update.