Gigabit Speed Halves when Transmitting and Receiving Simultaneously on Same NIC

[drakohen]

So yeah, I'm not entirely sure what the issue here is. I can transfer at ~110MB/s and then receive at ~110MB/s but the moment I try to transfer and receive simultaneously, my rate drops down to ~55MB/s each way over SMB. This also occurs when one client is transmitting and a different one is receiving files from my FreeNAS server at the same time.

I have already ruled out a Windows/desktop issue because I get ~112MB/s up and down simultaneously with a direct connection between my desktop and laptop. I also get similar speeds when my desktop and laptop are doing the same test when connected to my Cisco SG300-24P Switch which rules that out as well. My server is no slouch being a Supermicro X10SL7-F with an E3-1231 v3, 32GB of RAM, and dual Intel I210 NICs. Also to rule out disk IOPS issues, I setup my second NIC on a different subnet with a direct connection to the second NIC on my desktop. From there I sent files on one subnet and received files on the other, averaged ~105MB/s but suffered the same rate of ~55MB/s when doing a simultaneous transmit and receive exchange. I highly doubt it's a hardware limitation so it must be some sort of misconfiguration unless SMB on FreeNAS isn't the greatest but I think my last test ruled that out. What are you all's take on this situation? I have been unsuccessful in finding any posts anywhere that talk about this same exact issue that I am having when trying to transmit and receive files simultaneously on a singular connection. Any assistance would be greatly appreciated, if you need more information just let me know what you need, and thanks for your time!

 

[Kuhaku]

With the information at hand to me it sounds like you're running half duplex on your server side interfaces, hard-set everything to 1000/full and see how it runs after this instead of letting the NICs trying to auto-negotiate, do it on the switch as well those Cisco small business switches always have their hassles I personally avoid them and stick with Cisco catalyst switches. Post up some reports so the guys here with more knowledge on how FreeNAS handles the SMB protocol might be able to help you out with more in-depth knowledge.

 

[drakohen]

Here's my read out of what my network interfaces are set to. To my knowledge, it looks ok but I don't consider myself an expert. Let me know if something seems off. Thanks.

Show : Console Text

% ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:33:59:c8
inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:33:59:c9
inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:bd:c7:0b:f8:00
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 4125
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:76:c7:5b:57:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000

 

[Kuhaku]

Yeah that looks alright still auto-negotiating so change that to hard set 1000/full to be safe, could you from your client machine do an arp table for your server's IP, and flick that through as well for me, just so I can get a better understanding on how you're connecting to your NAS.

On a Windows machine it is arp -a 192.168.0.100 or 192.168.1.100 depending on which interface you're connected to.

 

[drakohen]

Here you go.

Show : ARP

Interface: 192.168.0.101 --- 0x6
Internet Address Physical Address Type
192.168.0.100 0c-c4-7a-33-59-c9 dynamic

So I think I have narrowed down the issue and it being SMB on the server side of things. I just stood up FTP temporarily and did some massive data movement and I was able to saturate my gigabit connection on my desktop and server when transmitting and receiving simultaneously. Any ideas on if I can boost the throughput of the protocol or am I hitting a limitation?

 

[Kuhaku]

Excellent we can rule out your networking and more target focus on the protocol at hand. First things I would check is protocol version that is active, and making sure that you're running the latest version your client can handle should be atleast running SMB3 if you're running W8/Server 2012 from memory.

 

[drakohen]

I have the minimum set to "SMB3" and the maximum to "SMB3_11". My daily driver and my laptop both run Windows 10 Pro so maybe I should boost my minimum up?

 

[Stux]

% ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>

Your network links are configured for simplex not full duplex (simplex means one direction at a time)

Confirm your network bandwidth using iperf. Once you have that sorted out, move on to SMB tuning.

 

[Kuhaku]

Your network links are configured for simplex not full duplex (simplex means one direction at a time)

Confirm your network bandwidth using iperf. Once you have that sorted out, move on to SMB tuning.

Thanks for the pick up mate, showing on my newbness on how the FreeBSD works there haha, guess media type is max trained speed of the interface then :)

 

[drakohen]

Your network links are configured for simplex not full duplex (simplex means one direction at a time)

Confirm your network bandwidth using iperf. Once you have that sorted out, move on to SMB tuning.

No issues with iperf:

Show : Console

% iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.100 port 5001 connected with 192.168.1.101 port 49291
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.06 GBytes 911 Mbits/sec

I am not sure why SIMPLEX is listed but from what I do know, Gigabit Ethernet can only fun in Full Duplex mode and I am definitely getting that. When I think of simplex, I think of fiber optic cable which I do have experience in. From what I've been digging up on the forums, it must be a CIFS limitation with it being single-threaded and all since I have no issues saturating my connection using FTP and iperf. I guess 3.8Ghz Turbo is only able to get me ~112MB/s total on CIFS unless someone has some other ideas.

 

[Stux]

You're right, I was confused between simplex/half-duplex.

I see (now that I'm not on a phone), that your Ethernet is actually trained at Full Duplex

"media: Ethernet autoselect (1000baseT <full-duplex>)"

Still, to test full duplex, you need to run two iperf streams, one in each direction. You could try the -d option

Once you've proved the full duplex networking using iperf, then its time to look at the SMB stack.

 

[drakohen]

Apologies, I selected the wrong console output. Here is the simultaneous transfers, again, no issues:

Show : Console

% iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.100 port 5001 connected with 192.168.1.24 port 35122
------------------------------------------------------------
Client connecting to 192.168.1.24, TCP port 5001
TCP window size: 233 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.1.100 port 34187 connected with 192.168.1.24 port 5001
Waiting for server threads to complete. Interrupt again to force quit.
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 1.07 GBytes 915 Mbits/sec
[ 4] 0.0-10.0 sec 1.09 GBytes 931 Mbits/sec

Thoughts?

 

[Kuhaku]

No issues with iperf:

Show : Console

% iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.100 port 5001 connected with 192.168.1.101 port 49291
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.06 GBytes 911 Mbits/sec

I am not sure why SIMPLEX is listed but from what I do know, Gigabit Ethernet can only fun in Full Duplex mode and I am definitely getting that. When I think of simplex, I think of fiber optic cable which I do have experience in. From what I've been digging up on the forums, it must be a CIFS limitation with it being single-threaded and all since I have no issues saturating my connection using FTP and iperf. I guess 3.8Ghz Turbo is only able to get me ~112MB/s total on CIFS unless someone has some other ideas.

Yeah I am trying to find some documentation on simplex flag equalizing half duplex mode on the interface, because mine is set to hard 1000/full but my ifconfig still shows simplex aswell. You wouldn't be using CIFS that is old legacy stuff you should be on SMB3+. On your client machine run this command in powershell with admin rights for me.
Get-SmbConnection

SMB 3.1.1 should be pumping a lot more throughput than you're experiencing, could we grab those reports at all on all the utilization for the server.

Ignore simplex, thanks for clearing that up.

 

[drakohen]

Readout from Powershell:

Show : Console

PS C:\Windows\system32> get-smbconnection

ServerName ShareName UserName Credential Dialect NumOpens
---------- --------- -------- ---------- ------- --------
xxxxxx backups XXXX\XXXXXXXXXX XXXX\XXXXXXXXXX 3.1.1 4

Client checks out. As for utilization, if SMB is represented by smbd service, it's only pulling 20% when i have a simultaneous transfer and receive going on between the desktop and server.

 

[drakohen]

Any advice on if this is now into the realm of SMB/Samba protocol or is the consensus still at the hardware level? From all the testing I've done, I am getting the throughput I should be via iperf for simultaneous transfer/receive and I experience the same speeds via the FTP protocol. If it really is a limitation of samba, I can live with that but if there is a chance I can get a boost to performance when doing simultaneous transfer/receive actions to the same client, I'll take it where I can. Thanks again for the assistance with this matter.

 

[drakohen]

Just giving this a bump to see if anyone has any further input relating to this matter.

 

[farmerpling2]

You could wireshark and see what is going on between the two machines. You could see what the SMB protocol is doing.

I use a small switch and ghost a port to watch all traffic to my wireshark laptop.

Every now and then, monitor the traffic. You would be amazed how much crap flows over the wires.