Supermicro X10Dri-T with Dual onboard Intel X540 NIC's

Status
Not open for further replies.

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
Guys,

I just purchased a supermicro X10Dri-T dual cpu board, which has dual 10g nics. I'm having an issue where the nics can't get above 4gbps on iperf. I've tried this on ubuntu and freenas 11, and freenas 11.2, same result. I've even tried creating a LACP Lagg between the two 10gig nics, however they were still capped at 4 gigs.

I've contacted supermicro and they told me to update the driver, however there doesn't seem to be a way to do that on Freenas.

Does anyone know if there's a workaround or a way to update? Worst case a way to request an update to the driver in the next version?

Any suggestions would be appreciated!

Edit: Some more details
Freenas Version: 11.1-U6 (Also tried 11.2-RC1)
Motherboard: Supermicro X10DRI-T
CPU: Dual Xeon E5-2650 V4 2.2ghz
Freenas NIC Intel X540
Switch: Unifi switch 16XG.
Client: Intel X550 nic.

Also it's worth mentioning that I've run iperf through the same switch, to an Ubuntu server with an Intel X520-DA2 card and got speeds of 8-9gbps.

Code:
root@freenas:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55125
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  4.07 GBytes  3.50 Gbits/sec
[  5] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55143
[  5]  0.0-10.3 sec  4.06 GBytes  3.38 Gbits/sec
[  4] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55148
[  4]  0.0-10.0 sec  3.71 GBytes  3.19 Gbits/sec
[  5] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55177
[  5]  0.0-10.0 sec  3.50 GBytes  3.00 Gbits/sec
[  4] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55193
[  4]  0.0-10.0 sec  3.85 GBytes  3.30 Gbits/sec
[  5] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55231
[  5]  0.0-10.3 sec  3.42 GBytes  2.85 Gbits/sec
[  4] local 192.168.1.35 port 5001 connected with 192.168.1.11 port 55261
[  4]  0.0-10.0 sec  4.20 GBytes  3.60 Gbits/sec


Code:
root@freenas:~ # netstat -sI ix0
ip6 on ix0:
		0 total input datagrams
		0 datagrams with invalid header received
		0 datagrams exceeded MTU received
		0 datagrams with no route received
		0 datagrams with invalid dst received
		0 datagrams with unknown proto received
		0 truncated datagrams received
		79 input datagrams discarded
		0 datagrams delivered to an upper layer protocol
		0 datagrams forwarded to this interface
		0 datagrams sent from an upper layer protocol
		0 total discarded output datagrams
		0 output datagrams fragmented
		0 output datagrams failed on fragment
		0 output datagrams succeeded on fragment
		0 incoming datagrams fragmented
		0 datagrams reassembled
		0 datagrams failed on reassembly
		0 multicast datagrams received
		0 multicast datagrams sent
icmp6 on ix0:
		0 total input messages
		0 total input error messages
		0 input destination unreachable errors
		0 input administratively prohibited errors
		0 input time exceeded errors
		0 input parameter problem errors
		0 input packet too big errors
		0 input echo requests
		0 input echo replies
		0 input router solicitations
		0 input router advertisements
		0 input neighbor solicitations
		0 input neighbor advertisements
		0 input MLD queries
		0 input MLD reports
		0 input MLD dones
		0 total output messages
		0 total output error messages
		0 output destination unreachable errors
		0 output administratively prohibited errors
		0 output time exceeded errors
		0 output parameter problem errors
		0 output packet too big errors
		0 output echo requests
		0 output echo replies
		0 output router solicitations
		0 output router advertisements
		0 output neighbor solicitations
		0 output neighbor advertisements
		0 output redirects
		0 output MLD queries
		0 output MLD reports
		0 output MLD dones



Code:
root@freenas:~ # netstat -s
tcp:
		102381 packets sent
				45566 data packets (4238560 bytes)
				0 data packets (0 bytes) retransmitted
				0 data packets unnecessarily retransmitted
				0 resends initiated by MTU discovery
				49845 ack-only packets (4 delayed)
				0 URG only packets
				0 window probe packets
				1626 window update packets
				5372 control packets
		104486 packets received
				49320 acks (for 4224564 bytes)
				2119 duplicate acks
				0 acks for unsent data
				46064 packets (3539719 bytes) received in-sequence
				283 completely duplicate packets (39 bytes)
				0 old duplicate packets
				0 packets with some dup. data (0 bytes duped)
				0 out-of-order packets (0 bytes)
				0 packets (0 bytes) of data after window
				0 window probes
				1682 window update packets
				1235 packets received after close
				0 discarded for bad checksums
				0 discarded for bad header offset fields
				0 discarded because packet too short
				0 discarded due to memory problems
		1948 connection requests
		1706 connection accepts
		0 bad connection attempts
		0 listen queue overflows
		1227 ignored RSTs in the windows
		3452 connections established (including accepts)
				3341 times used RTT from hostcache
				3341 times used RTT variance from hostcache
				0 times used slow-start threshold from hostcache
		3679 connections closed (including 1271 drops)
				2956 connections updated cached RTT on close
				2956 connections updated cached RTT variance on close
				0 connections updated cached ssthresh on close
		180 embryonic connections dropped
		49254 segments updated rtt (of 46233 attempts)
		90 retransmit timeouts
				0 connections dropped by rexmit timeout
		0 persist timeouts
				0 connections dropped by persist timeout
		0 Connections (fin_wait_2) dropped because of timeout
		240 keepalive timeouts
				240 keepalive probes sent
				0 connections dropped by keepalive
		30760 correct ACK header predictions
		40746 correct data packet header predictions
		1706 syncache entries added
				0 retransmitted
				0 dupsyn
				0 dropped
				1706 completed
				0 bucket overflow
				0 cache overflow
				0 reset
				0 stale
				0 aborted
				0 badack
				0 unreach
				0 zone failures
		1706 cookies sent
		0 cookies received
		11 hostcache entries added
				0 bucket overflow
		0 SACK recovery episodes
		0 segment rexmits in SACK recovery episodes
		0 byte rexmits in SACK recovery episodes
		0 SACK options (SACK blocks) received
		0 SACK options (SACK blocks) sent
		0 SACK scoreboard overflow
		0 packets with ECN CE bit set
		0 packets with ECN ECT(0) bit set
		0 packets with ECN ECT(1) bit set
		0 successful ECN handshakes
		0 times ECN reduced the congestion window
		0 packets with matching signature received
		0 packets with bad signature received
		0 times failed to make signature due to no SA
		0 times unexpected signature received
		0 times no signature provided by segment
TCP connection count by state:
		0 connections in CLOSED state
		32 connections in LISTEN state
		2 connections in SYN_SENT state
		0 connections in SYN_RCVD state
		18 connections in ESTABLISHED state
		1 connection  in CLOSE_WAIT state
		0 connections in FIN_WAIT_1 state
		0 connections in CLOSING state
		0 connections in LAST_ACK state
		0 connections in FIN_WAIT_2 state
		14 connections in TIME_WAIT state
udp:
		5657 datagrams received
		0 with incomplete header
		0 with bad data length field
		0 with bad checksum
		0 with no checksum
		77 dropped due to no socket
		4606 broadcast/multicast datagrams undelivered
		0 dropped due to full socket buffers
		0 not for hashed pcb
		974 delivered
		1629 datagrams output
		0 times multicast source filter matched
ip:
		111310 total packets received
		0 bad header checksums
		0 with size smaller than minimum
		0 with data size < data length
		0 with ip length > max ip packet size
		0 with header length < data size
		0 with data length < header length
		0 with bad options
		0 with incorrect version number
		0 fragments received
		0 fragments dropped (dup or out of space)
		0 fragments dropped after timeout
		0 packets reassembled ok
		110135 packets for this host
		0 packets for unknown/unsupported protocol
		0 packets forwarded (0 packets fast forwarded)
		1175 packets not forwardable
		237 packets received for unknown multicast group
		0 redirects sent
		106998 packets sent from this host
		0 packets sent with fabricated ip header
		0 output packets dropped due to no bufs, etc.
		0 output packets discarded due to no route
		0 output datagrams fragmented
		0 fragments created
		0 datagrams that can't be fragmented
		0 tunneling packets that can't find gif
		0 datagrams with bad address in header
icmp:
		57 calls to icmp_error
		0 errors not generated in response to an icmp message
		Output histogram:
				destination unreachable: 57
		0 messages with bad code fields
		0 messages less than the minimum length
		0 messages with bad checksum
		0 messages with bad length
		0 multicast echo requests ignored
		0 multicast timestamp requests ignored
		Input histogram:
				destination unreachable: 2
		0 message responses generated
		0 invalid return addresses
		0 no return routes
		ICMP address mask responses are disabled
igmp:
		90 messages received
		0 messages received with too few bytes
		0 messages received with wrong TTL
		0 messages received with bad checksum
		0 V1/V2 membership queries received
		0 V3 membership queries received
		0 membership queries received with invalid field(s)
		0 general queries received
		0 group queries received
		0 group-source queries received
		0 group-source queries dropped
		0 membership reports received
		0 membership reports received with invalid field(s)
		0 membership reports received for groups to which we belong
		0 V3 reports received without Router Alert
		22 membership reports sent
carp:
		0 packets received (IPv4)
		0 packets received (IPv6)
				0 packets discarded for wrong TTL
				0 packets shorter than header
				0 discarded for bad checksums
				0 discarded packets with a bad version
				0 discarded because packet too short
				0 discarded for bad authentication
				0 discarded for bad vhid
				0 discarded because of a bad address list
		0 packets sent (IPv4)
		0 packets sent (IPv6)
				0 send failed due to mbuf memory error
arp:
		6 ARP requests sent
		71 ARP replies sent
		2155 ARP requests received
		3 ARP replies received
		4193 ARP packets received
		0 total packets dropped due to no ARP entry
		0 ARP entrys timed out
		0 Duplicate IPs seen
ip6:
		142 total packets received
		0 with size smaller than minimum
		0 with data size < data length
		0 with bad options
		0 with incorrect version number
		0 fragments received
		0 fragments dropped (dup or out of space)
		0 fragments dropped after timeout
		0 fragments that exceeded limit
		0 packets reassembled ok
		140 packets for this host
		0 packets forwarded
		0 packets not forwardable
		0 redirects sent
		90 packets sent from this host
		0 packets sent with fabricated ip header
		0 output packets dropped due to no bufs, etc.
		4 output packets discarded due to no route
		0 output datagrams fragmented
		0 fragments created
		0 datagrams that can't be fragmented
		0 packets that violated scope rules
		34 multicast packets which we don't join
		Input histogram:
				hop by hop: 27
				TCP: 86
				UDP: 20
				ICMP6: 9
		Mbuf statistics:
				140 one mbuf
				two or more mbuf:
						lo0= 2
				0 one ext mbuf
				0 two or more ext mbuf
		0 packets whose headers are not contiguous
		0 tunneling packets that can't find gif
		0 packets discarded because of too many headers
		1326 failures of source address selection
		source addresses on a non-outgoing I/F
				1326 addresses scope=0xf
		Source addresses selection rule applied:
				1326 same address
icmp6:
		2 calls to icmp6_error
		0 errors not generated in response to an icmp6 message
		0 errors not generated because of rate limitation
		Output histogram:
				unreach: 2
		0 messages with bad code fields
		0 messages < minimum length
		0 bad checksums
		0 messages with bad length
		Input histogram:
				unreach: 2
		Histogram of error messages to be generated:
				0 no route
				0 administratively prohibited
				0 beyond scope
				0 address unreachable
				2 port unreachable
				0 packet too big
				0 time exceed transit
				0 time exceed reassembly
				0 erroneous header field
				0 unrecognized next header
				0 unrecognized option
				0 redirect
				0 unknown
		0 message responses generated
		0 messages with too many ND options
		0 messages with bad ND options
		0 bad neighbor solicitation messages
		0 bad neighbor advertisement messages
		0 bad router solicitation messages
		0 bad router advertisement messages
		0 bad redirect messages
		0 path MTU changes
rip6:
		0 messages received
		0 checksum calculations on inbound
		0 messages with bad checksum
		0 messages dropped due to no socket
		0 multicast messages dropped due to no socket
		0 messages dropped due to full socket buffers
		0 delivered
		0 datagrams output
root@freenas:~ #
 
Last edited:

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
I'm having no issues on an X10 xeon-d 1541 board. All of my issues were client side involving TCP window size (Windows 10), but I fixed that.

I haven't even set the recommended 10gigE tunables yet: http://45drives.blogspot.com/2016/05/how-to-tune-nas-for-direct-from-server.html

You may want to try that though.

Also, do you have Jumbo frames setup on FreeNAS, the switch ports, and on the client?

-- Dave
No i haven't touched jumbo frames yet. My other system didn't need them, but i can give that a try.
 
Joined
Dec 29, 2014
Messages
1,135
No i haven't touched jumbo frames yet. My other system didn't need them, but i can give that a try.

Do so at your own risk. The switch and the client, and everything else along the entire path need to have jumbo frame support enabled. Otherwise the jumbo frames get dropped. IMHO, the small gains achieved by enabling jumbo frames are pretty small compared to the disaster potential. There are some specific use cases where it helps, but not many (again IMHO).
 

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
Do so at your own risk. The switch and the client, and everything else along the entire path need to have jumbo frame support enabled. Otherwise the jumbo frames get dropped. IMHO, the small gains achieved by enabling jumbo frames are pretty small compared to the disaster potential. There are some specific use cases where it helps, but not many (again IMHO).

Yeah just tried it and things didn't play nice, and it didn't help.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
I just tested iperf To/From Freenas and my Ubuntu server, results were the same.

Freenas side
Code:
root@freenas:~ # iperf -c 192.168.1.45
------------------------------------------------------------
Client connecting to 192.168.1.45, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.35 port 18430 connected with 192.168.1.45 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  4.35 GBytes  3.73 Gbits/sec

root@freenas:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.35 port 5001 connected with 192.168.1.45 port 58530
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  3.75 GBytes  3.22 Gbits/sec



Ubuntu side
Code:
mjk79@Paradox-Server:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.45 port 5001 connected with 192.168.1.35 port 18430
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  4.35 GBytes  3.73 Gbits/sec

mjk79@Paradox-Server:~$ iperf -c 192.168.1.35
------------------------------------------------------------
Client connecting to 192.168.1.35, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.45 port 58530 connected with 192.168.1.35 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  3.75 GBytes  3.22 Gbits/sec
 

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
At the request of supermicro i've installed an OS that they've validated (here) in this case windows 7, updated with the supermicro latest drivers, ran iperf again and got the exact same results on both nics.

I'm wondering if this is a bad lan controller.
 

acquacow

Explorer
Joined
Sep 7, 2018
Messages
51
Direct connect the boxes for a test

Sent from my Moto Z (2) using Tapatalk
 

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67

acquacow

Explorer
Joined
Sep 7, 2018
Messages
51
Interesting. I have two x540-T2 cards, the 550 that is onboard the supermicro, and one of those cheap aquantia cards. No issues maxing things out here.

Since you have a dual cpu box, is the pci-e card attached to the cpu that is running iperf? You could be running into some NUMA overhead. Also make sure c-states and p-states are disabled in the bios for max perf.

Sent from my Moto Z (2) using Tapatalk
 

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
Interesting. I have two x540-T2 cards, the 550 that is onboard the supermicro, and one of those cheap aquantia cards. No issues maxing things out here.

Since you have a dual cpu box, is the pci-e card attached to the cpu that is running iperf? You could be running into some NUMA overhead. Also make sure c-states and p-states are disabled in the bios for max perf.

Sent from my Moto Z (2) using Tapatalk

The dual cpu box has two X540's on the motherboard. Currently the only PCI-e Cards attached are 3x, M1015 raid cards in IT mode, those are in the bottom 3 slots, which are cpu1. I'm not sure how to check which one is running iperf.

The X550 is in my PC, which is acting as a client.
 

mjk79

Explorer
Joined
Nov 4, 2014
Messages
67
There doesn't appear to be a c-state or p-state option in the bios. Nothing mentioning it in the manual either.
 
Status
Not open for further replies.
Top