Intel x540t2 wont run past 300mbps

[kdragon75]

I'm using jumbo frames, so no 1500 here.
I do fully saturate 10gige, but not with iperf for some reason.

If you need help, please start a new thread. When multiple people respond to advice, it can get confusing for the people trying to help the OP.

 

[acquacow]

I'm not asking for any help, my setup is 100% I was offering suggestions to the op based on my struggles getting up to full 10gig speeds and relaying information from the #freenas IRC room.

Sent from my Moto Z (2) using Tapatalk

 

[oumpa31]

So your "MAX" speed statement should be qualified and not a blanket statement. Please try not to spread invalid or incomplete information. Also, I have a X5650 running at 1.6GHz that will do full 10gbe over iSCSI (1147MB/s). Interestingly enough, my 10gbe card is also a 82599ES (the same chip and driver for the x540).

Triple check that the MTU is 1500 on all points, quality cables of appropriate lengths are used, and that all "tunables" are disabled and removed. If your able, check for frame/crc errors on the switch.

From there, open two ssh sessions, one for iperf, and one for top. Monitor CPU usage on BOTH ends while running iperf. If you see the CPU pegged at 1.6 or 16% (1/6th of all (6)cores at 100%), it may be limited to one core. check to see if iperf3 allow to specify the number of threads and set it to two.

With the talk about jumbo frames and settings and after running iperf and not seeing really any activity on my cpu i looked into how to change the jumbo frames on freenas. I came across a post that said this card in a server machine such as mine needs to be set to 9000 if you are looking for full speed. Once set with no tunables i am finally getting full speed. Thank you everyone for your help.

 

[c32767a]

With the talk about jumbo frames and settings and after running iperf and not seeing really any activity on my cpu i looked into how to change the jumbo frames on freenas. I came across a post that said this card in a server machine such as mine needs to be set to 9000 if you are looking for full speed. Once set with no tunables i am finally getting full speed. Thank you everyone for your help.

Just out of curiosity, can you check the ifconfig on the interface now that you're getting satisfactory results? I'm curious if you ever took the port out of promiscuous mode or not.
I'm also curious if you set the same MTU on your client machine? Or is it still at 1500?

The X540 should be able to do wire speed on both ports with 1500 byte packets. It does not make sense that all things being equal only an MTU change would improve performance that much. One of the explanations as to why you need to run Jumbos would be that the TCP acceleration functions of the card are disabled due to the fact that it's in promiscuous mode.

The nas in the test I posted earlier had an mtu of 1500:

Code:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: connected to 2960-1 (Te1/0/2)
 options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 
	hwaddr 
	inet 192.168.1.51 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active

The other endpoint was a VM with 1 Xeon-D CPU. It doesn't take all that much CPU horsepower to generate 0s and pipe them down an ethernet card as long as the IO path is optimized.

 

[oumpa31]

It didn't make sense to me either. But for some reason it made a huge difference. Here are all the tests i ran.