Intel x540t2 wont run past 300mbps

[jgreco]

Promiscuous would be a side effect of the bridge.

 

[oumpa31]

I have plex and netdata running on on this box. I had turned netdata on when i was getting slower than expected transfer speeds. I have not configured any vm's yet but i was going to probably run a linux vm on this eventually.

How would i go about getting it out of promiscuous mode?

i did reboot after changing the tunables. i will try and adjust that tunable and see where i am.

i have a 14.04 and 16.04 live linux boot usb's. i can plug one of those in tonight and see if i'm getting full transfer speeds to my windows machine. i'll make sure that i share my ram disk drive.

should i put one run one live on my server and one on my workstation?

 

[oumpa31]

So in your ifconfig output, for ix0, the interface is showing in promiscuous mode. Are you running something that's doing packet capture? Or do you have VMs configured? At least for testing purposes you might want to disable any of that stuff and get the card out of promiscuous mode. Notably, tcp segmentation offload and some other performance enhancements appear to be disabled on the interface when it's in promiscuous mode. That could be the source of some of your issues.

Also, this may be a dumb question, but you did reboot after you set all the kernel tunables, but before you tested, correct?

A few comments on the tunables:

net.inet.tcp.recvspace (and sendspace) seem small. we typically use 4194304.

Most of the zfs tunables in your list are better left at their defaults. The zfs.arc parameters are largely ignored by the most recent versions of ZFS and most of that code is self tuning and better left at the defaults. In any event, based on your hardware config, none of those ZFS tunables will make a measurable difference on your read and write speeds.

There are also Linux and BSD live CDs around.. you could always try booting one on your windows workstation to rule out(or in) Windows as the problem.

So i tried the value you posted twice just to make sure i didn't type it wrong and both times it failed so i had to go restart it from the console its self. The message on the console that it had 0 buffer. Not really sure what it meant by that.

 

[oumpa31]

I had some help but i ran iperf with both ports connected to each other and this is what i ended up with.

As you'll see even the card is not spitting out full speed between the ports

 

[acquacow]

That should be fine for maxing out 10gig for samba. There's a thread somewhere about freebsd support for the intel 10gig nics maxing out somewhere around 6.5Gbit/sec.

 

[oumpa31]

Checked the freeBSD forum and here and everywhere else i can think of can't find anyone saying that intel maxing out at 6.5

 

[acquacow]

I spent 20min trying to find it after I mentioned it, someone on here linked me to it at one point.

Aah, actually it was folks in the #freenas irc channel on freenode that mentioned it, but they didn't link to any numbers.

 

[acquacow]

What's the CPU on your FreeNAS box? Is the BIOS set to max performance? Are c-states and p-states turned off?

 

[oumpa31]

Acquacow

Starting to think i'm stuck with what i have and i'll just have to load balance link aggregate them.

I'm running FreeNAS on a HP ProLiant DL360 G7 8-Port with 2 3.46 GHz Hex-Core Intel Xeon X5690. The full build is in my signature.

Its a server but i'll have to look into that. But i was getting full 10gig band with when i had windows on it when i was doing testing with an ASUS XG-C100C card

 

[acquacow]

Acquacow

Starting to think i'm stuck with what i have and i'll just have to load balance link aggregate them.

I'm running FreeNAS on a HP ProLiant DL360 G7 8-Port with 2 3.46 GHz Hex-Core Intel Xeon X5690. The full build is in my signature.

Its a server but i'll have to look into that. But i was getting full 10gig band with when i had windows on it when i was doing testing with an ASUS XG-C100C card

Well, windows will do smb multichannel by default as well. You need to enable that in freenas.

Since that is a multi-proc box, you may also have to do cpu pinning to ensure your nfs and smb processes are running on the same cpu that the NIC is plugged into.

Sent from my Moto Z (2) using Tapatalk

 

[oumpa31]

First things first lets make sure that smb multichannel is on. And how do i turn it on

 

[oumpa31]

i think i turned it on

but it doesn't have the option for server maximum protocol

 

[c32767a]

Promiscuous would be a side effect of the bridge.

Makes sense.. Not something I run across much.. our NASs are just NASs.. :)
Question becomes how much of a performance impact is there in disabling TSO and those other features on the card?

 

[c32767a]

So i tried the value you posted twice just to make sure i didn't type it wrong and both times it failed so i had to go restart it from the console its self. The message on the console that it had 0 buffer. Not really sure what it meant by that.

Not sure what wasn't working, here's how one of our boxes is set:

Code:

root@nasbench:~ # sysctl net.inet.tcp | grep space
net.inet.tcp.sendspace: 2097152
net.inet.tcp.recvspace: 4194304
root@nasbench:~ #

By the way, this is unlikely to be all that helpful for a lan connection, now that I think about it, becuase this simply changes how large tcp window scaling is allowed to scale. If you have very close connections, (eg same switch) it's not going to make much difference.

 

[c32767a]

Checked the freeBSD forum and here and everywhere else i can think of can't find anyone saying that intel maxing out at 6.5

Code:

root@nasbench:~ # dmesg | grep CPU:
CPU: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (3700.07-MHz K8-class CPU)

root@nas1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.51 port 21212 connected to 192.168.1.2 port 5201
[ ID] Interval		   Transfer	 Bitrate		 Retr  Cwnd
[  5]   0.00-1.00   sec  1.05 GBytes  9.00 Gbits/sec	0	567 KBytes	   
[  5]   1.00-2.00   sec  1.09 GBytes  9.35 Gbits/sec	0	586 KBytes	   
[  5]   2.00-3.00   sec  1.09 GBytes  9.37 Gbits/sec	0	603 KBytes	   
[  5]   3.00-4.00   sec  1.09 GBytes  9.35 Gbits/sec	0	620 KBytes	   
[  5]   4.00-5.00   sec  1.09 GBytes  9.37 Gbits/sec	0	637 KBytes	   
[  5]   5.00-6.00   sec  1.09 GBytes  9.36 Gbits/sec	0	653 KBytes	   
[  5]   6.00-7.00   sec  1.09 GBytes  9.35 Gbits/sec	0	670 KBytes	   
[  5]   7.00-8.00   sec  1.09 GBytes  9.36 Gbits/sec	0	685 KBytes	   
[  5]   8.00-9.00   sec  1.09 GBytes  9.35 Gbits/sec	0	700 KBytes	   
[  5]   9.00-10.00  sec  1.09 GBytes  9.37 Gbits/sec	0	715 KBytes	   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec	0			 sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec				  receiver

iperf Done.

It's very possible to get nearly wire speed from the 540 in FreeNAS. This was a run to a linux box with another 540 card.

 

[acquacow]

That's a very fast CPU though. I'm working with xeon-d and other 1.8/1.9GHz L-series xeons in my lab =)

 

[oumpa31]

I didn't have much time at all this weekend to play with it after Saturday morning. But to answer a few posts:

Yes it is on the same 10GB switch. port 1=workstation port 11= NAS

Tonight i'm hoping to have time to load start it up with my live linux usb and see what kind of speed i'm getting and do some iperf testing.

the only other thing i can think of is that the 540 in the FreeNAS box doesn't like the ASUS card in my work station.

 

[kdragon75]

That's a very fast CPU though. I'm working with xeon-d and other 1.8/1.9GHz L-series xeons in my lab =)

So your "MAX" speed statement should be qualified and not a blanket statement. Please try not to spread invalid or incomplete information. Also, I have a X5650 running at 1.6GHz that will do full 10gbe over iSCSI (1147MB/s). Interestingly enough, my 10gbe card is also a 82599ES (the same chip and driver for the x540).

Triple check that the MTU is 1500 on all points, quality cables of appropriate lengths are used, and that all "tunables" are disabled and removed. If your able, check for frame/crc errors on the switch.

From there, open two ssh sessions, one for iperf, and one for top. Monitor CPU usage on BOTH ends while running iperf. If you see the CPU pegged at 1.6 or 16% (1/6th of all (6)cores at 100%), it may be limited to one core. check to see if iperf3 allow to specify the number of threads and set it to two.

 

[acquacow]

I'm using jumbo frames, so no 1500 here.

I do fully saturate 10gige, but not with iperf for some reason.

Sent from my Moto Z (2) using Tapatalk

 

[oumpa31]

So your "MAX" speed statement should be qualified and not a blanket statement. Please try not to spread invalid or incomplete information. Also, I have a X5650 running at 1.6GHz that will do full 10gbe over iSCSI (1147MB/s). Interestingly enough, my 10gbe card is also a 82599ES (the same chip and driver for the x540).

Triple check that the MTU is 1500 on all points, quality cables of appropriate lengths are used, and that all "tunables" are disabled and removed. If your able, check for frame/crc errors on the switch.

From there, open two ssh sessions, one for iperf, and one for top. Monitor CPU usage on BOTH ends while running iperf. If you see the CPU pegged at 1.6 or 16% (1/6th of all (6)cores at 100%), it may be limited to one core. check to see if iperf3 allow to specify the number of threads and set it to two.

I'll look into that tonight. Thanks