Can't change network options

Status
Not open for further replies.

Nik

Explorer
Joined
May 28, 2011
Messages
50
Hi all,

After a reboot, I'm getting the following messages in the log.

Screen Shot 2018-07-17 at 21.52.13.png


I tried to add "txscum tso4" to the network adapters' options, but no change.

I assume this is a reason why my performance is roughly maxed at 3.75G with my 10G Chelsio S320-CR.

This is the ifconfig output.
cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
ether 00:07:43:06:79:03
hwaddr 00:07:43:06:79:03
inet 10.16.1.34 netmask 0xffffff00 broadcast 10.16.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet Unknown <full-duplex>
status: active

Am I doing something wrong?

Regards,
Nik
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
Exactly how did you try to change the options?

Like this...

Screen Shot 2018-07-19 at 17.29.04.png

Then, save and reboot.

The messages in the boot log will appear, like as if nothing would change.

Is that not correct?

Regards,
Nik
 
Joined
Dec 29, 2014
Messages
1,135
I think it is a mistake to assume that the problem must be the network. You are only as fast as the slowest component. If you speed up the network, something else becomes the slowest part. Have you tested something else on the network that doesn't involve reading data from storage?
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
I think it is a mistake to assume that the problem must be the network. You are only as fast as the slowest component. If you speed up the network, something else becomes the slowest part. Have you tested something else on the network that doesn't involve reading data from storage?

But when I test with iperf, what can be the slowest component?

I have my Freenas box, and two clients a Mac with 10G thunderbolt box and a PC with 10G Intel card. Between PC and Mac I'm getting 9,2G and from the Mac to the Freenas 3,5G and from the PC 3,75G.

All measured with iperf.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Your problem isn't TSO, it's the fact that 10Gbe isn't plug and play on freeNAS. You need to do some network tuning to achieve the through put you want.

Start with the tunables in this post and experiment with your system. The window sizes I used may not be appropriate for your box but it will give you a start.
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
Thanks, I think I have flipped the forum already upside down. I actually tested with a Chelsio T540-CR too with the same results. I tried your setting, tried to tweak, tried the ones from the guy with the Intel X550 in the 10G primer post. Always the same results or worse.

The only thing I never read somewhere, is the above message in the log.
Can there be somewhere a place where the -txcsum is set? A config file or something?
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
Where is the file that holds the sysctl? I've set some sysctl but I can't see them in /etc/sysctl.conf. Am I looking a the right place?
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
I found this in the /etc/loader.conf

# Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0

I would assume it affects the cxgbe driver, the one that the S320 uses too. Could that cause my problem?
 
Joined
Dec 29, 2014
Messages
1,135
But when I test with iperf, what can be the slowest component?

You are definitely using the right tools to test that. I wish I could offer something more constructive on the S320. I used that card in one of my systems a couple of builds ago. I am using a T520-CR now, and that works great. I do still have some S320's in a box, so I'll be interested to see what you find out about tunables.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I found this in the /etc/loader.conf

# Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0

I would assume it affects the cxgbe driver, the one that the S320 uses too. Could that cause my problem?
Your card uses the older cxgb driver, the newer T4 and T5 ASIC cards use the cxgbe driver, so you'd be un-affected by the above cxgbe loader settings.

Where is the file that holds the sysctl? I've set some sysctl but I can't see them in /etc/sysctl.conf. Am I looking a the right place?
You should be setting the sysctl values in the tunables section of the system tab.
 
Joined
Dec 29, 2014
Messages
1,135
Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
You should be setting the sysctl values in the tunables section of the system tab.

Sure, I'm doing that, but I would like to verify if they're being set correctly.
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?

I have the same effect/performance with a T540-CR.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Sure, I'm doing that, but I would like to verify if they're being set correctly.
The easiest way to check you sysctl settings would be to run from the cli:
Code:
sysctl -a | grep 

Then just add the sysctl you're looking for after 'grep'
 
Joined
Dec 29, 2014
Messages
1,135
Might be where it is connected to the motherboard. Which slot is it in? Here are the specs from ASUS.

Expansion Slots
Total Slot : 4
Slot Location 4 : PCI 32bit/33MHz
Slot Location 5 : PCI-E x8 (X4 Link)
Slot Location 6 : PCI-E x16
Slot Location 7 : PCI-E x8 (X4 Link)

I would say slot 6 looks like the highest performance slot.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?
That card is PCIe 1.0 x8, so you couldn't saturate two connections at once, but it is capable of saturating one connection. The max theoretical throughput for that PCIe connection is 2000MB/sec, and 10Gbe max theoretical throughput is 1280MB/s. In my experience with freeNAS 10Gbe when the throughput is off it's always required network tuning to fix it.
 
Last edited:

Nik

Explorer
Joined
May 28, 2011
Messages
50
Might be where it is connected to the motherboard. Which slot is it in? Here are the specs from ASUS.

Expansion Slots
Total Slot : 4
Slot Location 4 : PCI 32bit/33MHz
Slot Location 5 : PCI-E x8 (X4 Link)
Slot Location 6 : PCI-E x16
Slot Location 7 : PCI-E x8 (X4 Link)

I would say slot 6 looks like the highest performance slot.

It's in slot 5 so a PCIE with x4 should be capable of delivering 1000MB/sec.

I wonder what the x4 "Link" means.
 

Nik

Explorer
Joined
May 28, 2011
Messages
50
Your problem isn't TSO, it's the fact that 10Gbe isn't plug and play on freeNAS. You need to do some network tuning to achieve the through put you want.

Start with the tunables in this post and experiment with your system. The window sizes I used may not be appropriate for your box but it will give you a start.

What is the best approach when I start with the tunables again?

I guess:

1. deactivate autotune if it's on and remove all tunables from the system tab. --> reboot

With which tunable do I start? Is there a "strategy"?
Do I need to reboot after each tunable or is there a way to do it without reboot?
 
Status
Not open for further replies.
Top