Can't change network options

[Nik]

Hi all,

After a reboot, I'm getting the following messages in the log.

I tried to add "txscum tso4" to the network adapters' options, but no change.

I assume this is a reason why my performance is roughly maxed at 3.75G with my 10G Chelsio S320-CR.

This is the ifconfig output.
cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
ether 00:07:43:06:79:03
hwaddr 00:07:43:06:79:03
inet 10.16.1.34 netmask 0xffffff00 broadcast 10.16.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet Unknown <full-duplex>
status: active

Am I doing something wrong?

Regards,
Nik

 

[dlavigne]

Exactly how did you try to change the options?

 

[Nik]

Exactly how did you try to change the options?

Like this...



Then, save and reboot.

The messages in the boot log will appear, like as if nothing would change.

Is that not correct?

Regards,
Nik

 

[Elliot Dierksen]

I think it is a mistake to assume that the problem must be the network. You are only as fast as the slowest component. If you speed up the network, something else becomes the slowest part. Have you tested something else on the network that doesn't involve reading data from storage?

 

[Nik]

I think it is a mistake to assume that the problem must be the network. You are only as fast as the slowest component. If you speed up the network, something else becomes the slowest part. Have you tested something else on the network that doesn't involve reading data from storage?

But when I test with iperf, what can be the slowest component?

I have my Freenas box, and two clients a Mac with 10G thunderbolt box and a PC with 10G Intel card. Between PC and Mac I'm getting 9,2G and from the Mac to the Freenas 3,5G and from the PC 3,75G.

All measured with iperf.

 

[Mlovelace]

Your problem isn't TSO, it's the fact that 10Gbe isn't plug and play on freeNAS. You need to do some network tuning to achieve the through put you want.

Start with the tunables in this post and experiment with your system. The window sizes I used may not be appropriate for your box but it will give you a start.

 

[Nik]

Thanks, I think I have flipped the forum already upside down. I actually tested with a Chelsio T540-CR too with the same results. I tried your setting, tried to tweak, tried the ones from the guy with the Intel X550 in the 10G primer post. Always the same results or worse.

The only thing I never read somewhere, is the above message in the log.
Can there be somewhere a place where the -txcsum is set? A config file or something?

 

[Nik]

Where is the file that holds the sysctl? I've set some sysctl but I can't see them in /etc/sysctl.conf. Am I looking a the right place?

 

[Nik]

I found this in the /etc/loader.conf

# Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0

I would assume it affects the cxgbe driver, the one that the S320 uses too. Could that cause my problem?

 

[Elliot Dierksen]

But when I test with iperf, what can be the slowest component?

You are definitely using the right tools to test that. I wish I could offer something more constructive on the S320. I used that card in one of my systems a couple of builds ago. I am using a T520-CR now, and that works great. I do still have some S320's in a box, so I'll be interested to see what you find out about tunables.

 

[Mlovelace]

I found this in the /etc/loader.conf

# Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0

I would assume it affects the cxgbe driver, the one that the S320 uses too. Could that cause my problem?

Your card uses the older cxgb driver, the newer T4 and T5 ASIC cards use the cxgbe driver, so you'd be un-affected by the above cxgbe loader settings.

Where is the file that holds the sysctl? I've set some sysctl but I can't see them in /etc/sysctl.conf. Am I looking a the right place?

You should be setting the sysctl values in the tunables section of the system tab.

 

[Elliot Dierksen]

Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?

 

[Nik]

You should be setting the sysctl values in the tunables section of the system tab.

Sure, I'm doing that, but I would like to verify if they're being set correctly.

 

[Nik]

Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?

I have the same effect/performance with a T540-CR.

 

[Mlovelace]

Sure, I'm doing that, but I would like to verify if they're being set correctly.

The easiest way to check you sysctl settings would be to run from the cli:

Code:

sysctl -a | grep 

Then just add the sysctl you're looking for after 'grep'

 

[Elliot Dierksen]

Might be where it is connected to the motherboard. Which slot is it in? Here are the specs from ASUS.

Expansion Slots
Total Slot : 4
Slot Location 4 : PCI 32bit/33MHz
Slot Location 5 : PCI-E x8 (X4 Link)
Slot Location 6 : PCI-E x16
Slot Location 7 : PCI-E x8 (X4 Link)

I would say slot 6 looks like the highest performance slot.

 

[Mlovelace]

Hmm. I wonder if it could be the card itself. That is a much older card. Here is what Chelsio says about the specs on that card:

S320E-LP-CR
Connector Type : Twinax cable ready
Host Interface : PCI Express x8
Description : 2 X 10G Ethernet, Low Profile

It doesn't even say PCIe 2.0. It may just be a limitation of the card. I know that wouldn't be good news. I am assuming that the Intel 10G NIC's in the other boxes are of newer vintage. Is that correct?

That card is PCIe 1.0 x8, so you couldn't saturate two connections at once, but it is capable of saturating one connection. The max theoretical throughput for that PCIe connection is 2000MB/sec, and 10Gbe max theoretical throughput is 1280MB/s. In my experience with freeNAS 10Gbe when the throughput is off it's always required network tuning to fix it.

 

[Nik]

Might be where it is connected to the motherboard. Which slot is it in? Here are the specs from ASUS.

Expansion Slots
Total Slot : 4
Slot Location 4 : PCI 32bit/33MHz
Slot Location 5 : PCI-E x8 (X4 Link)
Slot Location 6 : PCI-E x16
Slot Location 7 : PCI-E x8 (X4 Link)

I would say slot 6 looks like the highest performance slot.

It's in slot 5 so a PCIE with x4 should be capable of delivering 1000MB/sec.

I wonder what the x4 "Link" means.

 

[Mlovelace]

It's in slot 5 so a PCIE with x4 should be capable of delivering 1000MB/sec.

I wonder what the x4 "Link" means.

It means that physically it's a x8 slot but electricly it's x4.

 

[Nik]

Your problem isn't TSO, it's the fact that 10Gbe isn't plug and play on freeNAS. You need to do some network tuning to achieve the through put you want.

Start with the tunables in this post and experiment with your system. The window sizes I used may not be appropriate for your box but it will give you a start.

What is the best approach when I start with the tunables again?

I guess:

1. deactivate autotune if it's on and remove all tunables from the system tab. --> reboot

With which tunable do I start? Is there a "strategy"?
Do I need to reboot after each tunable or is there a way to do it without reboot?