problems setting MTU on 10Gbit interface

[poarduh]

Hi,

I'm trying to set mtu 9000 on a 10Gbit link on freenas (FreeNAS-8.0.4-RELEASE-x64 (10351)), but if I do I lose network connectivity. It works one way on the Gbit interfaces. I'm really stuck here and am hoping someone has some clues about what's going on...
Here's the results of my attempts:

Interface igb1 on freenas works one way:

igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:30:48:cb:bb:fb
inet 10.88.88.252 netmask 0xffffff00 broadcast 10.88.88.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

ping from client to freenas:

# ping 10.88.88.252 -s 8972
PING 10.88.88.252 (10.88.88.252) 8972(9000) bytes of data.
8980 bytes from 10.88.88.252: icmp_seq=1 ttl=64 time=1.10 ms
8980 bytes from 10.88.88.252: icmp_seq=2 ttl=64 time=0.442 ms

ping from freenas to client:

#ping -s 8972 10.100.100.10
ping: packet size too large: 8972 > 56: Operation not permitted

Interface ix0 works fine by default:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:25:90:3b:12:be
inet 10.10.10.252 netmask 0xffffff00 broadcast 10.10.10.255
media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
status: active

ping from freenas to client:

# ping 10.88.88.10
PING 10.88.88.10 (10.88.88.10): 56 data bytes
64 bytes from 10.88.88.10: icmp_seq=0 ttl=64 time=0.096 ms
64 bytes from 10.88.88.10: icmp_seq=1 ttl=64 time=0.089 ms

Now if I set the option mtu 9000 on ix0 on freenas:

ix0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:25:90:3b:12:be
inet 10.10.10.252 netmask 0xffffff00 broadcast 10.10.10.255
media: Ethernet autoselect
status: no carrier

ping from freenas to client:

# ping 10.88.88.10
PING 10.88.88.10 (10.88.88.10): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down

Network is gone... now why am I not allowed to send 9000 bytes from freenas to the client on the Gbit interface, and why does the whole link go down on the 10Gbit interface when setting the mtu 9000 option ?

Cheers,
Poard

 

[louisk]

Did you configure the MTU on your switches?

 

[poarduh]

Yes, switch is configured with MTU 9000 as well... setup is 2 clients, a switch and a NAS running freenas. The 2 clients can ping each other with 9000 bytes packets, so it does work. So the trouble seems to be with the freenas interfaces.

 

[poarduh]

Ok, it seems the issue has to do with settings in sysctl.conf. These can be added in freenas using tunables:

kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
kern.ipc.nmbjumbo16=32000
kern.ipc.nmbjumbo9=64000

And the 10Gbit interface also works now!

 

[peterh]

Whats the speed difference when using jumbo frames ?

 

[poarduh]

Haven't had the chance to test properly yet; using jumbo frames was a requirement for an application which I'm currently deploying. Will probably start testing next week or so. I will then post the results.