SOLVED Jumbo Frame MTU set but not working

[Ken Kirchner]

Hi all,

I have read the "Jumbo frame notes" thread and decided I would continue to use JF since I have a dedicate storage network with JF already enabled.

I just built a FreeNAS 9.10.2-U4 box on a Dell R410 platform (H310 flashed to IT firmware). It has dual 1Gb NICs (QLogic NetXtreme II BCM5716 1000Base-T) and for the most part, seems to be working well. I am setting up the iSCSI interface on bce1 and have added the "mtu 9000" option.

bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether d4:ae:52:af:9f:b6
inet 10.10.10.5 netmask 0xfffffff8 broadcast 10.10.10.7
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

The other equipment on the iSCSI SAN are a QNAP NAS and 3 ESXi hosts (Dell R610). This is all connected through a TP-Link switch which by default supports a 10,000 MTU.

To test whether or not MTU is actually set, I am pinging from the ESXi host and the QNAP with a packet size of 8900 and the "do not fragment" bit set. It works fine between the QNAP and ESXi hosts. If I use a packet size of 9000 it fails as expected.

[root@red:~] ping -4 -c5 -d -s8900 10.10.10.1

PING 10.10.10.1 (10.10.10.1): 8900 data bytes
8908 bytes from 10.10.10.1 icmp_seq=0 ttl=64 time=0.707 ms
8908 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.645 ms
8908 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.713 ms
8908 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=0.731 ms
8908 bytes from 10.10.10.1: icmp_seq=4 ttl=64 time=0.643 ms
--- 10.10.10.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss

[root@red:~] ping -4 -c5 -d -s9000 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 9000 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
--- 10.10.10.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss


The problem is that this does not work for the FreeNAS. No matter what, I can only use an MTU of 1472 max despite the MTU setting I have put on the interface. The route command also shows the 1500 limit

[root@freenas] ~# route get 10.10.10.1
route to: 10.10.10.1
destination: 10.10.10.0
mask: 255.255.255.248
fib: 0
interface: bce1
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0

[root@freenas] ~# ping -c3 -s1472 -D 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 1472 data bytes
1480 bytes from 10.10.10.1: icmp_seq=0 ttl=64 time=0.396 ms
1480 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.241 ms
1480 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.216 ms
--- 10.10.10.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.216/0.284/0.396/0.080 ms

[root@freenas] ~# ping -c3 -s1473 -D 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 1473 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
--- 10.10.10.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Connectivity is fine otherwise and I can storage vmotion my VM's to this FreeNAS iSCSI datastore without issue. I am confused as to what the problem might be. I also swapped the bce1 connection on the switch with the QNAP to verify it wasn't a switch port issue, and it was not. The QNAP continued to work at the higher MTU and the FreeNAS did not.

Is there something wrong with my test methodology or is there some other spot in the FreeNAS configuration I need to enable JF support?

 

[Ken Kirchner]

Apparently I'm just an idiot and forgot to reboot the server... Now it's all working fine.

[root@freenas] ~# route get 10.10.10.1
route to: 10.10.10.1
destination: 10.10.10.0
mask: 255.255.255.248
fib: 0
interface: bce1
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 9000 1 0

[root@freenas] ~# ping -c3 -s8900 -D 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 8900 data bytes
8908 bytes from 10.10.10.1: icmp_seq=0 ttl=64 time=0.750 ms
8908 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.657 ms
8908 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.593 ms

--- 10.10.10.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.593/0.667/0.750/0.064 ms