iSCSI with Freenas and Cisco Nexus 5K

[marlonc]

Hello and Happy New Year to everyone.

I am in the process of building out a storage design using iSCSI from Intel / Dell X520-DA2 Ethernet Server Adapters on a Dell R730xd connected to a Cisco 5596UP switch. VCenter ESXi hosts are connected to the Nexus switch using the same Intel X520-DA2 adapters. On the Freenas, mtu 9000 is enabled on the NIC's for iSCSI and on the vmkernel adapters and virtual distributed switches in VCenter. I can only enable jumbo frames globally (mtu 9216) on the Nexus 5596UP, not under the specific interface as this is not a feature.

In VCenter, I can scan for the LUN's and the hosts sees the LUN's. The issue is when I try to add a Datastore I get a error message "The 'Create VMFS datastore' operation failed for the entity with the following error message. Cannot change the host configuration" and according to VMware, https://kb.vmware.com/s/article/2151431, I must change the interfaces mtu to 1500 or I can use 9000.

If I connect from Freenas directly to the hosts using mtu 9000 bypassing the Nexus Switch, I can add the datastores but this is not a scalable solution as I will have as many as 10 ESXi hosts.

Does anyone have a solution or a config for the Nexus?

Do I really need to set the mtu 9000 everywhere for iSCSI even when using the Intel X520-DA2 or can I get away with mtu 1500?

Thanks,

 

[HoneyBadger]

Have you enabled the MTU 9216 size globally on the Nexus? In order to use jumbo frames, they must be enabled on all devices on the network path; that includes your switch.

 

[marlonc]

Yes, it was enabled globally as 9216 on the Nexus. I was just reading another article on here cyberjock and jgreco titled Jumbo Frame Notes and they both say screw jumbo frames.

in my setup, do i really need it?

 

[marlonc]

by the way, the Freenas server has 6 x Intel / Dell X520-DA2 Ethernet Server Adapters broken down as such, but only NIC1 is being used as it is directly connected to my ESXi host.

Preference:

NIC 1 - iSCSI - Portal 172.16.13.1
NIC 2 - iSCSI - Portal 172.16.14.1
NIC 3/4 - SMB LACP File share
NIC 5 - Unused
NIC 6 - Unusd

 

[HoneyBadger]

The configuration looks right ... Did we visit this in another thread?

"Screw jumbo frames" usually comes as a result of the bugs and complexity getting it set up. Jumbo tends to be more beneficial in reducing CPU overhead, you're likely fine to go ahead with default MTU 1500.

 

[marlonc]

Yes we did review this under another thread but wanted to create a new posts.

Alright, I will use mtu 1500 on all my devices, switches, hosts and freenas.