ESXI 6.7 iSCSI with 4 NICs + FreeNAS with 4 NICs in LACP for iSCSI

Grist · Jul 15, 2020

Hi all,

I set up a FreeNAS-11.3-U3.2 in order to use it as a SAN (using i SCSI) for an ESXi 6.7 free version. I am aware that some features of the ESXi free version are not available (like LACP), therefore this thread :)

Code:

    ESXi 6.7 Free                   SWITCH                                 FreeNAS
   +--------------+             +------------+                           +----------+
   |              |   VLAN10    |            |        VLAN10             |          |
   |       vmnic0 |-------------|            |---------------------------| msk0     |
   |              |             |            |                           |          |
   |              |             |            |                           |          |
   |       vmnic6 |-------------|          L |-------------------\      /| igb0     |
   |       vmnic7 |-------------|          A |------------------- lagg0--| igb1     |
   |       vmnic8 |-------------|          C |-------------------/      \| igb2     |
   |       vmnic9 |-------------|          P |------------------/       \| igb3     |
   |              |   VLAN11    |            |         VLAN11            |          |
   +--------------+             +------------+                           +----------+

the FreeNAS has 5 NICS
- msk0 is used for management and is configured in VLAN10
- igb0 up to igb3 are building the (LACP) lagg0 interface in VLAN11. lagg0 has an IP accordingly in VLAN11 IP segment.
the switch is a L2 managed one. The ports on the switch - where the igb0 to igb3 from FreeNAS are connected - are configured appropriately as (active) LACP
the ESXi server has several NICs, too. vmnic0 is used for management.....vmnic6 to vmnic9 are used for iSCSI (also in VLAN11)

I have configured the iSCSI Software Adapter on the ESXi as described in the KB https://kb.vmware.com/s/article/2045040 appropriately for the 4 NICs (vmnic6 to vmnic9)...4 vm kernel ports with an IP for each one and so on. All iSCSI related network adapters on ESXi, switch, and FreeNAS were configured with Jumbo frames of 9000. I measured the speed using the I/O Analyzer https://flings.vmware.com/i-o-analyzer. The read/write performance was around 120MB/s. Looking in the switch bandwidth utilization, I found out that only one NIC from ESXi was in use, while the data were being transferred.

Does anyone has a better iSCSI-setup proposal for the ESXi free server?

Thanks

HoneyBadger · Jul 15, 2020

Welcome to the forums.

Your short answer is "Throw LACP out the window, use MPIO instead."

Code:

    ESXi 6.7 Free                   SWITCH                                 FreeNAS
   +--------------+             +------------+                           +----------+
   |              |   VLAN10    |            |        VLAN10             |          |
   |       vmnic0 |-------------|            |---------------------------| msk0     |
   |              |             |            |                           |          |
   |              |             |            |                           |          |
   |       vmnic6 |-------------|            |---------------------------| igb0     |
   |       vmnic7 |-------------|            |---------------------------| igb1     |
   |       vmnic8 |-------------|            |---------------------------| igb2     |
   |       vmnic9 |-------------|            |---------------------------| igb3     |
   |              |    VLAN     |            |           VLAN            |          |
   +--------------+ 11,12,13,14 +------------+       11,12,13,14         +----------+

Use four non-overlapping subnets (VLANs even) and each vmnic/igbX interface on FreeNAS gets its own IP. Add all four FreeNAS IPs to the single iSCSI portal, and scan it as a dynamic target from ESXi. Should detect four paths to your storage. Presto, 4x1Gbps links. You'll need to tune the path selection policy and IOPS switch settings but let's get the network fixed first.

Documentation for reference from FreeNAS:

9. Network — FreeNAS®11.3-U3.2 User Guide Table of Contents

www.ixsystems.com

And from VMware StorageHub:

VMware | The Cloud Platform Tech Zone

Technical information for the core of the Software-Defined Data Center.

storagehub.vmware.com

jgreco · Jul 15, 2020

Won't work the way you wish.

https://www.ixsystems.com/community/threads/lacp-friend-or-foe.30541/

https://www.ixsystems.com/community/threads/multiple-network-interfaces-on-a-single-subnet.20204/

ESXi only vaguely supports what you're trying to do, and only because many storage admins can't configure up an IPv4 network correctly if their life depended on it.

Set up four storage networks the proper way, set round-robin, and be done with it.

jgreco · Jul 15, 2020

HoneyBadger said:

And that's a good example of an alternative bad configuration, please don't put them all on the same VLAN either. A VLAN is a virtual network and each vlan should follow IP networking rules. Each IP network should be its own broadcast domain.

Set up vlan 11 thru vlan 14. Set:
vmnic6 and igb0 on vlan 11, numbered on 192.168.11.0/24
vmnic7 and igb1 on vlan 12, numbered on 192.168.12.0/24
vmnic8 and igb2 on vlan 13, numbered on 192.168.13.0/24
vmnic9 and igb3 on vlan14, numbered on 192.168.14.0/24

EDIT: Apparently that's what HoneyBadger meant anyways. It's good advice.

HoneyBadger · Jul 15, 2020

jgreco said:
And that's a good example of an alternative bad configuration, please don't put them all on the same VLAN either. A VLAN is a virtual network and each vlan should follow IP networking rules. Each IP network should be its own broadcast domain.

Set up vlan 11 thru vlan 14. Set:
vmnic6 and igb0 on vlan 11, numbered on 192.168.11.0/24
vmnic7 and igb1 on vlan 12, numbered on 192.168.12.0/24
vmnic8 and igb2 on vlan 13, numbered on 192.168.13.0/24
vmnic9 and igb3 on vlan14, numbered on 192.168.14.0/24

Literally edited as you were writing that, I didn't notice that it was specified in the diagram as being a single VLAN so I went back to update the ASCII.

Spearfoot · Jul 15, 2020

Wouldn't it be simpler just to pop 10G or 40G NICs into the FreeNAS and ESXi servers for the iSCSI link?

EDIT: Available on ebay:

40GB HP/Mellanox ConnectX-3: https://www.ebay.com/itm/223215337657

10GB SolarFlare SFN6122F: https://www.ebay.com/itm/162014302565

Or similar...

jgreco · Jul 15, 2020

HoneyBadger said:
Literally edited as you were writing that, I didn't notice that it was specified in the diagram as being a single VLAN so I went back to update the ASCII.

Okay, good, I was a little like

(confused) since you usually seem to have your iSCSI head on straight.

Spearfoot said:
Wouldn't it be simpler just to pop 10G or 40G NICs into the FreeNAS and ESXi servers for the iSCSI link?

EDIT: Available on ebay:

40GB HP/Mellanox ConnectX-3: https://www.ebay.com/itm/223215337657

10GB SolarFlare SFN6122F: https://www.ebay.com/itm/162014302565

Or similar...

That's a good point too. The Mikrotik folks have a great selection of inexpensive 10G switches that are perfect for home use, and used gear on eBay is cheap if you need a bigger/more feature-full switch.

Spearfoot · Jul 15, 2020

jgreco said:
Okay, good, I was a little like (confused) since you usually seem to have your iSCSI head on straight.

That's a good point too. The Mikrotik folks have a great selection of inexpensive 10G switches that are perfect for home use, and used gear on eBay is cheap if you need a bigger/more feature-full switch.

That's right. And you wouldn't even need a switch if the two servers are close enough to each other.

Grist · Jul 15, 2020

HoneyBadger said:
Welcome to the forums.

Your short answer is "Throw LACP out the window, use MPIO instead."

Code:
ESXi 6.7 Free SWITCH FreeNAS +--------------+ +------------+ +----------+ | | VLAN10 | | VLAN10 | | | vmnic0 |-------------| |---------------------------| msk0 | | | | | | | | | | | | | | vmnic6 |-------------| |---------------------------| igb0 | | vmnic7 |-------------| |---------------------------| igb1 | | vmnic8 |-------------| |---------------------------| igb2 | | vmnic9 |-------------| |---------------------------| igb3 | | | VLAN | | VLAN | | +--------------+ 11,12,13,14 +------------+ 11,12,13,14 +----------+

Use four non-overlapping subnets (VLANs even) and each vmnic/igbX interface on FreeNAS gets its own IP. Add all four FreeNAS IPs to the single iSCSI portal, and scan it as a dynamic target from ESXi. Should detect four paths to your storage. Presto, 4x1Gbps links. You'll need to tune the path selection policy and IOPS switch settings but let's get the network fixed first.

Documentation for reference from FreeNAS:

9. Network — FreeNAS®11.3-U3.2 User Guide Table of Contents

www.ixsystems.com

And from VMware StorageHub:

VMware | The Cloud Platform Tech Zone

Technical information for the core of the Software-Defined Data Center.

storagehub.vmware.com

jgreco said:
And that's a good example of an alternative bad configuration, please don't put them all on the same VLAN either. A VLAN is a virtual network and each vlan should follow IP networking rules. Each IP network should be its own broadcast domain.

Set up vlan 11 thru vlan 14. Set:
vmnic6 and igb0 on vlan 11, numbered on 192.168.11.0/24
vmnic7 and igb1 on vlan 12, numbered on 192.168.12.0/24
vmnic8 and igb2 on vlan 13, numbered on 192.168.13.0/24
vmnic9 and igb3 on vlan14, numbered on 192.168.14.0/24

EDIT: Apparently that's what HoneyBadger meant anyways. It's good advice.

Thanks all for the feedback! It makes sense...I was "fixed" to LACP due to a very good experience that I have had with HP StoreOnce....
I will implement the network as suggested and give you a feedback.

jgreco said:
Set up four storage networks the proper way, set round-robin, and be done with it.

The ESXi "free" doesn't support round-robin...unfortunately...

Spearfoot said:
Wouldn't it be simpler just to pop 10G or 40G NICs into the FreeNAS and ESXi servers for the iSCSI link?

EDIT: Available on ebay:

40GB HP/Mellanox ConnectX-3: https://www.ebay.com/itm/223215337657

10GB SolarFlare SFN6122F: https://www.ebay.com/itm/162014302565

Or similar...

Thanks for the tip! This is also a good idea, since the servers are close to each other -> no switch needed -> 2nd option as for now

Grist · Jul 17, 2020

I configured the network on the ESXi & FreeNAS as suggested (MTU is set to 9000 again on all sites):

vmnic6 and igb0 on vlan 11, numbered on 192.168.11.0/24
vmnic7 and igb1 on vlan 12, numbered on 192.168.12.0/24
vmnic8 and igb2 on vlan 13, numbered on 192.168.13.0/24
vmnic9 and igb3 on vlan14, numbered on 192.168.14.0/24

I configured the iSCSI Software Adapter on the ESXi as described in the KB https://kb.vmware.com/s/article/2045040 :
- 1 standard switch
- 4 vmkernel/port group for each VLAN...

The disc performance (measured with I/O Analyzer) is the same...and the bandwidth utilization, too -> one NIC was "active"during the tests.

Am I making a mistake with the iSCSI configuration on the FreeNAS?

HoneyBadger · Jul 17, 2020

Grist said:
The ESXi "free" doesn't support round-robin...unfortunately...

While it doesn't support LACP in the free version, I'm reasonably confident that MPIO still works.

Grist said:
I configured the network on the ESXi & FreeNAS as suggested (MTU is set to 9000 again on all sites):

vmnic6 and igb0 on vlan 11, numbered on 192.168.11.0/24
vmnic7 and igb1 on vlan 12, numbered on 192.168.12.0/24
vmnic8 and igb2 on vlan 13, numbered on 192.168.13.0/24
vmnic9 and igb3 on vlan14, numbered on 192.168.14.0/24

I configured the iSCSI Software Adapter on the ESXi as described in the KB https://kb.vmware.com/s/article/2045040 :
- 1 standard switch
- 4 vmkernel/port group for each VLAN...

The disc performance (measured with I/O Analyzer) is the same...and the bandwidth utilization, too -> one NIC was "active"during the tests.

Am I making a mistake with the iSCSI configuration on the FreeNAS?

Did you configure port binding on your vmkernel adaptors? This shouldn't be necessary if your VLANs have no routing between them (which they shouldn't here) - as port binding is intended for when you have multiple IPs in the same subnet (which we're trying to avoid.)

Regarding your throughput issues, check to see that you see four paths in VMware, and you should be able to change the path selection policy (PSP) through the VMware storage configuration for the datastore to VMW_PSP_RR - it's probably currently set to VMW_PSP_MRU (Most Recently Used) which of course is only going to allow a single data path.

Grist · Jul 17, 2020

VMW_PSP_RR (round robin) option is not available in the ESXi free version

The vSwitch config is as follows:

whereby the options for "Load balancing" are:

the port groups, e.g.:

HoneyBadger · Jul 17, 2020

That load balancing is for the network teaming, not storage multipathing.

Not sure what exact version of ESXi you're using but it looks a bit like the legacy Flash-based UI - check the screenshot below for an example. Find that screen for your datastore and choose "Edit Multipathing" - you should then be able to change to Round-Robin. Once that's done there's an extra tweak that can be put in place (IOPS switch threshold) but let's get you using all four paths first.

Edit: My mistake, it's been a while. If you're using ESXi free, you won't have any visibility to those goodies from within the host UI. They'll still work, but we need to ssh into your host to set them. One moment, I'll find some lines for you.

Edit 2: Here's your line that you need to enter into an ssh session - this will create a new VMware claim rule to set all FreeNAS disks to VMW_PSP_RR

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "FreeNAS" -M "iSCSI Disk" -P "VMW_PSP_RR"

You can also add (at the end of the line) the parameter -O "iops=1" - this tells VMware to switch paths every I/O as opposed to the default of switching after 1000 IOPS. For situations where your array throughput can exceed a single link speed (like a multiple-gigabit-link setup) it's worthwhile, but for users with 10Gbps networking it may not be as valuable. Test before and after. (Or test with different numbers. It's your network, after all. ;) )

You'll have to restart the host machine to have it apply the new claim rules to LUNs. You can change it live on the fly with esxcli storage nmp device set --device naa.id.goes.here --psp VMW_PSP_RR but adding a claim rule will ensure that all future devices are handled the same way.

Grist · Jul 20, 2020

HoneyBadger said:
esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "FreeNAS" -M "iSCSI Disk" -P "VMW_PSP_RR"

You can also add (at the end of the line) the parameter -O "iops=1" - this tells VMware to switch paths every I/O as opposed to the default of switching after 1000 IOPS. For situations where your array throughput can exceed a single link speed (like a multiple-gigabit-link setup) it's worthwhile, but for users with 10Gbps networking it may not be as valuable. Test before and after. (Or test with different numbers. It's your network, after all. ;) )

Thank you!!

We are coming closer to full utilization of the 4 NIC's :)

Let me pls summarize so far...only 1 NIC (out of 4 NICs) was utilized over the iSCSI connectivity to FreeNAS. I was getting a throughput of around 137MB/s by reading/writing.

After adding the command (and rebooting afterwards):

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "FreeNAS" -M "iSCSI Disk" -P "VMW_PSP_RR" -O "iops=1"

2 NICs (out of 4 NICs) were utilized over the iSCSI connectivity to FreeNAS. I am getting a throughput of around 235MB/s by reading/writing now

.

Any idea, why only 2 NICs (out of 4 NICs) are used from ESXi "free" for the iSCSI connectivity to FreeNAS?

I can see that only 2 paths are used:

Code:

[root@XXXXXXXXX]:~ esxcli storage nmp device list
naa.6589cfc0000006c11199e68d66726f6d
   Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc0000006c11199e68d66726f6d)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba64:C3:T0:L1, vmhba64:C2:T0:L1
   Is USB: false

HoneyBadger · Jul 20, 2020

VMware side: make sure each of your iSCSI vmkernel NICs has its failover order overridden correctly to only use a single connection.

FreeNAS side: ensure that your iSCSI portal is listening on all four interfaces and they all have a matching IP/VLAN to their "partners" on the VMware machine.

Test from the VMware CLI with vmkping and specify the source interface to ensure that each of them exist as expected.

Grist · Jul 20, 2020

That was my mistake...cabling for VLAN 11 & 12 between FreeNAS & switch was wrong !@&%§$%"§%$"§

Fixed cabling -> 4 iSCSI paths are available now -> the throughput for read/write is around 470MB/s in each case

On the other side...I am glad that I had this wrong configurations so far, since I found out...regarding disk latency over iSCSI:
- 1 path to FreeNAS: disk latency was around 130ms
- 2 paths to FreeNAS: disk latency dropped down to 65ms
- 4 paths to FreeNAS: disk latency dropped down to 33ms

Thanks HoneyBadger once again!

I will make some tests and post my configuration...that might be helpful for others searching the same thing...

Grist · Aug 26, 2020

Hi again,

I added a couple of faster disks to my FreeNAS and was about to post some benchmarks. Unfortunately I get ctl_datamove errors:

Code:

Aug 26 11:23:33 freenas ctl_datamove: tag 0x649a on (3:4:0) aborted
Aug 26 11:23:33 freenas ctl_datamove: tag 0x64ca on (2:4:0) aborted
Aug 26 11:23:33 freenas ctl_datamove: tag 0x649b on (3:4:0) aborted
Aug 26 11:23:33 freenas ctl_datamove: tag 0x64cb on (2:4:0) aborted

Following tasks took place (unfortunately) at the same time:

FreeNAS updated to 11.3-U4.1
ESXi 6.7 updated to Build 16713306 (https://docs.vmware.com/en/VMware-v...html#esxi-6-7-0-20200804001-standard-resolved)

The SATP rule is still there:

Code:

[root@XXXXXXXX:~] esxcli storage nmp satp rule list | grep FreeNAS
VMW_SATP_ALUA                FreeNAS   iSCSI Disk                                                       user                                             VMW_PSP_RR   iops=1

The iSCSI performance is degraded from around 470MB/s down to 80MB/s. I can "see" in the network traffic that all 4 NICs on both sides (ESXi & FreeNAS) are active during the iSCSI operations.

Any idea?

sretalla · Aug 26, 2020

Maybe the update nails those settings... try running the command to check:
esxcli storage nmp satp rule list -s "VMW_SATP_ALUA"

Then re-run as above if you don't see the iops=1 for each one.

Grist · Aug 26, 2020

I thought the same at the beginning...so I deleted the rule, rebooted, added it back again, and rebooted again:

Grist said:
The SATP rule is still there:

Code:
[root@XXXXXXXX:~] esxcli storage nmp satp rule list | grep FreeNAS
VMW_SATP_ALUA FreeNAS iSCSI Disk user VMW_PSP_RR iops=1

The rule for FreeNAS is there.

Important Announcement for The TrueNAS Community.

ESXI 6.7 iSCSI with 4 NICs + FreeNAS with 4 NICs in LACP for iSCSI

Cadet

actually does care

Resident Grinch

Resident Grinch

actually does care

He of the long foot

Resident Grinch

He of the long foot

Cadet

Cadet

actually does care

Cadet

actually does care

Attachments

Cadet

actually does care

Cadet

Cadet

Powered by Neutrality

Cadet

Similar threads

Important Announcement for The TrueNAS Community.