Register for the iXsystems Community to get an ad-free experience

Best way to link FreeNAS with ESXi via iSCSI with two links

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
122
Hey guys,

first of all thanks to @bigphil and many others for publishing content related to iSCSI here in the Community.
Although I read https://www.codyhosterman.com/2018/05/esxi-iscsi-multiple-subnets-and-port-binding/ and especially https://wahlnetwork.com/2015/03/09/when-to-use-multiple-subnet-iscsi-network-design/ it is still not clear to me what is the best practice in my setup. Often, many guides speak of large storage arrays with active-active designs. However, my setup is relatively small in comparison to that! I only have two servers right now. Each server uses two dedicated nics for iscsi only!

FreeNAS
freenas-iscsi-nic1: 192.168.1.1
freenas-iscsi-nic2: 192.168.2.1
Two Portals: 192.168.1.1:3260 and 192.168.2.1:3260

ESXi
esxi-iscsi-nic1: 192.168.1.2
esxi-iscsi-nic2: 192.168.2.2

freenas-iscsi-nic1 is directly connected, without any switch, etc. to esxi-iscsi-nic1 (10 Gbit/s fiber)
freenas-iscsi-nic2 is directly connected, without any switch, etc to esxi-iscsi-nic2 (10 Gbit/s fiber)

So far, everything is configured and working fine! When I pull one cable (e.g. freenas-icsi-nic1 <-> esxi-iscsi-nic1 cable), then automatically all traffic is sent without interruption via ...-nic2! However, when reading all the guides out there, I have the impression that I did not do this setup correctly!

I try to focus on the most important points:

To put it in simple words: In ESXi every iscsi-nic uses its own port-group and its own vswitch:

vswitch-iscsi-1, which uses iscsi-port-group-1; iscsi-port-group-1 uses vmk1/esxi-iscsi-nic 192.168.1.2
vswitch-iscsi-2, which uses iscsi-port-group-2; iscsi-port-group-2 uses vmk2/esxi-iscsi-nic 192.168.2.2

here is a screenshot of iscsi-port-group1:

iscsi-port-group1.jpg


here is a screenshot of corresponding vswitch-iscsi-1:

vswitch-iscsi-1.jpg


iscsi-port-group2 and vswitch-iscsi-2 are configured accordingly!

Here's a screenshot of when looking into "Configuring iSCSI":

iscsi.jpg


The question I have is:
- Is it correct to use two different subnets for this setup?
- Is it correct to use port binding?

Thank you so much! Help is highly appreciated!
 

HoneyBadger

Mushroom! Mushroom!
Joined
Feb 6, 2014
Messages
4,032
Two Portals: 192.168.1.1:3260 and 192.168.2.1:3260

For this configuration, you should have a single portal with both IPs listening.

The question I have is:
- Is it correct to use two different subnets for this setup?
- Is it correct to use port binding?

Yes to the subnets. Configuring multiple interfaces on the same subnet does not behave as people expect, and you will end up with a bottleneck (although being "bottlenecked" to a single 10Gbps port in a homelab scenario is likely not a real-world limitation.)

No to the port binding. You will have excessively long storage scan/refresh times as your initiators in the 192.168.1.0/24 subnet will try (and fail) to log into the targets in 192.168.2.0/24 - port binding is for single-subnet SANs.
 

bigphil

Patron
Joined
Jan 30, 2014
Messages
486
I agree with @HoneyBadger recommendations along with all of the info I've sent you in our PM conversation.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
122
@HoneyBadger and @bigphil
Thanks for your efforts!
I rebuilt my ESXi setup from scratch and followed all your tips. I also applied multipathing and it works like a charm. So thanks again! I also followed the iSCSI best practice guide I found here in the forums.


However, I'm still not sure if it's a better rule to use one or two vswitches for my setup. For better understanding, I created a picture of my setup:

iscsi.jpg


Right now following configuration is applied at ESXi:
  • iscsi-vswitch-1 has portgroup iscsipg-1 (vmk1: 192.168.1.2) and uses vmnic-1 as uplink
  • iscsi-vswitch-2 has portgroup iscsipg-2 (vmk2: 192.168.2.2) and uses vmnic-2 as uplink
This means every vswitch has one uplink only.

My question is, if it is better when:
  • i would use one vswitch only (instead of two)
  • ... and therefore assign two uplinks to this vswitch (vmnic-1 and vmnic-2)
  • ... and therefore append two portgroups (iscsipg-1/vmk1 and iscsipg-2/vmk2) to this vswitch
I'm not sure if this could be a better setup?

Thank you guys!
 

bigphil

Patron
Joined
Jan 30, 2014
Messages
486
I don't recall if a standard vSwitch allows you to have individual port group overrides. If it does, then 1 vSwitch is fine with two uplinks. In that scenario, you'd then setup teaming/failover overrides on the iSCSI port groups to have only one active uplink per port group...no standby, etc...just one active uplink. If the standard vSwitch doesn't support those overrides, two standard vSwitches would then be used with one active uplink each.
 
Top