Multi-NIC FreeNAS + ESXI Setup via iSCSI - Having some issues

[sbnetadmin]

Hello All!

Overview of our setup
We have two ESXi 5 servers that connect over Gigabit NICs to a Gigabit switch which in-turn connects them to our FreeNAS 8 box via a Gigabit NIC.

What I'm trying to do and why?
We recently realised that our I/O speeds were being limited by the fact that we were maxing out the Gigabit NIC on the FreeNAS. In-turn each of the ESXi machines was only reaching about 50% utilisation of their NICs.

I have added in a new Gigabit NIC to my FreeNAS box and am working on figuring out a way to load balance between the two hosts.

Where am I stuck? (This is the part where you jump in :P)
So I've tried a couple of different ways to achieve this:
1) My first approach involved Link Aggregation but I figured that wouldn't work since I was using iSCSI.
2) I then tried adding the iSCSI disks to each ESXi host with a different IP. This worked splendidly i.e. each ESXi sent its traffic to a different NIC. However, the return traffic was coming back over the default NIC in-turn creating the bottle neck again.

I would really like to figure out a way through which both the ESXi hosts could connect to the FreeNAS box over their own individual NICs. Any help to get me here will be appreciated!

Thanks!

 

[cyberjock]

As an idea(I'm not sure it would work for your exact configuration.. not alot of info included) but you map your data using your 2 NICs as different IPs. For instance, if your FN server uses 192.168.0.100 and 101 have a few machines go to 100 and a few to 101. Then you don't have to deal with link aggregation and the other stuff. Load Balancing doesn't work according to an admin in another thread. But if you do your own IP address sharing you can try to split up the load AND avoid having to purchase more expensive hardware that supports LACP.

 

[sbnetadmin]

As an idea(I'm not sure it would work for your exact configuration.. not alot of info included) but you map your data using your 2 NICs as different IPs. For instance, if your FN server uses 192.168.0.100 and 101 have a few machines go to 100 and a few to 101. Then you don't have to deal with link aggregation and the other stuff. Load Balancing doesn't work according to an admin in another thread. But if you do your own IP address sharing you can try to split up the load AND avoid having to purchase more expensive hardware that supports LACP.

Thanks for the help! I've tried this and it worked brilliantly for incoming traffic (i.e. traffic from ESXi hosts to FreeNAS). However, traffic going from FreeNAS to the ESXi machines would default over the primary NIC. Any idea on how I get it to exit via the same NIC it came from?

 

[cyberjock]

Umm.. I'm not sure how what you say is possible(that data can arrive from NIC2 but will leave only through NIC1. For TCP/IP connections if I contact your server on IP 192.168.0.100 then any data for that connection MUST reply on the same IP(which obviously would be the same NIC). How are you coming to the conclusion that incoming traffic is correct but outgoing traffic is only on one NIC? You currently aren't setup for link aggregation, load balancing, or anything right?

Either I'm very confused about something or you are.. or both I suppose. ;)

 

[sbnetadmin]

Umm.. I'm not sure how what you say is possible(that data can arrive from NIC2 but will leave only through NIC1. For TCP/IP connections if I contact your server on IP 192.168.0.100 then any data for that connection MUST reply on the same IP(which obviously would be the same NIC).

That was my assumption as well, but doesn't seem to be behaving in that manner for some reason.

How are you coming to the conclusion that incoming traffic is correct but outgoing traffic is only on one NIC? You currently aren't setup for link aggregation, load balancing, or anything right?

Based on the network utilization, I had RXs of 300MBPS & 200MBPS and TXs of 500MBPS & 0MBPS. It was a clean install at the time.

Either I'm very confused about something or you are.. or both I suppose. ;)

I certainly am! Going to try the whole setup once again from scratch just to double check everything.

 

[cyberjock]

If you aren't getting any info confused and I understand your configuration then something is very odd.

Any chance you were experimenting with the LACP and have some setting enabled still that is causing this odd behavior. Think of it as trying to contact FTP IP 192.168.0.101 but .100 is responding. There's no way that would work. The FTP client would never care about the traffic from the wrong IP. But if LACP were enabled then both cards share the same IP and all sorts of other weird things happen.

 

[jgreco]

If you aren't getting any info confused and I understand your configuration then something is very odd.

Any chance you were experimenting with the LACP and have some setting enabled still that is causing this odd behavior. Think of it as trying to contact FTP IP 192.168.0.101 but .100 is responding. There's no way that would work. The FTP client would never care about the traffic from the wrong IP. But if LACP were enabled then both cards share the same IP and all sorts of other weird things happen.

FreeBSD's routing table selects destination by destination IP, finding an appropriate network interface that's on the destination network to shoot the traffic out. It doesn't care if the interface has a different IP address; in many cases, such as IP aliases, no native interface will exist with the "right" IP address. Your assumptions are all wrong and your networking is broken as a result.

You've both assumed that you've installed "IP Address Interfaces" in your FreeNAS server. You've installed "Network Interfaces" though. Think about it that way and it's more likely to help you think about it more correctly.

So anyways. Yes, you can use link aggregation. It can and will be frustrating, however, and you're warned of that in advance.

FreeBSD theoretically supports round-robin link aggregation, but even if you get it working, don't use it. The results will be that your ESXi hosts end up getting out-of-order TCP traffic, which will result in the software initiator and its IP stack having to do extra caching and reordering work; it'll hurt performance and chew some extra CPU, and worse, it's non-deterministic in how bad those effects are at any given time (which means it'll seem to work "acceptably" for you until next month when suddenly it breaks horribly).

The remaining linkagg choice is using hashing, which seems like it'd be fine until your traffic all hashes onto one of your two ports, and since you only have two hosts, it's likely to happen and frustrate you. Probably not right away, even.

So the question now becomes: is there a smarter way to do this? And the answer is yes. But it's kind of dependent on your setup.

Do you have more than a single ethernet in your VMware hosts? Can you dedicate one to storage? If so, you can literally connect each host directly to your FreeNAS, use separate RFC1918 space for each network, and gain dedicated gigE connectivity between your NAS and hosts.

A more sophisticated solution involves setting up multiple storage networks (and gaining multipath!) and having one host prefer one path while the other prefers the other.

There are other options as well... but really this requires some more knowledge of the available resources and topology.

 

[cyberjock]

You've both assumed that you've installed "IP Address Interfaces" in your FreeNAS server. You've installed "Network Interfaces" though. Think about it that way and it's more likely to help you think about it more correctly.

I'm really confused. What if I have 2 NICS (192.168.0.100 and .101) and .101 is not physically connected to .100. It would then be necessary to use .101. Of course, using the same subnet mask would really screw things up so maybe this isn't a good example.

 

[paleoN]

I'm really confused.

If so, you can literally connect each host directly to your FreeNAS, use separate RFC1918 space for each network, and gain dedicated gigE connectivity between your NAS and hosts.

Separate being the key word here, i.e. different subnets.

A more sophisticated solution involves setting up multiple storage networks (and gaining multipath!) and having one host prefer one path while the other prefers the other.

IMO, this is the way to go. It's slightly more involved, but a more elegant solution as well.​

 

[jgreco]

I'm really confused. What if I have 2 NICS (192.168.0.100 and .101) and .101 is not physically connected to .100. It would then be necessary to use .101. Of course, using the same subnet mask would really screw things up so maybe this isn't a good example.

So you have two network segments, but you tell the system that they are part of the same network, and then wonder why it doesn't work?

PEBCAK.

Has very little to do with the subnet mask, by the way, except insofar as the subnet mask is one of the things that helps to determine the networking topology.

 

[vanhaakonnen]

Is there any change that LoadBalance / LACP is fixed in 8.3 and work without to build everything with two or more IPs in different Subnets? By the way - does this problem also aperas when using NFS instead of iSCSI?

 

[jgreco]

There's nothing wrong with LACP. This is the way it is intended to work, for good reason. The standard, currently 802.1ax, reads "Frame ordering must be preserved in aggregated links" and "The distribution function maintains frame ordering by [...t]ransmitting frames of a given conversation on a single link at any time" which effectively means that a conversation must occur over a single interface, and the systems go to some effort to implement that requirement.

 

[mattman555]

You might read these two guides... they are very help full... Especially this quote below...

http://doc.freenas.org/index.php/Link_Aggregations
http://doc.freenas.org/index.php/ISCSI#Portals

Heres the quote I mentioned...

Considerations When Using LACP, MPIO, NFS, or ESXi

LACP bonds Ethernet connections in order to improve bandwidth. For example, four phsycial interfaces can be used to create one mega interface.

LACP reads the sender and receiver IP addresses and, if they are deemed to belong to the same TCP connection, always sends the packet over the same interface to ensure that TCP does not need to reorder packets. This makes LACP ideal for load balancing many simultaneous TCP connections, but does nothing for increasing the speed over one TCP connection.

MPIO operates at the iSCSI protocol level. For example, if you create four IP addresses and there are four simultaneous TCP connections, MPIO will send the data over all available links. When configuring MPIO, make sure that the IP addresses on the interfaces are configured to be on separate subnets with non-overlapping netmasks or configure static routes to do point to point communication. Otherwise, all packets will pass through one interface.

LACP and other forms of link aggregation generally do not work well with virtualization solutions. In a virtualized environment, consider the use of iSCSI MPIO through the creation of an iSCSI Portal. This allows an iSCSI initiator to recognize multiple links to a target, utilizing them for increased bandwidth or redundancy. This how-to contains instructions for configuring MPIO on ESXi.

NFS does not understand MPIO. Therefore, you will need one fast interface since creating an iSCSI portal will not improve bandwidth when using NFS. LACP does not work well to increase the bandwidth for point to point NFS (one server and one client). LACP is a good solution for link redundancy or for one server and many clients.

 

[survive]

HI sbnetadmin,

Here is what I did with iSCSI between my filer & ESXi box:

The filer has an Intel quad-port nic in it. I aggregated 2 connections into 1 logical connection for data. I took each remaining port and put them into a portal and gave each interface an IP address on it's own /30 network (so there are exactly 2 usable IPs in each network, one for the filer-side, one for the ESXi -side). On ESXi I have another quad-port NIC and I did the same thing......I trunked 2 ports into it for data into 1 vswitch and connected the other 2 ports into their own individual vswitches, with each vswitch having the "other" IP address form each of the /30 networks. I have 2 LUN's presented to ESXi so I see 4 paths in the "Storage Adapter -> iSCSI Adapter -> paths" screen in the vSphere client, 2 paths for each LUN.

-Will

 

[Brad303]

Survive,

Are you using Round Robin in your ESXi Path Selection?

-Brad

 

[datnus]

@sbnetadmin: Are you able to fix the problem?
I hit the same problem as yours.

I use MPIO for iSCSCi FreeNas. Each ESXi and FreeNas have 2 NICs (in the same subnet)

When ESXi writes to FreeNas, the data goes to FreeNAS using FreeNAS's 2 nic. => Good
However, when I "read" data from FreeNAS, FreeNAS only uses 1 NIC => Not good

 

[William Grzybowski]

They need to be in different subnets, AFAIK.

 

[Dalibor Andzakovic]

For a single subnet MPIO setup you need working pf which implies 9.1. You can then use the reply-to rule to make sure that the reply packet leaves via the same interface it came on.

/etc/pf.conf

Code:

GW="172.16.241.2"
pass in quick on em0 reply-to (em0 $GW)
pass in quick on em1 reply-to (em1 $GW)
pass in quick on em2 reply-to (em2 $GW)
pass in quick on em3 reply-to (em3 $GW)
pass out quick