iscsi - LACP (lagg) - MPIO and vlans

[dpbrewer]

Got a quick question .. been fighting with this for about a week on an off. I think have the answer but would like to make sure i'm not missing something else.

I know ive read the preferred method on ISCI is to use 1 interface per connection
---create 4 interfaces on your server/freenas, give each a non overlapping IP range
Not to create a single 4 port LAGG and expect 4G of throughput. you get a single thread across a single 1G link.

what about creating the 4 port LAGG trunk and utilizing 4+ vlans within that link? Cant seem to find anyone that has documented reasons for or against it. Seems like it would work, in the office we have done similar, of course we are using them on much more than a host to host and normal LAGG works when you have more than 4 hosts using them.

What i think i found is when using windows server it does not support MPIO over vlans and thats been the issue iver been running into.

Will have to see if i can get my ESXI box up to do some further testing on something other than windows too :)



Thoughts?

 

[kdragon75]

Got a quick question

Nothing quick about any of this...

I know I've read the preferred method on ISCI is to use 1 interface per connection

Its one interface per SUBNET. this is a networking standard, not just for iSCSI. The homelab world and even a lot of vendors get this dead wrong (looking at you Dell). It's totally normal to have several iSCSI "connections" (are we talking TCP sessions or iSCSI sessions?) over one physical interface. That's the only way it can scale to work in a datacenter.

---create 4 interfaces on your server/freenas, give each a non overlapping IP range

Not only should they not be overlapping but also on separate layer two networks (think switches) or at least vlans. Please don't ever run multiple subnets on one layer 2 segment. Please. (looking at every hack MSP out there) One of the big reasons for this it to allow setting COS and prioritizing storage traffic (if you are sharing bandwidth with other services on the same network)

Not to create a single 4 port LAGG and expect 4G of throughput. you get a single thread across a single 1G link.

It not about processor threads (generally) its about connections and sessions. you can have multiple iSCSI sessions over one iSCSI TCP session but because that's all one TCP session it will al get routed over the same physical port so yeah you're still limited to 1 1gb link PER TCP SESSION. In theory if you have two hosts connecting to the same LAGG, you could (depending on a lot of things) see 2gb OUT from the server but each host would only get 1gb (someone correct me if it got the details on this one wrong)

what about creating the 4 port LAGG trunk and utilizing 4+ vlans within that link? can't seem to find anyone that has documented reasons for or against it. Seems like it would work, in the office we have done similar, of course we are using them on much more than a host to host and normal LAGG works when you have more than 4 hosts using them.

In theory this should work even if there is no reason for it but it wont. Don't use LACP/LAGGs/Port channels on your storage hosts. Between switches (matching of course) this all works great but not for your storage hosts. If you need redundancy, let iSCSI handle it with multiple subnets/VLANs. Even without fancy MPIO, if the intator knows about all paths to a given target it should failover to the next path as needed (at least in vmware unless you are using fixed paths).

If you have two mechanisms trying to handle the failover at the same time you just asking for problems. If a cable/card goes down LACP will try to move the traffic over and at the same time iSCSI will be doing the same thing further up the stack. While this MAY work fine, it may also cause delays and unstable performance for longer than needed. iSCSI was made to handle path failover, let it do its job.

Will have to see if i can get my ESXI box up to do some further testing on something other than windows too :)

They will definitely behave differently. ESXi includes a basic MPIO driver that uses a simple round robin per IO (configurable via CLI on the hosts defaults to 1000 IO best setting depends on a number of factors.)

 

[kdragon75]

Long story short, don't do it.