Multiple IP Addresses with same subnet on LAGG or NIC-Interface

[develissimo]

Hi folks,


SHORT story:

years ago i thought it was a wise idea to separate connections (NFS, iSCSI, ...) via different IP addresses. So i did use the feature to ADD additional IP Addresses to a network interface.

• I have been curious, but this setup seems to work fine. !? I thought only IP addresses on different non overlapping subnets would be allowed... but that was not the case.

________________________________________________________________________________________


DETAILED story:

We all know such setup (multiple IP addresses with overlapping or same subnet) could lead to problematic package routing.
But i thought:

• Hey: "If the TrueNAS-Team allows us to set multiple IP addresses with same subnet to one single NIC, for sure this problem is solved by some technique" i tested it... it worked... and i did not investigate further.

Right now we do run "successfully" a Link Aggregation LAGG with LACP protocol having 3 Cat Cables attached since some years already. But the scenario is the same if you do not run a Link Aggregation but just add this multiple IP addresses to a single NIC.
FreeNAS/TrueNAS and the connected managed switch are reporting fine about the Link Aggregation as you can see here:
Switch screenshot - reporting LAGG is fine and active:

* switch device: HPE 1920-24G Switch JG924A

Output of command ifconfig:

Code:

ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 30:9c:23:9e:05:b8
    media: Ethernet autoselect (none)
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:8a:0f:b0
    media: Ethernet autoselect
    status: no carrier
    nd6 options=1<PERFORMNUD>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:8a:0f:b1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:8a:0f:b1
    hwaddr a0:36:9f:8a:0f:b2
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:8a:0f:b1
    hwaddr a0:36:9f:8a:0f:b3
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether a0:36:9f:8a:0f:b1
    inet 192.168.0.193 netmask 0xffffff00 broadcast 192.168.0.255
    inet 192.168.0.191 netmask 0xffffff00 broadcast 192.168.0.255
    inet 192.168.0.194 netmask 0xffffff00 broadcast 192.168.0.255
    inet 192.168.0.192 netmask 0xffffff00 broadcast 192.168.0.255
    laggproto lacp lagghash l2,l3,l4
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>

Multiple iSCSI and NFS connections are connected via the IP addresses 192.168.0.191 up to 192.168.0.194 without any recognized issues.
So far so good!

INFO: related to this post: can-lan-port-have-multiple-ip-addresses-of-same-subnet-mask-or-different-subnet

• multiple IP addresses with SAME subnet are allowed.
• multiple IP addresses with OVERLAPPING subnets are disallowed.

But TrueNAS does not give a hint, that OVERLAPPING subnets are discouraged.

• maybe there should be a hint on such overlapping subnets being discouraged!?

QUESTIONs:

• Is it wise to have multiple IP addresses with same subnet on LAGG or NIC-Interface? Wouldn't this lead to routing issues or performance impacts?
• Why was/is it allowed in FreeNAS/TrueNAS GUI to have multiple IP addresses with SAME subnet? How are they separated from each other? Is this a special technique? Will this lead to performance loss or routing issues?
• Why was/is it allowed in FreeNAS/TrueNAS GUI to have multiple IP addresses with overlapping subnets? How are they separated from each other? Is this a special technique? Will this lead to performance loss or routing errors?
• Is having multiple IP addresses with same subnet the proper way to achieve another little step of separation for NFS and iSCSI? Besides from authentication techniques such as CHAP for iSCSI and Authorized Hosts/Authorized IP addresses for NFS etc..
• What is the best way in TrueNAS to detect errors in package routing/network traffic?

Finally:

• So the INFO about overlapping but not identical subnets was just informative and we do not use such setup.
• The multiple IP Addresses with same subnet on the other hand is used actively in our setup and we are concerned if returning back to just one single IP address is advised.

Thanks in advanced for your appreciated feedback.

Kind regards,
Raphael

 

[Patrick M. Hausen]

Multiple addresses in the same subnet on the same interface are perfectly fine. Addresses in the same subnet on different interfaces aren't.

 

[jgreco]

[*]Is it wise to have multiple IP addresses with same subnet on LAGG or NIC-Interface? Wouldn't this lead to routing issues or performance impacts?

Done correctly, IP aliases (the correct term for this) have negligible performance impact unless you are trying to do thousands of them, at which point it does become slightly problematic.

It isn't clear what routing issues you believe this could cause. Again, done correctly, it shouldn't, however, once you get up into the thousands, you do need to be aware that some switchgear has a limited size of MAC forwarding tables, and you can definitely run into a situation where the switch goes into ARPocalypse Now mode.

[*]Why was/is it allowed in FreeNAS/TrueNAS GUI to have multiple IP addresses with SAME subnet? How are they separated from each other? Is this a special technique? Will this lead to performance loss or routing issues?

It's allowed to have multiple IP addresses for the same reason you're allowed to have a single IP address. This is how we do IP networking. If you cannot have an address, your service is unreachable.

[*]Why was/is it allowed in FreeNAS/TrueNAS GUI to have multiple IP addresses with overlapping subnets? How are they separated from each other? Is this a special technique? Will this lead to performance loss or routing errors?

Overlapping networks is generally bad and you generally should not do it. However, FreeNAS is not your mommy and daddy, and there are lots of stupid things it does let you do.

[*]Is having multiple IP addresses with same subnet the proper way to achieve another little step of separation for NFS and iSCSI? Besides from authentication techniques such as CHAP for iSCSI and Authorized Hosts/Authorized IP addresses for NFS etc..

No, it is not "THE proper way". Again, FreeNAS is not your mommy or daddy, and will generally let you set you up things the way you need. It is perfectly fine, and doesn't bother anyone/anything, to have iSCSI and NFS on the same IP. Likewise, it doesn't bother anyone/anything to have them on separate IP's. "THE proper way" is the way in which you need it to work, as long as that is not at odds with proper IP networking design. So as long as you are not doing something tragically bad, YOU get to define "THE proper way" for your own network.

There are definitely things you can do that are tragically bad, such as putting multiple ethernet interfaces on the same network and expecting this to handle traffic input/output on a per-interface basis. That article will explain many of the details of that badness, and also provides insight into your related question of why multiple IP's on a single network works fine.

[*]What is the best way in TrueNAS to detect errors in package routing/network traffic?

You hire a competent server administrator and network administrator (maybe the same person, often not). Your NAS can only do what you tell it to do, but has no magic to analyze whether what you have told it to do is "wrong". FreeNAS is not your mommy or daddy, but these two hires I spoke of are definitely proxies for mommy and daddy. ;-)

[*]So the INFO about overlapping but not identical subnets was just informative and we do not use such setup.
[*]The multiple IP Addresses with same subnet on the other hand is used actively in our setup and we are concerned if returning back to just one single IP address is advised.

Hope this has cleared it up for you.

 

[develissimo]

Thank you @Patrick M. Hausen and @jgreco for your responses and clearing things up for me. Also thanks to jgreco for the explanation about the minimal overhead and performance impact on using multiple IPs on a single interface or LAG. So as fare as one stays away from the thousands, there shouldn't be any concerns. Hopefully this helps others with similar questions too.

Kind regards,
Raphael

 

[Chris Moore]

I know this has been answered but I thought I would throw this additional example in on top. Effectively, a virtualization host is another situation where multiple IP addresses share the same physical NIC. The MAC of the physical NIC is the same, only the IP address is different between the different virtual machines that share the physical NIC.

 

[Patrick M. Hausen]

The MAC of the physical NIC is the same, only the IP address is different between the different virtual machines that share the physical NIC.

True (hypervisor) VMs have their own MAC address for the virtual NIC. As do VNET jails.

 

[Chris Moore]

True (hypervisor) VMs have their own MAC address for the virtual NIC. As do VNET jails.

Yes, but the physical switch outside the hypervisor is still sending traffic to the MAC of the physical NIC of the hardware running the hypervisor. The details of how it works can be different by platform. I worked with one where there was a virtual switch configured inside the hypervisor that connected to the physical NIC and all the VMs were connected to the virtual switch.

The point is, more than one IP address can go to a single physical NIC on a host.

 

[Patrick M. Hausen]

The point is, more than one IP address can go to a single physical NIC on a host.

This is of course correct.

Yes, but the physical switch outside the hypervisor is still sending traffic to the MAC of the physical NIC of the hardware running the hypervisor.

But this isn't in the general case, IMHO.

With TrueNAS all VMs and VNET jails are visible with their own MAC addresses "on the wire":

Code:

$ ping freenas.local # TrueNAS host
PING freenas.local (192.168.1.10): 56 data bytes
64 bytes from 192.168.1.10: icmp_seq=0 ttl=64 time=0.669 ms

$ ping wiki-pmh.local # Ubuntu VM
PING wiki-pmh.local (192.168.1.21): 56 data bytes
64 bytes from 192.168.1.21: icmp_seq=0 ttl=64 time=1.405 ms

$ ping cloud.local # VNET jail
PING cloud.local (192.168.1.53): 56 data bytes
64 bytes from 192.168.1.53: icmp_seq=0 ttl=64 time=0.938 ms

$ arp -na | egrep '192\.168\.1\.(10|21|53)'
? (192.168.1.10) at ac:1f:6b:76:64:1c on en11 ifscope [ethernet]
? (192.168.1.21) at 0:a0:98:6a:81:e3 on en11 ifscope [ethernet]
? (192.168.1.53) at 2:ff:60:7:4c:d7 on en11 ifscope [ethernet]

And even with ESXi:

Code:

$ ping vm1 # ESXi host
PING vm1.intern.punkt.de (217.29.45.21): 56 data bytes
64 bytes from 217.29.45.21: icmp_seq=0 ttl=61 time=1.567 ms

$ ping admin # VM
PING admin.intern.punkt.de (217.29.45.101): 56 data bytes
64 bytes from 217.29.45.101: icmp_seq=0 ttl=125 time=1.260 ms

$ arp -na | egrep '217\.29\.45\.(21|101)'
? (217.29.45.101) at 00:50:56:83:f6:7d on igb0 expires in 1193 seconds [ethernet]
? (217.29.45.21) at 00:26:2d:06:07:32 on igb0 expires in 1191 seconds [ethernet]

 

[jgreco]

Yes, but the physical switch outside the hypervisor is still sending traffic to the MAC of the physical NIC of the hardware running the hypervisor.

This is patently false. There would be no way to run virtual machines with ethernet connections if it was true, except possibly to put it behind a host NAT (which is an option on some workstation-grade hypervisors such as Fusion/Workstation), and in such cases, the VM's are not reachable from the real network.

The point is, more than one IP address can go to a single physical NIC on a host.

That is true, but that is simply a matter of having ARP respond with requests for the solicited IP with the MAC of the host. Because each VM has its own IP stack, each one needs a separate MAC address so that the vSwitch can identify the traffic destination and forward it correctly. This is the same reason that real ethernet cards each have a unique MAC address.