Multiple interfaces and shared IPs

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
My network setup looks as following:
Zrzut ekranu 2020-03-19 o 18.34.18.png


GUI interface is at igb1 - 172.16.0.9.
igb2 is used as shared IPs NIC for jails, therefore it doesn't have assigned any IP - otherwise it won't accept jails IP assignment/alias. I don't know how or I'm not able to setup vlans following instructions in online manual. It simply doesn't work with jails (and maybe with Unifi hardware).
My problem is that main interface - igb1 (172.16.0.9, connected to untagged port) shows up (coming and going and sometimes stays longer) on switch clients list with IP assigned to one of jails (plex). And jail IP is shared IP assigned to other interface igb2, in other subnet (172.16.1.5, connected to tagged port, vlan11).
Config:
Code:
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: SMB
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    inet 172.16.1.54 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.5 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo

Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.0.1         UGS        igb1
10.55.0.0/16       link#2             U          vmx0
10.55.1.2          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0
172.16.0.0/24      link#4             U          igb1
172.16.0.9         link#4             UHS         lo0
172.16.1.0/26      link#3             U          vmx1
172.16.1.2         link#5             UHS         lo0
172.16.1.5         link#5             UHS         lo0
172.16.1.54        link#3             UHS         lo0

So, what's wrong in my setup?
 
Last edited:

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
I'm bumping it up. Can anybody explain me following:
Code:
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: igb1-zarządzanie
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4-Zasoby   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

and
Zrzut ekranu 2020-03-21 o 22.07.18.png


I mean igb1 (untagged network) hijacks one of IP addresses of igb2 (tagged network).
In the meantime igb1 IP address (172.16.0.9) also works - it's GUI address...
What's going on here?
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
You are talking about a tagged network, where are your virtual interfaces for each VLAN?

Confirm your switch is setup correctly first

 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
You are talking about a tagged network, where are your virtual interfaces for each VLAN?

Confirm your switch is setup correctly first

I've been considering jumbo frames as potential source of my problems. But on the other hand, everything else (Synology NAS, cameras, raspbpi, docker containers and wifi clients) has no problems at all. If nothing helps, I will try to change everything to default 1500.
I've changed today my setup to vlans (had to set unifi switch port to trunk mode):
Code:
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Dla Samba
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: igb1-zarządzanie
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c5
    hwaddr ac:1f:6b:d7:fb:c5
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4-Zasoby
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:11:2a:4b:ab:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.5 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 11 vlanpcp: 0 parent interface: igb2
    groups: vlan

I have no idea where bridge0 interface came from - I haven't set it up manually and it's not visible in GUI.
Routing table:
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.1.1         UGS      vlan11
10.55.0.0/16       link#2             U          vmx0
10.55.1.2          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0
172.16.0.0/24      link#4             U          igb1
172.16.0.9         link#4             UHS         lo0
172.16.1.0/26      link#8             U        vlan11
172.16.1.2         link#8             UHS         lo0
172.16.1.3         link#8             UHS         lo0
172.16.1.5         link#8             UHS         lo0

Could default routing has something to do with "floating" main GUI IP? But my main concern is SMB/Netbios broadcast. Synology's broadcast is rock solid when FreeNAS broadcast disappears after around half an hour after restarting SMB service (Netbios link in Finder doesn't work anymore and already opened connection is invalid)...
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
I've been considering jumbo frames as potential source of my problems.

Whenever I am troubleshooting I try to minimise things that could be the problem. I have no experience with Jumbo, but to me it doesn't look right that the bridge has an MTU of 1500 between two interfaces with an MTU of 9000. Just disable Jumbo to make your life easier.

Depending on what version of FreeNAS you are on, jail networking is slightly different. Bridge0 would have been created by the system to attach your jail to, because you likely have not instructed it to do otherwise. Look a the jails Network Properties -> Interfaces

But my main concern is SMB/Netbios broadcast

 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Whenever I am troubleshooting I try to minimise things that could be the problem. I have no experience with Jumbo, but to me it doesn't look right that the bridge has an MTU of 1500 between two interfaces with an MTU of 9000. Just disable Jumbo to make your life easier.

Depending on what version of FreeNAS you are on, jail networking is slightly different. Bridge0 would have been created by the system to attach your jail to, because you likely have not instructed it to do otherwise. Look a the jails Network Properties -> Interfaces
Like I said before, FreeNAS is the only part of my network which has some issues - therefore I would like to investigate it further, prior to changing my whole LAN. I'm on the latest and greatest stable version 11.3-U1.
I've changed configuration from tagged switch ports to trunk ports and setting vlan interface in FreeNAS. Jails connectivity is same in both cases as expected (iocage set ip4_addr="igb2|172.16.1.3/26" vs ip4_addr="vlan11: storage |172.16.1.3/26") but floating of IP addresses has changed. Now on switch side I can see vlan's parent interface toggles its IP between 2 assigned jails:
Zrzut ekranu 2020-03-23 o 12.58.52.png

Zrzut ekranu 2020-03-23 o 12.59.34.png

Is it normal? I'm used to docker's macvlan multiplied interfaces/Mac addresses...
Previously was toggling IPs between GUI and one of jails addresses. Default route is set to subnet, where jails are on, I haven't noticed any change when was set to GUI subnet. Which one is better?

I don't use Windows at all... :cool:
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
I've changed configuration from tagged switch ports to trunk ports

A tagged switch port is a trunk port. Maybe your network hardware has some other terminology!

I don't use Windows at all... :cool:

What do you mean by this then:
But my main concern is SMB/Netbios broadcast.
??


I wouldn't be trusting your switch monitoring, it might not know what to do with an IP alias.
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
A tagged switch port is a trunk port. Maybe your network hardware has some other terminology!
I meant untagged to trunks, I was typing too quick. Unless I completely mixed up definitions ;-)
What do you mean by this then:

??

I wouldn't be trusting your switch monitoring, it might not know what to do with an IP alias.
I use MacBooks (incl. Time Machine) and other Linux based machines with SMB and NFS. Like I said, SMB with Synology doesn't give me a headache.
Anyway, once again I've changed my network settings. I created trunking lagg interface as a base for all data connectivity across my vlans and virtual interface (ESXi, trunk as well) for GUI management. I will test its performance within next days...

Code:
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN3
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    hwaddr ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: storage net esxi
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:f5
    hwaddr 00:0c:29:eb:9e:f5
    inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk
    options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:0c:29:eb:9e:ff
    hwaddr 00:0c:29:eb:9e:ff
    inet 172.16.0.9 netmask 0xffffff00 broadcast 172.16.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN2  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    hwaddr ac:1f:6b:d7:fb:c5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: LAN4
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c7
    hwaddr ac:1f:6b:d7:fb:c7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg4095: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Trunk  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: Zasoby
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    inet 172.16.1.5 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    inet 172.16.1.2 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    groups: vlan
vlan15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: NordVPN
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d7:fb:c6
    inet 172.16.5.5 netmask 0xfffffff0 broadcast 172.16.5.15
    inet 172.16.5.4 netmask 0xfffffff0 broadcast 172.16.5.15
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 15 vlanpcp: 1 parent interface: lagg4095
    groups: vlan
 
Top