I found a problem.
My network setup looks like this:
OPNsense VM -> vmbr2 (bridge) -> bond0.2 (vlan) -> bond0 (failover) -> two NICs -> Switch <- two NICs <- lagg0 (failover) <- vlan2 <- bridge2 <- OPNsense2 VM
If I tcpdump vmbr2 I see packets from both VMs.
If I tcpdump lagg0 I see packets from both VMs.
If I tcpdump vlan2 or bridge2 I see only packets from OPNsense2 (the VM on the FreeNAS host).
Any idea how CARP packets can pass vlan2 to lagg0 but get lost if traveling from lagg0 to vlan2?
192.168.0.3 is my WAN on the OPNsense2 VM (on FreeNAS).
192.168.0.2 is my WAN on the OPNsense VM (on Proxmox).
Tcpdump vlan2:
Code:
tcpdump -B 16000 -i vlan2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 204.26.121.131,113.231.163.36,111.27.69.51,139.52.213.104,103.208.241.213,47.0.182.224,252.5.96.64
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 41.158.64.171,149.16.125.193,4.191.90.167,232.125.20.154,100.184.50.211,12.127.42.235,115.221.179.11
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 56.238.168.180,162.56.17.240,137.1.45.56,91.38.150.2,7.134.32.57,88.104.65.61,66.78.169.91
Tcpdump lagg0:
Code:
tcpdump -B 16000 -i lagg0 -s0 -vv -n | grep VRRPv2
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.21,135.66.16.237,211.4.211.42,117.193.93.14,212.76.235.95,104.10.216.75
Tcpdump vmbr2:
Code:
tcpdump -B 16000 -i vmbr2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.87,70.65.213.181,67.105.250.170,222.9.115.236,50.252.143.196,169.224.5.94
192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.88,58.219.37.12,15.210.142.91,192.209.83.22,41.242.4.18,137.13.56.178
192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.89,16.104.17.98,215.206.131.78,42.128.191.9,19.252.151.219,186.215.188.18
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.99,48.110.165.149,172.62.82.112,114.186.137.221,186.29.255.226,22.90.25.111
192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.100,86.107.71.132,149.44.192.90,221.144.124.56,111.128.79.123,117.35.199.20
192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.39,133.190.110.130,88.215.243.94,211.204.175.144,249.115.200.64,153.255.214.192
192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.40,151.189.205.255,16.22.81.156,215.5.36.88,254.222.251.113,140.137.102.211
And another strange thing...
If I look what my ISPs router (192.168.0.1) it lists me this as hosts on my Network:
OPNsense VM is on, OPNsense2 VM is off:
FE:41:DC:03:E2:67 192.168.0.4
00:00:5E:00:01:01 192.168.0.2
OPNsense VM is off, OPNsense2 VM is on:
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.3
OPNsense VM is on, OPNsense VM is on:
FE:41:DC:03:E2:67 192.168.0.4
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.2 or 192.168.0.3 but never together
192.168.0.4 is my virtual WAN IP and 192.168.0.2 + 192.168.0.3 are the "physical" WAN IPs of both VMs.
Why are the two "physical" IPs using a identical MAC and and the virtual IPs MAC changes depending on which OPNsense it belongs to?
I thought the idea was that only the virtual IPs MAC gets spoofed but here it is the MAC of the "physical" IPs.
If I want to create a exposed host or a port forward on my ISPs router I need a select a host on the network and that is defined by the unique MAC. So if two OPNsenses are sharing the same IP it only make sense to me if that shared virtual IP also get the MAC spoofed so it is always the same to let my ISPs router think, that nothing changed, if the OPNsenses are doing a failover.
ifconfig vtnet1 on OPNsense VM (the one on the Proxmox host):
Code:
ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
ether fe:41:dc:03:e2:67
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255 vhid 1
inet6 xxx%vtnet1 prefixlen 64 scopeid 0x2
carp: MASTER vhid 1 advbase 1 advskew 0
media: Ethernet 10Gbase-T <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ifconfig vtnet1 on OPNsense2 VM (the one on the FreeNAS host):
Code:
ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:6f:54:71
inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255 vhid 1
inet6 xxx%vtnet1 prefixlen 64 scopeid 0x2
carp: MASTER vhid 1 advbase 1 advskew 100
media: Ethernet 10Gbase-T <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>