Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Allow mac spoofing inside bhyve VM?

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
Hi,

I want to setup a OPNsense-VM with HA configuration and that needs CARP to be working and that needs to be able to change the MAC of the virtio NIC.
I already found out how to allow that for Proxmox and Esxi but I wan't able to find out how to do that with bhyve. Looks like my FreeNAS 11.3U4.1 got no option in GUI to allow that.
Can someone tell me if that is somehow possible using CLI?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
2,695
Expand the < at the right of the VM, and click the Devices button.

1613654243987.png


Click the 3 dots to the right of the VM's NIC, and select Edit.

1613654298356.png


You should be able to change the MAC to whatever you like. Click Save. Note, the new MAC only takes effect on the next start of the VM.

1613654352819.png
 

sretalla

Wizened Sage
Joined
Jan 1, 2016
Messages
3,936
I think the ask was to do it via CLI...

Maybe middlewared can do it via a midclt call... using vm.device.query to have a look for the name and vm.device.update to make the change
 
Last edited by a moderator:

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
I dont want to change the MAC by myself from outside the VM. I need to use virtual IPs so two VMs on different hosts can share the same IP and MAC. To do that CARP is used by OPNSense but for that OPNsense inside the VM needs to be able to change the MAC of the virtual NIC. That is normally prohibited by hypervisors because they use some kind of MAC filtering. The question is how to disable this so the guest is allowed to change the MAC.
 

sretalla

Wizened Sage
Joined
Jan 1, 2016
Messages
3,936
The question is how to disable this so the guest is allowed to change the MAC.
Have you tried it? maybe it's not necessary to disable anything.
 

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
I tried it but something with CARP isn't working right and I tried to figure out why.
 

Patrick M. Hausen

Dedicated Sage
Joined
Nov 25, 2013
Messages
2,767
If you can run your application in a jail, that definitely works. We had CARP deployed and running in jails for a project.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
2,695

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
I found a problem.

My network setup looks like this:
OPNsense VM -> vmbr2 (bridge) -> bond0.2 (vlan) -> bond0 (failover) -> two NICs -> Switch <- two NICs <- lagg0 (failover) <- vlan2 <- bridge2 <- OPNsense2 VM

If I tcpdump vmbr2 I see packets from both VMs.
If I tcpdump lagg0 I see packets from both VMs.
If I tcpdump vlan2 or bridge2 I see only packets from OPNsense2 (the VM on the FreeNAS host).

Any idea how CARP packets can pass vlan2 to lagg0 but get lost if traveling from lagg0 to vlan2?

192.168.0.3 is my WAN on the OPNsense2 VM (on FreeNAS).
192.168.0.2 is my WAN on the OPNsense VM (on Proxmox).

Tcpdump vlan2:
Code:
tcpdump -B 16000 -i vlan2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 204.26.121.131,113.231.163.36,111.27.69.51,139.52.213.104,103.208.241.213,47.0.182.224,252.5.96.64
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 41.158.64.171,149.16.125.193,4.191.90.167,232.125.20.154,100.184.50.211,12.127.42.235,115.221.179.11
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 56.238.168.180,162.56.17.240,137.1.45.56,91.38.150.2,7.134.32.57,88.104.65.61,66.78.169.91


Tcpdump lagg0:
Code:
tcpdump -B 16000 -i lagg0 -s0 -vv -n | grep VRRPv2
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
    192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
    192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
    192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
    192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.21,135.66.16.237,211.4.211.42,117.193.93.14,212.76.235.95,104.10.216.75


Tcpdump vmbr2:
Code:
tcpdump -B 16000 -i vmbr2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.87,70.65.213.181,67.105.250.170,222.9.115.236,50.252.143.196,169.224.5.94
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.88,58.219.37.12,15.210.142.91,192.209.83.22,41.242.4.18,137.13.56.178
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.89,16.104.17.98,215.206.131.78,42.128.191.9,19.252.151.219,186.215.188.18
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.99,48.110.165.149,172.62.82.112,114.186.137.221,186.29.255.226,22.90.25.111
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.100,86.107.71.132,149.44.192.90,221.144.124.56,111.128.79.123,117.35.199.20
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.39,133.190.110.130,88.215.243.94,211.204.175.144,249.115.200.64,153.255.214.192
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.40,151.189.205.255,16.22.81.156,215.5.36.88,254.222.251.113,140.137.102.211


And another strange thing...

If I look what my ISPs router (192.168.0.1) it lists me this as hosts on my Network:

OPNsense VM is on, OPNsense2 VM is off:
FE:41:DC:03:E2:67 192.168.0.4
00:00:5E:00:01:01 192.168.0.2

OPNsense VM is off, OPNsense2 VM is on:
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.3

OPNsense VM is on, OPNsense VM is on:
FE:41:DC:03:E2:67 192.168.0.4
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.2 or 192.168.0.3 but never together

192.168.0.4 is my virtual WAN IP and 192.168.0.2 + 192.168.0.3 are the "physical" WAN IPs of both VMs.
Why are the two "physical" IPs using a identical MAC and and the virtual IPs MAC changes depending on which OPNsense it belongs to?
I thought the idea was that only the virtual IPs MAC gets spoofed but here it is the MAC of the "physical" IPs.
If I want to create a exposed host or a port forward on my ISPs router I need a select a host on the network and that is defined by the unique MAC. So if two OPNsenses are sharing the same IP it only make sense to me if that shared virtual IP also get the MAC spoofed so it is always the same to let my ISPs router think, that nothing changed, if the OPNsenses are doing a failover.

ifconfig vtnet1 on OPNsense VM (the one on the Proxmox host):
Code:
ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether fe:41:dc:03:e2:67
        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
        inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255 vhid 1
        inet6 xxx%vtnet1 prefixlen 64 scopeid 0x2
        carp: MASTER vhid 1 advbase 1 advskew 0
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


ifconfig vtnet1 on OPNsense2 VM (the one on the FreeNAS host):
Code:
 ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:6f:54:71
        inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
        inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255 vhid 1
        inet6 xxx%vtnet1 prefixlen 64 scopeid 0x2
        carp: MASTER vhid 1 advbase 1 advskew 100
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
Last edited:

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
I think I now know whats wrong.

The lagg0 consists of a ConnectX3 (mcx311a-xcat) and a Intel i217. The lagg0 is showing me CARP packets of the other server but don't let them pass to the "vlan2" because only the Intel i217 is receiving CARP packets but the ConnectX3 is the Master of the failover bond.
So I can verify that the ConnectX3 is sending CARP packets to the switch and I can verify that the switch is sending CARP packets to the FreeNAS server.

So why isn't the ConnectX3 receiving CARP packets if it can send them?
 

Dunuin

Member
Joined
Mar 7, 2013
Messages
61
I flashed the ConnectX3 NICs to the latest firmware and swapped them between the servers. Same problem. FreeNAS will send but not receive CARP packets on the ConnectX3 NIC. Again, all fine on the Proxmox host. I also removed the switch and tested it with just a DAC connecting both ConnectX3 NICs. Still the same. So the switch isn't the problem.

Only thing that I now can think off is, that the FreeBSD driver got problems or there are some configs that prevent die ConnectX3 on FreeNAS to receive CARP Multicast packets.

I can't do much to edit the ConnectX3s configuration if I unterstand it right. All the Mellanox documentations tell me to use ethtool to change configs but that seems not to be available on FreeBSD. I read the ifconfig manual to disable some offloading things and used "-rxcsum -txcsum -rxcsum6 -txcsum6 -tso -lro -vlanmtu -vlanhwtag -vlanhwfilter -vlanhwtso -vlanhwcsum" but it looks like many stuff that ethtool can do is just not
supported with ifconfig.

That I'm not allowed to install new packages to the FreeNAS host itself makes the things also not easier.

Any ideas?

Edit:
What I find strange is that FreeNAS loads the mlx4_core driver and the mlx5en driver. The mlxen5 is for the ConnectX4. The right driver for my ConnectX3 would be mlx4en. I can see that mlx4en is used to initialize the NIC but I can't find any entry in the logs that the mlx4en driver is loaded. There should be a like that tell the mlx4en version:
Code:
dmesg | grep mlx
mlx5en: Mellanox Ethernet driver 3.5.1 (April 2019)
mlx4_core0: <mlx4_core> mem 0xf7400000-0xf74fffff,0xf5800000-0xf5ffffff irq 16 at device 0.0 on pci1
mlx4_core: Mellanox ConnectX core driver v3.5.1 (April 2019)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: f4:52:14:89:23:b0
mlx4_en: mlx4_core0: Port 1: Using 8 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 8 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
mlxen0: tso4 disabled due to -txcsum.
mlxen0: tso6 disabled due to -txcsum6.
mlxen0: link state changed to UP
mlxen0: promiscuous mode enabled
mlx4_en: mlxen0: Link Down
mlxen0: link state changed to DOWN
mlxen0: link state changed to UP
 
Last edited:
Top