10gbs switchless bridge issue

[Swampass]

Hi all, I have a strange problem that I cant seem to figure out. I am attempting to link two R710s(esxi) to a R510(FreeNAS) with a dual 10gbs sfp+ nic. (link to a similar setup)
VMware 6.0 expects each host to see a datastore on the same IP in order for DRS/HA to work, which is why the bridge is necessary instead of configuring two different /30 subnets for each host.

FreeNAS version: 11 u1
FreeNAS nic: HP NC550SFP 10GB 2-Port (latest fw)
ESXi nics: HP 10GB CONNECTX2 single port
FreeNAS to host: oce0 -> esxi02, oce1-> esxi01

The problem I am having is that when i configure the bridge, I get arp traffic to the host via tcpdump, but the host responds and freenas never gets the entry in arp.
currently oce0 is configured in the webui as 10.10.10.2/28, and esxi02 can mount nfs and pass traffic (it also works in reverse if i swap the dac cables esxi01 will work as well)

What am I missing? Any ideas are appreciated.

here is my ifconfig

Code:

$ ifconfig bridge0
bridge0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
		ether 02:61:6e:08:fe:00
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: oce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 4 priority 128 path cost 2000
		member: oce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 3 priority 128 path cost 2000


$ ifconfig oce0
oce0: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
		options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO>
		ether d4:85:64:23:54:b0
		inet 10.10.10.2 netmask 0xfffffff0 broadcast 10.10.10.15
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-SR <full-duplex>)
		status: active


$ ifconfig oce1
oce1: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
		options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO>
		ether d4:85:64:23:54:b4
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-SR <full-duplex>)
		status: active

$ arp -a | grep 10.10
? (10.10.10.2) at d4:85:64:23:54:b0 on oce0 permanent [ethernet]
? (10.10.10.3) at (incomplete) on oce0 expired [ethernet]
? (10.10.10.4) at 00:50:56:6a:6a:16 on oce0 expires in 782 seconds [ethernet]

and tcpdump on vmware side answering arp when i ping the host from freenas

Code:

$ ping 10.10.10.3
PING 10.10.10.3 (10.10.10.3): 56 data bytes
--- 10.10.10.3 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss


[root@esxi01:~] tcpdump-uw -i vmk2
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:45:26.150393 ARP, Request who-has 10.10.10.3 tell 10.10.10.2, length 46
17:45:26.173941 ARP, Reply 10.10.10.3 is-at 00:50:56:68:46:e5 (oui Unknown), length 28

 

[Thomas102]

Hi,

I'm new to FreeBSD, but on linux you must define the IP on the bridge, not on the interface.
In your case on bridge0, not oce0.

 

[Swampass]

I have tried assigning an ip only to bridge0, and could not ping it from either hosts. The only way I could get at least one interface to work was assigning an IP in the UI to a nic directly.
I used

Code:

 ifconfig bridge0 inet 10.10.10.2/28 

and made sure to ifconfig up all interfaces with no success.

 

[Thomas102]

You should check this related thread : https://forums.freenas.org/index.php?threads/bridge-between-network-interfaces.41566/