Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Bridging - 2 ESXi connected using FreeNAS Dual Port NIC

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

SanEerf

Cadet
Joined
Oct 14, 2017
Messages
2
I'm at my wits end with a bridge config and need a sounding board.

To begin, yes I realize this isn't a supported, recommended, whatever config for FreeNAS.

My topology is 2 ESXi hosts with 2x10gb NICs and the FreeNAS server with a 2 port 10gb NIC.
I had this all connected to 2 nexus 5010's to dork around with VPC and other nexus stuff but the power usage is too high for a 24x7 runtime.

I've done bridging before in Linux for other projects and it just worked. I browsed forum threads here and elsewhere but didn't find any newer posts, a lot of older posts with configs that I tried.



Couple of threads I followed:

https://forums.freenas.org/index.ph...1-bridging-ethernet-change.59876/#post-424525 - I'm using his identical commands, with my interfaces swapped in, at post-init to build my bridge

https://forums.freenas.org/index.ph...-without-10gb-switch.25259/page-3#post-161894

https://forums.freenas.org/index.php?threads/bridging-10gbe-with-chelsio-t420-cr.44605/

https://forums.freenas.org/index.ph...-drops-when-client-reboots.53052/#post-366851



I tried init scripts, tunables, just making the bridge from the cli the standard bsd/nix way but I cannot get traffic to pass across the bridge cleanly. I also tried creating a jail to let FreeNAS naturally create the bridge and then add my interfaces to that bridge. I'm inclined to believe its an ARP issue due to what I am seeing in the arp table.

What I see is that neither ESXi host can ping across the bridge to the other ESXi host. Only the ESXi 10.10.10.4 host connected to oce0 can ping the FreeNAS IP config'd on oce0 via the web interface. The ESXi host at 10.10.10.14 connected to oce1 cannot ping FreeNAS or the other ESXi host across the bridge. All hosts are seen in the arp table but the ESXi host on oce1 is seen twice, once "incomplete", I'm guessing due to the bridge. I tried disabling arp and doing static arp entries but still no luck. On both ESXi I tried enabling and disabling Promiscuous as well. It's gotta be an issue with arp and bridging. See below for config.
Code:
root@freenas:~ # arp -a

? (10.10.10.14) at 00:50:56:61:54:ee on oce1 expires in 1200 seconds [ethernet]

? (10.10.10.14) at (incomplete) on oce0 expired [ethernet]

? (10.10.10.2) at 2c:44:fd:e1:33:70 on oce0 permanent [ethernet]

? (10.10.10.4) at 00:50:56:66:55:1c on oce0 expires in 1113 seconds [ethernet]



root@freenas:~ # ping 10.10.10.14

PING 10.10.10.14 (10.10.10.14): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 10.10.10.14 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss



root@freenas:~ # ping 10.10.10.4

PING 10.10.10.4 (10.10.10.4): 56 data bytes
64 bytes from 10.10.10.4: icmp_seq=0 ttl=64 time=0.303 ms
64 bytes from 10.10.10.4: icmp_seq=1 ttl=64 time=0.375 ms
64 bytes from 10.10.10.4: icmp_seq=2 ttl=64 time=0.221 ms
^C

--- 10.10.10.4 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.221/0.300/0.375/0.063 ms



root@freenas:~ # ifconfig

~Removed Interfaces not related to config~

oce0: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO>
ether 2c:44:fd:e1:33:70
hwaddr 2c:44:fd:e1:33:70
inet 10.10.10.2 netmask 0xffffff00 broadcast 10.10.10.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active

oce1: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO>
ether 2c:44:fd:e1:33:74
hwaddr 2c:44:fd:e1:33:74
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active

bridge100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:6e:6b:57:d2:64
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: oce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
		ifmaxaddr 0 port 4 priority 128 path cost 2000
member: oce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
		ifmaxaddr 0 port 3 priority 128 path cost 2000





ESXI1 - 10.10.10.4 - Atached to oce0
[root@esxi2:~] esxcli network ip neighbor list
Neighbor	 Mac Address		Vmknic	 Expiry  State  Type
-----------  -----------------  ------  ---------  -----  -------
10.10.10.14  00:50:56:61:54:ee  vmk2	 1198 sec		 Unknown
10.10.10.2   2c:44:fd:e1:33:70  vmk2	 1196 sec		 Unknown

[root@esxi2:~] ping 10.10.10.4
PING 10.10.10.4 (10.10.10.4): 56 data bytes
64 bytes from 10.10.10.4: icmp_seq=0 ttl=64 time=0.057 ms
64 bytes from 10.10.10.4: icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from 10.10.10.4: icmp_seq=2 ttl=64 time=0.068 ms

--- 10.10.10.4 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.057/0.066/0.073 ms
[root@esxi2:~] ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2): 56 data bytes
64 bytes from 10.10.10.2: icmp_seq=0 ttl=64 time=0.567 ms
64 bytes from 10.10.10.2: icmp_seq=1 ttl=64 time=0.448 ms

--- 10.10.10.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.448/0.507/0.567 ms
[root@esxi2:~] ping 10.10.10.14
PING 10.10.10.14 (10.10.10.14): 56 data bytes

--- 10.10.10.14 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss



ESXI2 - 10.10.10.14 - Connected to oce1
[root@r210esxi:~] esxcli network ip neighbor list
Neighbor	Mac Address		Vmknic	 Expiry  State  Type
----------  -----------------  ------  ---------  -----  -------
10.10.10.2  2c:44:fd:e1:33:70  vmk1	 1192 sec		 Unknown
10.10.10.4  (incomplete)	   vmk1	   -1 sec		 Unknown

[root@r210esxi:~] ping 10.10.10.4
PING 10.10.10.4 (10.10.10.4): 56 data bytes
sendto() failed (Host is down)
[root@r210esxi:~] ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2): 56 data bytes

--- 10.10.10.2 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
 
Last edited by a moderator:

SanEerf

Cadet
Joined
Oct 14, 2017
Messages
2
Were you able to figure this out?


I gave up and bought a Nexus 3k that runs around 80 watts and am retiring the Nexus 5k's.

I had a reddit thread going that had some redditors posting configs along the same trains of thought as the threads I linked in my OP, none of them worked. I'm guessing it was NIC related because there seem to be a few proven cases of this working for other freenas users.
 
Status
Not open for further replies.
Top