I'm at my wits end with a bridge config and need a sounding board.
To begin, yes I realize this isn't a supported, recommended, whatever config for FreeNAS.
My topology is 2 ESXi hosts with 2x10gb NICs and the FreeNAS server with a 2 port 10gb NIC.
I had this all connected to 2 nexus 5010's to dork around with VPC and other nexus stuff but the power usage is too high for a 24x7 runtime.
I've done bridging before in Linux for other projects and it just worked. I browsed forum threads here and elsewhere but didn't find any newer posts, a lot of older posts with configs that I tried.
Couple of threads I followed:
https://forums.freenas.org/index.ph...1-bridging-ethernet-change.59876/#post-424525 - I'm using his identical commands, with my interfaces swapped in, at post-init to build my bridge
https://forums.freenas.org/index.ph...-without-10gb-switch.25259/page-3#post-161894
https://forums.freenas.org/index.php?threads/bridging-10gbe-with-chelsio-t420-cr.44605/
https://forums.freenas.org/index.ph...-drops-when-client-reboots.53052/#post-366851
I tried init scripts, tunables, just making the bridge from the cli the standard bsd/nix way but I cannot get traffic to pass across the bridge cleanly. I also tried creating a jail to let FreeNAS naturally create the bridge and then add my interfaces to that bridge. I'm inclined to believe its an ARP issue due to what I am seeing in the arp table.
What I see is that neither ESXi host can ping across the bridge to the other ESXi host. Only the ESXi 10.10.10.4 host connected to oce0 can ping the FreeNAS IP config'd on oce0 via the web interface. The ESXi host at 10.10.10.14 connected to oce1 cannot ping FreeNAS or the other ESXi host across the bridge. All hosts are seen in the arp table but the ESXi host on oce1 is seen twice, once "incomplete", I'm guessing due to the bridge. I tried disabling arp and doing static arp entries but still no luck. On both ESXi I tried enabling and disabling Promiscuous as well. It's gotta be an issue with arp and bridging. See below for config.
To begin, yes I realize this isn't a supported, recommended, whatever config for FreeNAS.
My topology is 2 ESXi hosts with 2x10gb NICs and the FreeNAS server with a 2 port 10gb NIC.
I had this all connected to 2 nexus 5010's to dork around with VPC and other nexus stuff but the power usage is too high for a 24x7 runtime.
I've done bridging before in Linux for other projects and it just worked. I browsed forum threads here and elsewhere but didn't find any newer posts, a lot of older posts with configs that I tried.
Couple of threads I followed:
https://forums.freenas.org/index.ph...1-bridging-ethernet-change.59876/#post-424525 - I'm using his identical commands, with my interfaces swapped in, at post-init to build my bridge
https://forums.freenas.org/index.ph...-without-10gb-switch.25259/page-3#post-161894
https://forums.freenas.org/index.php?threads/bridging-10gbe-with-chelsio-t420-cr.44605/
https://forums.freenas.org/index.ph...-drops-when-client-reboots.53052/#post-366851
I tried init scripts, tunables, just making the bridge from the cli the standard bsd/nix way but I cannot get traffic to pass across the bridge cleanly. I also tried creating a jail to let FreeNAS naturally create the bridge and then add my interfaces to that bridge. I'm inclined to believe its an ARP issue due to what I am seeing in the arp table.
What I see is that neither ESXi host can ping across the bridge to the other ESXi host. Only the ESXi 10.10.10.4 host connected to oce0 can ping the FreeNAS IP config'd on oce0 via the web interface. The ESXi host at 10.10.10.14 connected to oce1 cannot ping FreeNAS or the other ESXi host across the bridge. All hosts are seen in the arp table but the ESXi host on oce1 is seen twice, once "incomplete", I'm guessing due to the bridge. I tried disabling arp and doing static arp entries but still no luck. On both ESXi I tried enabling and disabling Promiscuous as well. It's gotta be an issue with arp and bridging. See below for config.
Code:
root@freenas:~ # arp -a ? (10.10.10.14) at 00:50:56:61:54:ee on oce1 expires in 1200 seconds [ethernet] ? (10.10.10.14) at (incomplete) on oce0 expired [ethernet] ? (10.10.10.2) at 2c:44:fd:e1:33:70 on oce0 permanent [ethernet] ? (10.10.10.4) at 00:50:56:66:55:1c on oce0 expires in 1113 seconds [ethernet] root@freenas:~ # ping 10.10.10.14 PING 10.10.10.14 (10.10.10.14): 56 data bytes ping: sendto: Host is down ping: sendto: Host is down ping: sendto: Host is down ^C --- 10.10.10.14 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss root@freenas:~ # ping 10.10.10.4 PING 10.10.10.4 (10.10.10.4): 56 data bytes 64 bytes from 10.10.10.4: icmp_seq=0 ttl=64 time=0.303 ms 64 bytes from 10.10.10.4: icmp_seq=1 ttl=64 time=0.375 ms 64 bytes from 10.10.10.4: icmp_seq=2 ttl=64 time=0.221 ms ^C --- 10.10.10.4 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.221/0.300/0.375/0.063 ms root@freenas:~ # ifconfig ~Removed Interfaces not related to config~ oce0: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500 options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO> ether 2c:44:fd:e1:33:70 hwaddr 2c:44:fd:e1:33:70 inet 10.10.10.2 netmask 0xffffff00 broadcast 10.10.10.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active oce1: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500 options=503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO> ether 2c:44:fd:e1:33:74 hwaddr 2c:44:fd:e1:33:74 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active bridge100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:6e:6b:57:d2:64 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: oce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 2000 member: oce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 3 priority 128 path cost 2000 ESXI1 - 10.10.10.4 - Atached to oce0 [root@esxi2:~] esxcli network ip neighbor list Neighbor Mac Address Vmknic Expiry State Type ----------- ----------------- ------ --------- ----- ------- 10.10.10.14 00:50:56:61:54:ee vmk2 1198 sec Unknown 10.10.10.2 2c:44:fd:e1:33:70 vmk2 1196 sec Unknown [root@esxi2:~] ping 10.10.10.4 PING 10.10.10.4 (10.10.10.4): 56 data bytes 64 bytes from 10.10.10.4: icmp_seq=0 ttl=64 time=0.057 ms 64 bytes from 10.10.10.4: icmp_seq=1 ttl=64 time=0.073 ms 64 bytes from 10.10.10.4: icmp_seq=2 ttl=64 time=0.068 ms --- 10.10.10.4 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.057/0.066/0.073 ms [root@esxi2:~] ping 10.10.10.2 PING 10.10.10.2 (10.10.10.2): 56 data bytes 64 bytes from 10.10.10.2: icmp_seq=0 ttl=64 time=0.567 ms 64 bytes from 10.10.10.2: icmp_seq=1 ttl=64 time=0.448 ms --- 10.10.10.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.448/0.507/0.567 ms [root@esxi2:~] ping 10.10.10.14 PING 10.10.10.14 (10.10.10.14): 56 data bytes --- 10.10.10.14 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss ESXI2 - 10.10.10.14 - Connected to oce1 [root@r210esxi:~] esxcli network ip neighbor list Neighbor Mac Address Vmknic Expiry State Type ---------- ----------------- ------ --------- ----- ------- 10.10.10.2 2c:44:fd:e1:33:70 vmk1 1192 sec Unknown 10.10.10.4 (incomplete) vmk1 -1 sec Unknown [root@r210esxi:~] ping 10.10.10.4 PING 10.10.10.4 (10.10.10.4): 56 data bytes sendto() failed (Host is down) [root@r210esxi:~] ping 10.10.10.2 PING 10.10.10.2 (10.10.10.2): 56 data bytes --- 10.10.10.2 ping statistics --- 3 packets transmitted, 0 packets received, 100% packet loss
Last edited by a moderator:
