SOLVED Jail networking issues

[zey]

Excuse my lack of knowledge in this situation. I know there's most likely a thread create everyday about jail networking issues.

I am having issues with getting my jails to communicate on my network. I assume it has to do with the way it bridges to the interface. The GUI seriously lacks a way of view the bridge that the jails use. Yes I am sure if I jump into the console and know what I'm doing I'd most likely figure out the issue.

I have two interfaces
em0 to the network 10.10.10.11/24
ix0 10gbe peer to peer with my desktop 10.10.11.3/24

The jails are not working regardless to whether I use static or DHCP. With DHCP it does not get an IP. Which leads me to think there's an issue with how the bridging it working. My knowledge on FreeBSD is limited.

Jail 1:

Code:

root@resilio_1:/ # ifconfig													 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384			   
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>				   
		inet6 ::1 prefixlen 128												 
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1							 
		inet 127.0.0.1 netmask 0xff000000									   
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>							   
		groups: lo															 
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500   
		options=8<VLAN_MTU>													 
		ether a2:85:ef:cb:ee:d7												 
		inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255			   
		nd6 options=9<PERFORMNUD,IFDISABLED>									
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)					 
		status: active														 
		groups: epair														   
root@resilio_1:/ # dhclient epair0b											 
dhclient already running, pid: 26698.										   
exiting.																		
root@resilio_1:/ #

Server:

Code:

root@dobby:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
		ether 00:0c:29:00:56:92
		hwaddr 00:0c:29:00:56:92
		inet 10.10.10.11 netmask 0xffffff00 broadcast 10.10.10.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
		options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:25:90:5d:ff:ce
		hwaddr 00:25:90:5d:ff:ce
		inet 10.10.11.3 netmask 0xffffff00 broadcast 10.10.11.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
		status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:25:90:5d:ff:cf
		hwaddr 00:25:90:5d:ff:cf
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:9b:ea:41:dd:00
		nd6 options=1<PERFORMNUD>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000
		member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 6 priority 128 path cost 2000
		member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=8<VLAN_MTU>
		ether 02:24:50:00:06:0a
		hwaddr 02:24:50:00:06:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=8<VLAN_MTU>
		ether 02:24:50:00:07:0a
		hwaddr 02:24:50:00:07:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
root@dobby:~ #

 

[joeschmuck]

So are you running FreeNAS in a VM on ESXi 6.5?

Do you have promiscuous mode enabled in ESXi?

If you have done this then you will need to describe you setup a bit more.

 

[zey]

joeschmuck,

Thanks for the reply. Yes it is ESXi 6.5 update 1 and promiscuous mode is set to reject.

Not sure what other information you would need. The jails are freshly created. I can try anything you would recommend.

I've tried setting promiscuous to accept and restart the VM. I am still experiencing the same issue.

I am still seeing that it's enabled.

Code:

Dec 26 05:34:22 dobby bridge0: Ethernet address: 02:9b:ea:41:dd:00
Dec 26 05:34:22 dobby kernel: bridge0: link state changed to UP
Dec 26 05:34:22 dobby kernel: bridge0: link state changed to UP
Dec 26 05:34:22 dobby kernel: em0: promiscuous mode enabled
Dec 26 05:34:22 dobby epair0a: Ethernet address: 02:24:50:00:06:0a
Dec 26 05:34:22 dobby epair0b: Ethernet address: 02:24:a0:00:07:0b
Dec 26 05:34:22 dobby kernel: epair0a: link state changed to UP
Dec 26 05:34:22 dobby kernel: epair0a: link state changed to UP
Dec 26 05:34:22 dobby kernel: epair0b: link state changed to UP
Dec 26 05:34:22 dobby kernel: epair0b: link state changed to UP
Dec 26 05:34:22 dobby kernel: epair0a: promiscuous mode enabled
Dec 26 05:34:25 dobby epair1a: Ethernet address: 02:24:50:00:07:0a
Dec 26 05:34:25 dobby epair1b: Ethernet address: 02:24:a0:00:08:0b
Dec 26 05:34:25 dobby kernel: epair1a: link state changed to UP
Dec 26 05:34:25 dobby kernel: epair1a: link state changed to UP
Dec 26 05:34:25 dobby kernel: epair1b: link state changed to UP
Dec 26 05:34:25 dobby kernel: epair1b: link state changed to UP
Dec 26 05:34:25 dobby kernel: epair1a: promiscuous mode enabled
Dec 26 05:34:25 dobby daemon[3203]:	 2017/12/26 05:34:25 [ERR] agent: failed to sync remote state: No cluster leader
Dec 26 05:34:26 dobby daemon[3203]:	 2017/12/26 05:34:26 [WARN] raft: Heartbeat timeout from "" reached, starting election
Dec 26 05:34:34 dobby daemon[3203]: ==> Newer Consul version available: 1.0.2 (currently running: 1.0.0)
Dec 26 05:36:11 dobby uwsgi: [freeadmin.navtree:562] Couldn't retrieve http://10.10.10.11/plugins/sabnzbd/2/_s/treemenu: HTTP Error 503: Service Unavailable
Dec 26 05:36:21 dobby uwsgi: [freeadmin.navtree:562] Couldn't retrieve http://10.10.10.11/plugins/resilio/1/_s/treemenu:

 

[joeschmuck]

promiscuous mode is set to reject.

This must be enabled, it will not work otherwise. Here is the link I used to set it up: https://kb.vmware.com/s/article/1004099

Do you have the FreeNAS network settings all good? You have a gateway and name server established?

A few days ago I rebuilt my FreeNAS VM as well and created it from new. The configuration was very easy, just setup the FreeNAS network (static IP, gateway, name server) and left my jails to DHCP and it worked without any issues. Under Jails -> Configuration -> Advanced I changed the DHCP start and stop to a specific range, for me it was 53 to 59 so I could use 192.168.1.53 as my first jail IP address. My ESXi was previously setup to promiscuous mode.

 

[zey]

I did exactly what the link explained. Slightly different due to the version difference. Reboot the host still no go. I even performed a factory reset. Still no go.

Fresh install of FreeNAS 11.1 everything new. STILL no go.

I ended up passing through the network port and sure enough the issue is no longer. My guess is it has something to do with the ESXi network driver. A setting I just couldn't figure out some where. I then set promiscuous to accept so I don't know.

 

[joeschmuck]

Your problem is not FreeNAS, it's ESXi and your setup of your vSwitch. Passing through the NIC does work but you shouldn't have to do that. And if you use vmx connections vice E1000 then you can transfer data between VMs very fast. But start with E1000 connections until it all works properly, it is the most compatable virtual NIC.

Here is a screen capture of my vSwitch, not sure if it will help at all. Note that I have a physical NIC on one side and then I have a "port group" called LAN. You can connect all your VMs to the LAN port group and they will be routed to the physical NIC. You need a vSwitch for each physical NIC that you plan to use. In most home situations I would think only one NIC is enough unless you are running a VM firewall and then you would take the second NIC and create a new vSwitch for it and new port group, for me I called it WAN, and then you connect your firewall VM to this new vSwitch and the physical NIC to your internet connection. This works great by the way.

 

[zey]

Here's mine. The only difference now from yesterday is that I had a second uplink. I am using E1000 as the NIC. I don't have any idea as to why it's not working for the jails. Also yes gateway is set and IP for FreeNAS and so on.

 

[joeschmuck]

So it looks like ESXi is configured properly.

Have you tried this? Set your FreeNAS to a static IP, ensure your gateway and DNS is setup.
In the jails configuration you can use DHCP, just configure what I have previously stated.

Can you provide me a screen shot of the FreeNAS GUI Network -> Network Summary ? And a screen shot of Network -> Interfaces?

Also on the Network -> Global Configuration screen, place a check mark in "Enable netwait feature" to hold off networking until after they system is up.

Thanks.

 

[zey]

I checked the checkbox "Enable netwait feature"

 

[joeschmuck]

What ip address are you using to access the GUI? Don't tell me it's 10.10.11.3 because that is likely going to be your issue. Remove your 10Gbe from the equation for now, just use a single Ethernet connection. Once you get this working then add in the second NIC.

 

[zey]

Yea no I'd have to be a bit of a retard to be using 10.10.11.3. So no to that question. As for removing my 10gbe interface. Yesterday when I factory reset I moved the 10gbe interface. Well actually it was never re-added. I was still having the same issue. I'll remove for now just to make this as simple as possible. It still leaves me with a odd issue.

The jails now get IPs but when I attempt to browse to the web interface it does not respond.

 

[joeschmuck]

Yea no I'd have to be a bit of a retard to be using 10.10.11.3.

You would be surprised at what I see. I try to make no assumptions.

The jails now get IPs but when I attempt to browse to the web interface it does not respond.

Can you be a little more specific? I assume you mean that you cannot get to the FreeNAS GUI, and what IPs are working/not working?

If you got this far then I suspect you will figure out the rest. It can take time to learn how this all works, even when it's frustrating.

 

[zey]

You would be surprised at what I see. I try to make no assumptions.

This! Is true..

Can you be a little more specific? I assume you mean that you cannot get to the FreeNAS GUI, and what IPs are working/not working?

If you got this far then I suspect you will figure out the rest. It can take time to learn how this all works, even when it's frustrating.

I am a damn idiot. From troubleshooting so much I actually forgot to turn on the plugin from the Installed Plugins tab.


However to be clear. I do not know what was the issue nor how it was resolved. I basically started fresh. I believe the issue may have been linked to the fact. That I reused a FreeNAS VM I was using a while back. There may have been something with in the VM causing the issue.

 

[joeschmuck]

Hey, if you got it working then take the win and you can stop stressing about it. Have no fear, something else will happen, could be a flat tire or someone throws eggs at house, or you win the lottery and have everyone hounding you for money. Actaully I'm hoping to have the stress of winning the lottery myself but first I've got to win. I keep telling my wife that any million dollars is better than none at all. Of coruse we do plan how we will spend it all if we hit 50 million or more. Anything under 5 million before taxes means we don't share, we just retire and have a good life.

 

[zey]

lol, nice! I'll take it.