bcunningham86
Cadet
- Joined
- May 2, 2020
- Messages
- 2
Here we go, first post on the FreeNAS forums!
I currently run a full Ubiquiti network stack and am running into a some issues trying to get an external VPN router VLAN connected to FreeNAS for one of the Network Interface cards and jails specifically. I setup my old Edgerouter as an external VPN router using this guide to get the router working with the UDMP, and this guide to get OpenVPN with Private Internet Access to work with the Edgerouter. This works beautifully on my Windows machine.
While attempting to setup a VLAN in FreeNAS, I ran into an issue (which I believe to be a NAT loopback issue) that took down my whole network and caused the UDMP to go unresponsive and required me to re-setup and restore it. I have read posts where people claim this is possible to accomplish, but I'm not sure what static routes or firewall rules I need to setup and whether or not they need to be in the UDMP, Edgerouter, or FreeNAS. My guess is that it's a combination of all of those, but mostly in FreeNAS, since other devices seem to work well consistently. I have had to hash together several different guides, but I will list out my configuration and resources and do my best to explain the steps to recreate the situation clearly. And yes, I've posted on the Ubiquiti forums as well.
Here is a list of my current configuration of hardware and software:
SuperMicro X10SDV-TLN4F, Intel(R) Xeon(R) CPU D-1541, 128GB ECC RAM
4x NICs:
igb0 - 192.168.1.8
igb1 - no IP (Temporarily set to VPN Internal LAN IP range 192.168.2.x, then it broke)
ix0 - no IP or VLAN yet
ix1 - no IP or VLAN yet
FreeNAS 11.3-U3.2 - default interface igb0 at 192.168.1.8
Unifi Dream Machine Pro:
Gateway at 192.168.1.1
LAN 192.168.1.0/24
VLAN162 172.16.2.0/28 (VPN Egress VLAN to Router WAN)
VLAN125 VLAN Only (VPN Internal VLAN to Router LAN and VPN clients)
Unifi Switch 48 (non-PoE) with 10Gb DAC to UDMP:
4 Important Configured ports:
1 port for VLAN162 VPN Egress plugged into WAN of external router
3 ports for VLAN125 VPN Internal VLAN:
One that plugs into LAN1 of external router
Two that are used for client devices (My Windows machine and FreeNAS NIC igb1)
Unifi EdgeRouter-PoE-5 - Gateway at 192.168.2.1, LAN network 192.168.2.0/24, Internet coming in from 172.16.2.x/28, receiving a 172.16.2.6 address from the UDMP via DHCP.
I also have other things like a Unifi UAP AC PRO and an IoT network, but the FreeNAS is on the wired LAN network for now, so that's irrelevant here.
The steps to getting the VPN part working can be broken down like this:
1. Create a small corporate network in the UDMP with a VLAN for Egress traffic from the WAN of the external router allowing access to the internet. I chose 172.16.2.0/28 for that range and VLAN 162, set the DNS to my VPN service provider's DNS, and enabled IGMP snooping.
2. Reset the External EdgeRouter to factory default and plug your computer directly into eth0 to setup router. DO NOT check the box that says "Internet connection is on VLAN", otherwise the router will not obtain an IP address or internet from the UDMP for some reason. I set DHCP for the ISP connection on the external router, chose 192.168.2.1 as the gateway, bridged all LANS together, finished the wizard and rebooted.
3. Set a port on the Unifi Switch 48 to use the VPN Egress VLAN162 and plug that into the WAN of the external router. Set a second port on the USW-48 to use the VPN Internal VLAN125 and plug that into LAN1 of the external router. Create a third port on the USW-48 again for the VPN Internal VLAN125, but this time plug it into your computer LAN. You should be able to connect to the internet with a 192.168.2.x IP address.
4. Install OpenVPN and setup PIA on EdgeRouter, and reboot. I also manually narrowed the external routers DHCP range to 10 IPs, since that is the maximum number of concurrent connections that my VPN provider allows.
5. Set any other switch port on the USW-48 or UDMP to use the VPN Internal VLAN125 and it will route that device through the external router/VPN. I plugged in my Windows machine, did ipconfig /release and /renew, checked IP and I am successfully routing through PIA. Tested both on the UDMP and USW-48.
As a side note:
After a few hours of troubleshooting, I learned that whenever the primary gateway is rebooted it breaks the VPN and internet connection. You don't need to reboot the external router and that won't actually work either. By following the openVPN Edgerouter guide, an entry for Private Internet Access vtun0 was created on the EdgeRouter dashboard. If you select Actions > Disable, then Actions > Enable, it fixes the VPN connection. Also, Disabling it will allow regular internet access without the VPN again.
So far so good!
Configuring FreeNAS to use the VPN for only one NIC or jails is where I've run into major problems. The main thing I'm having trouble doing is setting up a VLAN inside FreeNAS that uses the VPN Internal VLAN125 for one of my NIC's to send directly to a jail. I have been fighting with FreeNAS VLANS since 11.3 and I haven't been able to get it working properly. I followed this guide from lawrence systems and looked over the original article as well, but also saw this post that indicates it no longer works the same for FreeNAS 11.3U2 and above (which I needed to upgrade from since it broke my ix0 and ix1 NICs), so I ended up pretty much figuring it out through trial and error. I did succeed in getting a jail to use the VPN for a short time, but it quickly broke everything and killed my gateway completely.
Here is what I did:
I created and attached Bridge125 < VLAN125 (VPN Internal VLAN) < Network Interface Card (igb1 - No IP). I set the desired VPN client jail to use the new vnet0:bridge125. With some tinkering of things like vnet_default_interface, resolver, and manually specifying IPs/using DHCP settings I was finally able to get the jail to boot with DHCP and the external router successfully leased an IP to the jail in the 192.168.2.x range as specified in my external router. Although, I wasn't really able to take note of what was working about the jail before the issues started.
To test things, I consoled into the jail attached to the VPN and did a curl, no response. Then I thought, well since I can't access the external router gateway without being plugged into that network (another issue) then maybe I could see the jail from the VPN network.
This is where things start to go rogue. I switched my Windows machine LAN cable to the VPN network, but couldn't SSH into the FreeNAS 192.168.1.8 address because of VLAN segregation, so I switched the LAN back to the UDMP network and added an IP of 192.168.2.8 to the VLAN125 on FreeNAS, so that I may be able to access the GUI on a 192.168.2.x address from my external router. After switching again to the VPN network, I could see it successfully showed up in the external router as a leased IP address (Now there were 3 leased IPs, my Windows machine 192.168.2.6, the FreeNAS jail 192.168.2.7 and the FreeNAS VLAN 192.168.2.8) and I could access the GUI from the VPN network.
I was then able to SSH into FreeNAS from the VPN network on 192.168.2.8. So far so great! I SSH into the FreeNAS and enter the jail with iocage console. I run a curl ifconfig.me command and to my delight I get an IP from the VPN service! Awesome! It works! Not quite...
I rebooted the FreeNAS and after a minute or two, the whole network went down! I could not get the gateway to reboot until unplugging FreeNAS from the network entirely. I rebooted FreeNAS again, the UDMP, the switch, and external router, then reconnected the FreeNAS and it crashed again immediately. I was about to remove the IP from the VLAN125 on FreeNAS, but before I could do anymore troubleshooting the UDMP had crashed beyond recovery and I had to do the setup process, adoptions, and upload my config backup and re-setup to the FreeNAS VLAN point. So here I am a little hesitant to try it again without some further guidance.
My guess is that attaching the IP to the VLAN on the NIC directly created some type of NAT loopback in FreeNAS that allowed VPN traffic to go back into the UDM through other NICs and caused it to go haywire. I'm not sure where to go from here. I would like to be able to isolate one NIC or just a few jails on the VPN network with the rest of the traffic going through the UDMP on the regular network and I'd like to be able to access them from both local networks and be able to work on the jails from either. I don't understand static routes or firewalls very well, which I'm almost positive is the missing element. If anyone has the answer to what static or firewall rules I need to add and where (UDMP, EdgeRouter, FreeNAS, etc.) Or can at least get me pointed in the right direction I would greatly appreciate it. If you have any other ideas or suggestions they are greatly appreciated and if anymore information is needed, please let me know.
Summary of issues:
1. I need to know how to attach VPN VLAN to NIC/jails properly without allowing traffic to loopback to gateway.
2. I need to know how to set static routes and firewall rules (which is also probably the answer to issue #1) to allow access to all of the jails on either local network. I would like to be able to access both router control panels (192.168.1.1 and 192.168.2.1) as well. I wonder if this one has to do with the UDMP VPN Egress 172.16.2.0/28 range and the external EdgeRouter 192.168.2.0/24 range. I was a little confused by the instructions of the setup process and was trying to overlap them, but couldn't get that to work. I.e. setting the UDMP to issue 172.16.2.6-172.16.2.14 and setting the external router gateway to 172.16.2.1 and DHCP range to 172.16.2.51-60/28. I'm not sure if your supposed to do that. I interpreted it as something about allowing the external router to have the same gateway so that the networks can see each other, but I was unsuccessful and maybe this is not what I want. I know in general you don't want any overlap of gateway and broadcast addresses, but it seemed like this was a suggested workaround exception. If anyone can clarify that for me that would be great also.
Thanks again for any help!
I currently run a full Ubiquiti network stack and am running into a some issues trying to get an external VPN router VLAN connected to FreeNAS for one of the Network Interface cards and jails specifically. I setup my old Edgerouter as an external VPN router using this guide to get the router working with the UDMP, and this guide to get OpenVPN with Private Internet Access to work with the Edgerouter. This works beautifully on my Windows machine.
While attempting to setup a VLAN in FreeNAS, I ran into an issue (which I believe to be a NAT loopback issue) that took down my whole network and caused the UDMP to go unresponsive and required me to re-setup and restore it. I have read posts where people claim this is possible to accomplish, but I'm not sure what static routes or firewall rules I need to setup and whether or not they need to be in the UDMP, Edgerouter, or FreeNAS. My guess is that it's a combination of all of those, but mostly in FreeNAS, since other devices seem to work well consistently. I have had to hash together several different guides, but I will list out my configuration and resources and do my best to explain the steps to recreate the situation clearly. And yes, I've posted on the Ubiquiti forums as well.
Here is a list of my current configuration of hardware and software:
SuperMicro X10SDV-TLN4F, Intel(R) Xeon(R) CPU D-1541, 128GB ECC RAM
4x NICs:
igb0 - 192.168.1.8
igb1 - no IP (Temporarily set to VPN Internal LAN IP range 192.168.2.x, then it broke)
ix0 - no IP or VLAN yet
ix1 - no IP or VLAN yet
FreeNAS 11.3-U3.2 - default interface igb0 at 192.168.1.8
Unifi Dream Machine Pro:
Gateway at 192.168.1.1
LAN 192.168.1.0/24
VLAN162 172.16.2.0/28 (VPN Egress VLAN to Router WAN)
VLAN125 VLAN Only (VPN Internal VLAN to Router LAN and VPN clients)
Unifi Switch 48 (non-PoE) with 10Gb DAC to UDMP:
4 Important Configured ports:
1 port for VLAN162 VPN Egress plugged into WAN of external router
3 ports for VLAN125 VPN Internal VLAN:
One that plugs into LAN1 of external router
Two that are used for client devices (My Windows machine and FreeNAS NIC igb1)
Unifi EdgeRouter-PoE-5 - Gateway at 192.168.2.1, LAN network 192.168.2.0/24, Internet coming in from 172.16.2.x/28, receiving a 172.16.2.6 address from the UDMP via DHCP.
I also have other things like a Unifi UAP AC PRO and an IoT network, but the FreeNAS is on the wired LAN network for now, so that's irrelevant here.
The steps to getting the VPN part working can be broken down like this:
1. Create a small corporate network in the UDMP with a VLAN for Egress traffic from the WAN of the external router allowing access to the internet. I chose 172.16.2.0/28 for that range and VLAN 162, set the DNS to my VPN service provider's DNS, and enabled IGMP snooping.
2. Reset the External EdgeRouter to factory default and plug your computer directly into eth0 to setup router. DO NOT check the box that says "Internet connection is on VLAN", otherwise the router will not obtain an IP address or internet from the UDMP for some reason. I set DHCP for the ISP connection on the external router, chose 192.168.2.1 as the gateway, bridged all LANS together, finished the wizard and rebooted.
3. Set a port on the Unifi Switch 48 to use the VPN Egress VLAN162 and plug that into the WAN of the external router. Set a second port on the USW-48 to use the VPN Internal VLAN125 and plug that into LAN1 of the external router. Create a third port on the USW-48 again for the VPN Internal VLAN125, but this time plug it into your computer LAN. You should be able to connect to the internet with a 192.168.2.x IP address.
4. Install OpenVPN and setup PIA on EdgeRouter, and reboot. I also manually narrowed the external routers DHCP range to 10 IPs, since that is the maximum number of concurrent connections that my VPN provider allows.
5. Set any other switch port on the USW-48 or UDMP to use the VPN Internal VLAN125 and it will route that device through the external router/VPN. I plugged in my Windows machine, did ipconfig /release and /renew, checked IP and I am successfully routing through PIA. Tested both on the UDMP and USW-48.
As a side note:
After a few hours of troubleshooting, I learned that whenever the primary gateway is rebooted it breaks the VPN and internet connection. You don't need to reboot the external router and that won't actually work either. By following the openVPN Edgerouter guide, an entry for Private Internet Access vtun0 was created on the EdgeRouter dashboard. If you select Actions > Disable, then Actions > Enable, it fixes the VPN connection. Also, Disabling it will allow regular internet access without the VPN again.
So far so good!
Configuring FreeNAS to use the VPN for only one NIC or jails is where I've run into major problems. The main thing I'm having trouble doing is setting up a VLAN inside FreeNAS that uses the VPN Internal VLAN125 for one of my NIC's to send directly to a jail. I have been fighting with FreeNAS VLANS since 11.3 and I haven't been able to get it working properly. I followed this guide from lawrence systems and looked over the original article as well, but also saw this post that indicates it no longer works the same for FreeNAS 11.3U2 and above (which I needed to upgrade from since it broke my ix0 and ix1 NICs), so I ended up pretty much figuring it out through trial and error. I did succeed in getting a jail to use the VPN for a short time, but it quickly broke everything and killed my gateway completely.
Here is what I did:
I created and attached Bridge125 < VLAN125 (VPN Internal VLAN) < Network Interface Card (igb1 - No IP). I set the desired VPN client jail to use the new vnet0:bridge125. With some tinkering of things like vnet_default_interface, resolver, and manually specifying IPs/using DHCP settings I was finally able to get the jail to boot with DHCP and the external router successfully leased an IP to the jail in the 192.168.2.x range as specified in my external router. Although, I wasn't really able to take note of what was working about the jail before the issues started.
To test things, I consoled into the jail attached to the VPN and did a curl, no response. Then I thought, well since I can't access the external router gateway without being plugged into that network (another issue) then maybe I could see the jail from the VPN network.
This is where things start to go rogue. I switched my Windows machine LAN cable to the VPN network, but couldn't SSH into the FreeNAS 192.168.1.8 address because of VLAN segregation, so I switched the LAN back to the UDMP network and added an IP of 192.168.2.8 to the VLAN125 on FreeNAS, so that I may be able to access the GUI on a 192.168.2.x address from my external router. After switching again to the VPN network, I could see it successfully showed up in the external router as a leased IP address (Now there were 3 leased IPs, my Windows machine 192.168.2.6, the FreeNAS jail 192.168.2.7 and the FreeNAS VLAN 192.168.2.8) and I could access the GUI from the VPN network.
I was then able to SSH into FreeNAS from the VPN network on 192.168.2.8. So far so great! I SSH into the FreeNAS and enter the jail with iocage console. I run a curl ifconfig.me command and to my delight I get an IP from the VPN service! Awesome! It works! Not quite...
I rebooted the FreeNAS and after a minute or two, the whole network went down! I could not get the gateway to reboot until unplugging FreeNAS from the network entirely. I rebooted FreeNAS again, the UDMP, the switch, and external router, then reconnected the FreeNAS and it crashed again immediately. I was about to remove the IP from the VLAN125 on FreeNAS, but before I could do anymore troubleshooting the UDMP had crashed beyond recovery and I had to do the setup process, adoptions, and upload my config backup and re-setup to the FreeNAS VLAN point. So here I am a little hesitant to try it again without some further guidance.
My guess is that attaching the IP to the VLAN on the NIC directly created some type of NAT loopback in FreeNAS that allowed VPN traffic to go back into the UDM through other NICs and caused it to go haywire. I'm not sure where to go from here. I would like to be able to isolate one NIC or just a few jails on the VPN network with the rest of the traffic going through the UDMP on the regular network and I'd like to be able to access them from both local networks and be able to work on the jails from either. I don't understand static routes or firewalls very well, which I'm almost positive is the missing element. If anyone has the answer to what static or firewall rules I need to add and where (UDMP, EdgeRouter, FreeNAS, etc.) Or can at least get me pointed in the right direction I would greatly appreciate it. If you have any other ideas or suggestions they are greatly appreciated and if anymore information is needed, please let me know.
Summary of issues:
1. I need to know how to attach VPN VLAN to NIC/jails properly without allowing traffic to loopback to gateway.
2. I need to know how to set static routes and firewall rules (which is also probably the answer to issue #1) to allow access to all of the jails on either local network. I would like to be able to access both router control panels (192.168.1.1 and 192.168.2.1) as well. I wonder if this one has to do with the UDMP VPN Egress 172.16.2.0/28 range and the external EdgeRouter 192.168.2.0/24 range. I was a little confused by the instructions of the setup process and was trying to overlap them, but couldn't get that to work. I.e. setting the UDMP to issue 172.16.2.6-172.16.2.14 and setting the external router gateway to 172.16.2.1 and DHCP range to 172.16.2.51-60/28. I'm not sure if your supposed to do that. I interpreted it as something about allowing the external router to have the same gateway so that the networks can see each other, but I was unsuccessful and maybe this is not what I want. I know in general you don't want any overlap of gateway and broadcast addresses, but it seemed like this was a suggested workaround exception. If anyone can clarify that for me that would be great also.
Thanks again for any help!
