This post is a follow-up solution for solving how to setup VLANS with 11.2U7 (https://forums.lawrencesystems.com/t/freenas-jails-with-vlan/3046/11). Both 11.2 and 11.3 are VLAN able, however the documentation for getting a working setup is lacking.
Pre-requisites:
1. This guide assumes your hardware is all VLAN compatible -- meaning your router and switches are 802.1q compatible. This usually means that switches most be managed rather than unmanaged.
2. This guide also assumes that you have created and setup the appropriate VLANs with your hardware. My reference system is a virtualized pfsense router running inside of xcp-ng (Type I open-source Citrix hypervisor) using a variety of managed switches -- unifi and D-Link. For each one of these components, I needed to login to either the device or management software and create the VLANs with appropriate tags. This step of creating the VLANs within FreeNAS will be demonstrated below.
-- Implicitly this guide assumes you have setup a DHCP server and gateway for each of your VLANs within your router. Firewall rules can be created to limit intra-VLAN communication, however initially its probably best to allow each VLAN to have access to all ports and IP addresses of the other LANS. For example a pfsense firewall rule on each VLAN that states all traffic originating on the VLAN is able to pass to all networks and all ports could be used.
3. VLAN hardware usually involves passing traffic for one network that is untagged. The the remainder of the networks that pass through the device, they must be tagged to include the appropriate VLAN ID. By default, usually most switches consider VLAN 1 to be the native or untagged network. In contrast the traditional setup of untagged/tagged traffic, many within the FreeNAS community recommend to tag all network traffic presented to FreeNAS. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/. This involves creation of a trunk port on the switch to which FreeNAS is attached that tags all traffic. Specifically VLAN1 traffic needs to be tagged egressing the switch port attached to the router.
a. Unfortunately this guide can not be so comprehensive to provide details instruction how to tag VLAN1 traffic. The individual user may need to consult their switch documentation how to tag VLAN1 on the port attached to the FreeNAS machine.
b. Given my reference switch is a unifi device, I'll provide an example of how one would tag all traffic using unifi. This would involved two steps:
i. Create a port profile that tags all networks -- (Settings->Profiles->Switch Ports). Click [Add Network Profile]. Under Tagged Networks-->Select All. (Caveat -- its possible that you may not want to include every single VLAN if they are not involved with FreeNAS, however Selecting the "LAN Network" ensures that VLAN 1 traffic is tagged. Choose any other networks that you want passed to FreeNAS here).
ii. Assign this port profile specifically to the port connected to FreeNAS (In the example below Port 2 is the port attached to FreeNAS)
4. This guide assumes traffic to the main FreeNAS installation comes in over one cable. (For my system, my freenas device has one cable connected to the IPMI for motherboard control, and the other is connected to FreeNAS with the FreeNAS port labeled igb0).
5. Ensure you can identify the name of the interface through which the cable is connected. Usually this can down done at the FreeNAS command line interface (CLI) and look for the network adapter that is shown to be active. Picture below shows two network adapter labeled igb0 and igb1. igb0 is the active network adapter whereas igb1 is labeled as "no carrier" since no networking cable is plugged into the port.
6. Make sure you have an alternative way to access your FreeNAS installation if network connectivity is disrupted (as it will most likely happen in setting up the VLANs). For me, my alternative access method was accessing the FreeNAS installation through the IPMI where I could open a console window into freenas. Once inside the console, I could manually bring up the network with the following command (substitute igb0 for your appropriate interface and 10.0.1.197 for the IP address currently associated with your freenas installation):
Once igb0 was manually up, I was able to either to continue accessing FreeNAS through the IPMI, or through an ssh shell, however it is imperative that you are able to access FreeNAS through the WebGUI, as most of the configuration steps need to be done through the GUI. This process described above could likely be avoided if there was a control terminal connected to your FreeNAS installation. Unfortunately I run my FreeNAS installation headless so I had to complete the steps described above.
Please be sure you are very comfortable with this step as it is a requirement to complete the VLAN setup process.
VLAN configuration with FreeNAS
Overall Concepts
The overall concepts of working with VLANs inside of FreeNAS are:
a. Define VLANs inside of the FreeNAS GUI
b. Create a network bridge (virtual switch) for each VLAN and associate each VLAN to its corresponding network bridge.
c. Attach each jails VNET interface or each Virtual Machine's (VM's) tap interface to the appropriate bridge depending on what VLAN you would like the jail/VM to be a member.
d. Ensure the parent physical interface (ie igb0) DOES NOT have an associated IP address.
Traffic from the switch will enter FreeNAS and the traffic will be segregated by VLAN ID so that each VLAN will be passed to the appropriate bridge. Traffic ingressing to the bridge will be strip of it's VLAN ID and then passed untagged to the network interfaces of the member jails/VMs. Conversely, traffic ingressing to the bridge will enter untagged from the VM/jail. Inside the bridge, the traffic will be tagged with the VLAN ID associated with the bridge and then passed back to the network switch tagged.
I suggest associating VLAN1 with the default bridge bridge0, and then correspondingly associate VLAN10 with bridge10, VLAN20 with bridge20, etc. Please note that I am not associating VLAN1 with bridge1. In prior iterations of FreeNAS where the iohyve hypervisor was used -- FreeNAS 9.10 -- (hypervisor predating the current bhyve hypervisor), bridge0 was specifically hardcoded into the program and could not be altered). I've also found that bridge0 is usually created by some process if not specifically created during our setup. Due to numerous problems I've ran into, I found it best to associate usually the native VLAN1 traffic with the native bridge -- bridge0.)
0. Take snapshot of current FreeNAS installation prior to proceeding in case something goes terribly wrong (which it wont!!)
a. System->General->Scroll to bottom and Click Save Config (I'd recommend Exporting Password Secret Seed and Pool Encryption Keys
b. Save Config file to friendly place.
1. Define VLANs inside FreeNAS. For each VLAN you need to add or define:
a. Within Network->Interfaces Click ADD
b. Type->VLAN, Name->VLAN#, Parent Interface->igb0 (or your appropriate network adapter), VLAN Tag->#, Options->up
Once completed with the process, your Interfaces should be similar to the following:
2. Define the bridges
a. Similar to adding the VLANs add a bridge for each VLAN EXCEPT FOR BRIDGE0 ---This will be created with a system tunable.
b. Create Non-bridge0 bridges:--- Network-->Interfaces--> Click ADD
i. Type->Bridge, Name->Bridge#, Bridge Members->Corresponding VLAN#, Options->up
c. Create bridge0 via use of System Tunables
i. System->Tunables--> Click ADD
ii. Variable->cloned_interfaces, Value->bridge0, Type->rc.conf, Enabled->Yes
iii. System->Tunables-->Click ADD
iv. Variable->ifconfig_bridge0, Value->addm vlan1 up, Type->rc.conf, Enabled->Yes
3. Associate VLAN1 with FreeNAS IP address
This is probably the most tricky part of the entire process since network connectivity will be lost and need to re-established manually at the command line or other means. Please read following instructions a few times prior to proceeding:
a. Delete main network interface adapter -- (in my example this would be igb0). Once deleting the main network adapter your connectivity to FreeNAS will be lost
b. Re-establish connectivity to FreeNAS through alternative method as discussed above and reconnect to GUI.
c. Within GUI, Edit the Network->Interfaces->VLAN1 Entry. Within the IP address field, type the IP address of the freenas installation (in my case it was 10.0.1.197/24)
4. Disable any jails or VMs that start at boot then reboot system.
5. Re-evaluate system after boot.
Hopefully system will reboot appropriately and you'll be able to access your installation over the network. If for any reason you find you are unable to connect to your FreeNAS installation, try re-establishing contact through using your alternative method. Please bea ware I lost contact with my system multiple times trying to figure out the process and had to access the IPMI FreeNAS multiple times to access the system.
Once things have rebooted, please examine the network settings via the CLI using ifconfig. I'll post my ifconfig to discuss the salient points:
Points to notice:
- igb0 apapter does not have an associated IP address. If you have an IP address associated with the main network adapter, you will need to correct your interfaces
- vlan1adapter has associated inet address and netmask. The IP address should be the same IP address you use to access the FreeNAS GUI.
- Make sure for all the defined vlan, the parent adapter is the main network interface (in my example its igb0)
- Examine each bridge. Make sure each appropriate bridge contains the appropriate vlan member. For example above, vlan1 is a member of bridge0, vlan90 is a member of bridge90, etc.
Add jails/VM's to appropriate network bridge.
a. Jail Setup -- Prior to proceeding know there is a lot of information on the internet discussing on how to add a jail to a bridge. Jails have the option of either using a vnet driver which gives each jail its own virtual networking stack, or not using the vnet driver. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver (since it's usually not explicitly clear on discussed by the author). This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/.
I'm proceeding assuming the VNET interface will be used as this gives better security and network separation of the jails from the parent FreeNAS installation
For each jail I made the following entries
Notes:
***resolver-(Please note that in 11.2U7 I could use the auto option, with 11.3 I needed to explicitly name the domain name servers for the VLAN on this line. Please alter for your specific VLAN, ie for VLAN 40 I would use nameserver 10.0.40.1 instead of 10.0.1.1 -- VLAN90 = nameserver 10.0.90.1, etc).
For jail troubleshooting, I recommend you create a shell into the GUI from the FreeNAS interface, verify the IP address, netmask are correct. Verify the routing table is correct for your VLAN. Two examples of this information are shown below (#1 for VLAN1, #2 for VLAN40)
VLAN1
VLAN40
b: VM setup
VM's are easier than jails to associate with each VM.
For each VM:
1. Virtual Machine->Devices->NIC (EDIT)
2. Under Nic to Attach -> Choose the appropriate Network bridge for the desired VLAN
Debugging Jails/VMs
Once again in the main FreeNAS installation, examine ifconfig. Here is a more complete example
Salient Points
- Examine each vnet interface and its association
- Make sure each listed vnet interface is a member of a bridge (ie vnet013 is a member of bridge0)
- For each jail and tap interface, make sure each tap interface of the appropriate bridge (ie tap0 is a member of bridge0).
- If you have verified the information within "ifconfig" looks correct and there is still no internet accessibility for the jail/VM, ensure the route and gateway is set appropriately (Within the jail - netstat -rn) and the nameservers are defined (Within the jail - /etc/resolv.conf) . You ideally would like to change configurations within the web gui menus rather than within the jail or VM itself since parameters might be modified on next boot of the jail or VM).
Future Directions
1. I'm unclear if all network traffic needs to be tagged when using VLANs with FreeNAS. I've yet to deconstruct my network setup to take on this challenge. If someone finds a setup where an untagged network could be used in conjunction with tagged networks, I'd be happy to modify the guide
2. Please be sure to examine and re-examine your ifconfig if you have problems. Usually there is a problem with a typo in the configuration of some options like resolver are not completed appropriately. When manually entering values be very mindful of syntax errors.
Good Luck and happy VLANing.
Pre-requisites:
1. This guide assumes your hardware is all VLAN compatible -- meaning your router and switches are 802.1q compatible. This usually means that switches most be managed rather than unmanaged.
2. This guide also assumes that you have created and setup the appropriate VLANs with your hardware. My reference system is a virtualized pfsense router running inside of xcp-ng (Type I open-source Citrix hypervisor) using a variety of managed switches -- unifi and D-Link. For each one of these components, I needed to login to either the device or management software and create the VLANs with appropriate tags. This step of creating the VLANs within FreeNAS will be demonstrated below.
-- Implicitly this guide assumes you have setup a DHCP server and gateway for each of your VLANs within your router. Firewall rules can be created to limit intra-VLAN communication, however initially its probably best to allow each VLAN to have access to all ports and IP addresses of the other LANS. For example a pfsense firewall rule on each VLAN that states all traffic originating on the VLAN is able to pass to all networks and all ports could be used.
3. VLAN hardware usually involves passing traffic for one network that is untagged. The the remainder of the networks that pass through the device, they must be tagged to include the appropriate VLAN ID. By default, usually most switches consider VLAN 1 to be the native or untagged network. In contrast the traditional setup of untagged/tagged traffic, many within the FreeNAS community recommend to tag all network traffic presented to FreeNAS. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/. This involves creation of a trunk port on the switch to which FreeNAS is attached that tags all traffic. Specifically VLAN1 traffic needs to be tagged egressing the switch port attached to the router.
a. Unfortunately this guide can not be so comprehensive to provide details instruction how to tag VLAN1 traffic. The individual user may need to consult their switch documentation how to tag VLAN1 on the port attached to the FreeNAS machine.
b. Given my reference switch is a unifi device, I'll provide an example of how one would tag all traffic using unifi. This would involved two steps:
i. Create a port profile that tags all networks -- (Settings->Profiles->Switch Ports). Click [Add Network Profile]. Under Tagged Networks-->Select All. (Caveat -- its possible that you may not want to include every single VLAN if they are not involved with FreeNAS, however Selecting the "LAN Network" ensures that VLAN 1 traffic is tagged. Choose any other networks that you want passed to FreeNAS here).
ii. Assign this port profile specifically to the port connected to FreeNAS (In the example below Port 2 is the port attached to FreeNAS)
4. This guide assumes traffic to the main FreeNAS installation comes in over one cable. (For my system, my freenas device has one cable connected to the IPMI for motherboard control, and the other is connected to FreeNAS with the FreeNAS port labeled igb0).
5. Ensure you can identify the name of the interface through which the cable is connected. Usually this can down done at the FreeNAS command line interface (CLI) and look for the network adapter that is shown to be active. Picture below shows two network adapter labeled igb0 and igb1. igb0 is the active network adapter whereas igb1 is labeled as "no carrier" since no networking cable is plugged into the port.
6. Make sure you have an alternative way to access your FreeNAS installation if network connectivity is disrupted (as it will most likely happen in setting up the VLANs). For me, my alternative access method was accessing the FreeNAS installation through the IPMI where I could open a console window into freenas. Once inside the console, I could manually bring up the network with the following command (substitute igb0 for your appropriate interface and 10.0.1.197 for the IP address currently associated with your freenas installation):
Code:
(Command run as root) ifconfig igb0 inet 10.0.1.197 netmask 255.255.255.0 up
Once igb0 was manually up, I was able to either to continue accessing FreeNAS through the IPMI, or through an ssh shell, however it is imperative that you are able to access FreeNAS through the WebGUI, as most of the configuration steps need to be done through the GUI. This process described above could likely be avoided if there was a control terminal connected to your FreeNAS installation. Unfortunately I run my FreeNAS installation headless so I had to complete the steps described above.
Please be sure you are very comfortable with this step as it is a requirement to complete the VLAN setup process.
VLAN configuration with FreeNAS
Overall Concepts
The overall concepts of working with VLANs inside of FreeNAS are:
a. Define VLANs inside of the FreeNAS GUI
b. Create a network bridge (virtual switch) for each VLAN and associate each VLAN to its corresponding network bridge.
c. Attach each jails VNET interface or each Virtual Machine's (VM's) tap interface to the appropriate bridge depending on what VLAN you would like the jail/VM to be a member.
d. Ensure the parent physical interface (ie igb0) DOES NOT have an associated IP address.
Traffic from the switch will enter FreeNAS and the traffic will be segregated by VLAN ID so that each VLAN will be passed to the appropriate bridge. Traffic ingressing to the bridge will be strip of it's VLAN ID and then passed untagged to the network interfaces of the member jails/VMs. Conversely, traffic ingressing to the bridge will enter untagged from the VM/jail. Inside the bridge, the traffic will be tagged with the VLAN ID associated with the bridge and then passed back to the network switch tagged.
I suggest associating VLAN1 with the default bridge bridge0, and then correspondingly associate VLAN10 with bridge10, VLAN20 with bridge20, etc. Please note that I am not associating VLAN1 with bridge1. In prior iterations of FreeNAS where the iohyve hypervisor was used -- FreeNAS 9.10 -- (hypervisor predating the current bhyve hypervisor), bridge0 was specifically hardcoded into the program and could not be altered). I've also found that bridge0 is usually created by some process if not specifically created during our setup. Due to numerous problems I've ran into, I found it best to associate usually the native VLAN1 traffic with the native bridge -- bridge0.)
0. Take snapshot of current FreeNAS installation prior to proceeding in case something goes terribly wrong (which it wont!!)
a. System->General->Scroll to bottom and Click Save Config (I'd recommend Exporting Password Secret Seed and Pool Encryption Keys
b. Save Config file to friendly place.
1. Define VLANs inside FreeNAS. For each VLAN you need to add or define:
a. Within Network->Interfaces Click ADD
b. Type->VLAN, Name->VLAN#, Parent Interface->igb0 (or your appropriate network adapter), VLAN Tag->#, Options->up
Once completed with the process, your Interfaces should be similar to the following:
2. Define the bridges
a. Similar to adding the VLANs add a bridge for each VLAN EXCEPT FOR BRIDGE0 ---This will be created with a system tunable.
b. Create Non-bridge0 bridges:--- Network-->Interfaces--> Click ADD
i. Type->Bridge, Name->Bridge#, Bridge Members->Corresponding VLAN#, Options->up
c. Create bridge0 via use of System Tunables
i. System->Tunables--> Click ADD
ii. Variable->cloned_interfaces, Value->bridge0, Type->rc.conf, Enabled->Yes
iii. System->Tunables-->Click ADD
iv. Variable->ifconfig_bridge0, Value->addm vlan1 up, Type->rc.conf, Enabled->Yes
3. Associate VLAN1 with FreeNAS IP address
This is probably the most tricky part of the entire process since network connectivity will be lost and need to re-established manually at the command line or other means. Please read following instructions a few times prior to proceeding:
a. Delete main network interface adapter -- (in my example this would be igb0). Once deleting the main network adapter your connectivity to FreeNAS will be lost
b. Re-establish connectivity to FreeNAS through alternative method as discussed above and reconnect to GUI.
c. Within GUI, Edit the Network->Interfaces->VLAN1 Entry. Within the IP address field, type the IP address of the freenas installation (in my case it was 10.0.1.197/24)
4. Disable any jails or VMs that start at boot then reboot system.
5. Re-evaluate system after boot.
Hopefully system will reboot appropriately and you'll be able to access your installation over the network. If for any reason you find you are unable to connect to your FreeNAS installation, try re-establishing contact through using your alternative method. Please bea ware I lost contact with my system multiple times trying to figure out the process and had to access the IPMI FreeNAS multiple times to access the system.
Once things have rebooted, please examine the network settings via the CLI using ifconfig. I'll post my ifconfig to discuss the salient points:
Code:
# ifconfig igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: igb0 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 hwaddr 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:95 hwaddr 0c:c4:7a:84:a5:95 nd6 options=1<PERFORMNUD> media: Ethernet autoselect status: no carrier lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan1 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1 vlanpcp: 0 parent interface: igb0 groups: vlan vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 inet 10.0.20.197 netmask 0xffffff00 broadcast 10.0.20.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 20 vlanpcp: 0 parent interface: igb0 groups: vlan vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan30 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 30 vlanpcp: 0 parent interface: igb0 groups: vlan vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan40 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 40 vlanpcp: 0 parent interface: igb0 groups: vlan vlan90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 90 vlanpcp: 0 parent interface: igb0 groups: vlan bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:14 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 55 bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:1e nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 20 priority 128 path cost 2000 member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 55 bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:28 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 17 priority 128 path cost 2000 member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 16 priority 128 path cost 2000 member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 55 bridge90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:5a nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan90 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 55 bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:00 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 21 priority 128 path cost 2000 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 15 priority 128 path cost 2000000 member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 14 priority 128 path cost 2000000 member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 20000
Points to notice:
- igb0 apapter does not have an associated IP address. If you have an IP address associated with the main network adapter, you will need to correct your interfaces
- vlan1adapter has associated inet address and netmask. The IP address should be the same IP address you use to access the FreeNAS GUI.
- Make sure for all the defined vlan, the parent adapter is the main network interface (in my example its igb0)
- Examine each bridge. Make sure each appropriate bridge contains the appropriate vlan member. For example above, vlan1 is a member of bridge0, vlan90 is a member of bridge90, etc.
Add jails/VM's to appropriate network bridge.
a. Jail Setup -- Prior to proceeding know there is a lot of information on the internet discussing on how to add a jail to a bridge. Jails have the option of either using a vnet driver which gives each jail its own virtual networking stack, or not using the vnet driver. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver (since it's usually not explicitly clear on discussed by the author). This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/.
I'm proceeding assuming the VNET interface will be used as this gives better security and network separation of the jails from the parent FreeNAS installation
For each jail I made the following entries
Code:
VNET on Berkley Packet Filter on IPV4 Interface - vnet0 IPV4 Address - 10.0.1.156 #Option only needed if not using DHCP - Assign desired IP address for the machine -- ie 10.0.1.156 for VLAN1, 10.0.40.156 for VLAN40 IPV4 Netmask - 24 #Option only needed if not using DHCP IPV4 Default Route - 10.0.1.1 #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1 vnet_interfaces none interfaces - vnet0:bridge0 #match this for your appropriate VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30 exec_fib 0 resolver - search local,nameserver 10.0.1.1,nameserver 1.1.1.1,nameserver 1.0.0.1 (See ***Note below) vnet_default_interface - auto
Notes:
***resolver-(Please note that in 11.2U7 I could use the auto option, with 11.3 I needed to explicitly name the domain name servers for the VLAN on this line. Please alter for your specific VLAN, ie for VLAN 40 I would use nameserver 10.0.40.1 instead of 10.0.1.1 -- VLAN90 = nameserver 10.0.90.1, etc).
For jail troubleshooting, I recommend you create a shell into the GUI from the FreeNAS interface, verify the IP address, netmask are correct. Verify the routing table is correct for your VLAN. Two examples of this information are shown below (#1 for VLAN1, #2 for VLAN40)
VLAN1
VLAN40
b: VM setup
VM's are easier than jails to associate with each VM.
For each VM:
1. Virtual Machine->Devices->NIC (EDIT)
2. Under Nic to Attach -> Choose the appropriate Network bridge for the desired VLAN
Debugging Jails/VMs
Once again in the main FreeNAS installation, examine ifconfig. Here is a more complete example
Code:
freenas% ifconfig igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: igb0 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 hwaddr 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:95 hwaddr 0c:c4:7a:84:a5:95 nd6 options=1<PERFORMNUD> media: Ethernet autoselect status: no carrier lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan1 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 1 vlanpcp: 0 parent interface: igb0 groups: vlan vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 inet 10.0.20.197 netmask 0xffffff00 broadcast 10.0.20.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 20 vlanpcp: 0 parent interface: igb0 groups: vlan vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan30 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 30 vlanpcp: 0 parent interface: igb0 groups: vlan vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: vlan40 options=200001<RXCSUM,RXCSUM_IPV6> ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 40 vlanpcp: 0 parent interface: igb0 groups: vlan vlan90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 0c:c4:7a:84:a5:94 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active vlan: 90 vlanpcp: 0 parent interface: igb0 groups: vlan bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:14 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 55 bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:1e nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 20 priority 128 path cost 2000 member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 55 bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:28 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 17 priority 128 path cost 2000 member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 16 priority 128 path cost 2000 member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 55 bridge90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:5a nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan90 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 55 tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: iohyve-arch-on-FreeNAS-- options=80000<LINKSTATE> ether 00:bd:15:81:f8:01 hwaddr 00:bd:15:81:f8:01 nd6 options=1<PERFORMNUD> media: Ethernet autoselect status: active groups: tap Opened by PID 2254 vnet0.1: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: ThrowAway as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:30:ba:15 hwaddr 02:6b:d0:00:10:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active groups: epair bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:00 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.13 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 19 priority 128 path cost 2000 member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 21 priority 128 path cost 2000 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 15 priority 128 path cost 2000000 member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 14 priority 128 path cost 2000000 member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 20000 tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: Attached to UbuntuMC options=80000<LINKSTATE> ether 00:bd:b5:13:ff:00 hwaddr 00:bd:b5:13:ff:00 nd6 options=1<PERFORMNUD> media: Ethernet autoselect status: active groups: tap Opened by PID 3156 bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f6:c7:64:02:01 nd6 options=1<PERFORMNUD> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: clamav as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:ac:89:b2 hwaddr 02:6b:d0:00:14:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active groups: epair vnet0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: mineos as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:c5:1d:5f hwaddr 02:6b:d0:00:11:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active groups: epair vnet0.12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: nextcloud as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:ba:b5:81 hwaddr 02:6b:d0:00:15:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active groups: epair vnet0.13: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: Borg as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:4f:ca:3f hwaddr 02:6b:d0:00:13:0a nd6 options=1<PERFORMNUD> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active groups: epair
Salient Points
- Examine each vnet interface and its association
- Make sure each listed vnet interface is a member of a bridge (ie vnet013 is a member of bridge0)
- For each jail and tap interface, make sure each tap interface of the appropriate bridge (ie tap0 is a member of bridge0).
- If you have verified the information within "ifconfig" looks correct and there is still no internet accessibility for the jail/VM, ensure the route and gateway is set appropriately (Within the jail - netstat -rn) and the nameservers are defined (Within the jail - /etc/resolv.conf) . You ideally would like to change configurations within the web gui menus rather than within the jail or VM itself since parameters might be modified on next boot of the jail or VM).
Future Directions
1. I'm unclear if all network traffic needs to be tagged when using VLANs with FreeNAS. I've yet to deconstruct my network setup to take on this challenge. If someone finds a setup where an untagged network could be used in conjunction with tagged networks, I'd be happy to modify the guide
2. Please be sure to examine and re-examine your ifconfig if you have problems. Usually there is a problem with a typo in the configuration of some options like resolver are not completed appropriately. When manually entering values be very mindful of syntax errors.
Good Luck and happy VLANing.
Last edited: