How to setup VLANs within FreeNAS 11.3

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
This post is a follow-up solution for solving how to setup VLANS with 11.2U7 (https://forums.lawrencesystems.com/t/freenas-jails-with-vlan/3046/11). Both 11.2 and 11.3 are VLAN able, however the documentation for getting a working setup is lacking.

Pre-requisites:
1. This guide assumes your hardware is all VLAN compatible -- meaning your router and switches are 802.1q compatible. This usually means that switches most be managed rather than unmanaged.

2. This guide also assumes that you have created and setup the appropriate VLANs with your hardware. My reference system is a virtualized pfsense router running inside of xcp-ng (Type I open-source Citrix hypervisor) using a variety of managed switches -- unifi and D-Link. For each one of these components, I needed to login to either the device or management software and create the VLANs with appropriate tags. This step of creating the VLANs within FreeNAS will be demonstrated below.
-- Implicitly this guide assumes you have setup a DHCP server and gateway for each of your VLANs within your router. Firewall rules can be created to limit intra-VLAN communication, however initially its probably best to allow each VLAN to have access to all ports and IP addresses of the other LANS. For example a pfsense firewall rule on each VLAN that states all traffic originating on the VLAN is able to pass to all networks and all ports could be used.

3. VLAN hardware usually involves passing traffic for one network that is untagged. The the remainder of the networks that pass through the device, they must be tagged to include the appropriate VLAN ID. By default, usually most switches consider VLAN 1 to be the native or untagged network. In contrast the traditional setup of untagged/tagged traffic, many within the FreeNAS community recommend to tag all network traffic presented to FreeNAS. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/. This involves creation of a trunk port on the switch to which FreeNAS is attached that tags all traffic. Specifically VLAN1 traffic needs to be tagged egressing the switch port attached to the router.

a. Unfortunately this guide can not be so comprehensive to provide details instruction how to tag VLAN1 traffic. The individual user may need to consult their switch documentation how to tag VLAN1 on the port attached to the FreeNAS machine.

b. Given my reference switch is a unifi device, I'll provide an example of how one would tag all traffic using unifi. This would involved two steps:
i. Create a port profile that tags all networks -- (Settings->Profiles->Switch Ports). Click [Add Network Profile]. Under Tagged Networks-->Select All. (Caveat -- its possible that you may not want to include every single VLAN if they are not involved with FreeNAS, however Selecting the "LAN Network" ensures that VLAN 1 traffic is tagged. Choose any other networks that you want passed to FreeNAS here).

Screen Shot 2020-01-21 at 8.18.19 PM.png


ii. Assign this port profile specifically to the port connected to FreeNAS (In the example below Port 2 is the port attached to FreeNAS)
Screen Shot 2020-01-21 at 8.22.46 PM.png


4. This guide assumes traffic to the main FreeNAS installation comes in over one cable. (For my system, my freenas device has one cable connected to the IPMI for motherboard control, and the other is connected to FreeNAS with the FreeNAS port labeled igb0).

5. Ensure you can identify the name of the interface through which the cable is connected. Usually this can down done at the FreeNAS command line interface (CLI) and look for the network adapter that is shown to be active. Picture below shows two network adapter labeled igb0 and igb1. igb0 is the active network adapter whereas igb1 is labeled as "no carrier" since no networking cable is plugged into the port.

Screen Shot 2020-01-21 at 8.30.59 PM.png


6. Make sure you have an alternative way to access your FreeNAS installation if network connectivity is disrupted (as it will most likely happen in setting up the VLANs). For me, my alternative access method was accessing the FreeNAS installation through the IPMI where I could open a console window into freenas. Once inside the console, I could manually bring up the network with the following command (substitute igb0 for your appropriate interface and 10.0.1.197 for the IP address currently associated with your freenas installation):
Code:
(Command run as root)
ifconfig igb0 inet 10.0.1.197 netmask 255.255.255.0 up

Once igb0 was manually up, I was able to either to continue accessing FreeNAS through the IPMI, or through an ssh shell, however it is imperative that you are able to access FreeNAS through the WebGUI, as most of the configuration steps need to be done through the GUI. This process described above could likely be avoided if there was a control terminal connected to your FreeNAS installation. Unfortunately I run my FreeNAS installation headless so I had to complete the steps described above.
Please be sure you are very comfortable with this step as it is a requirement to complete the VLAN setup process.

VLAN configuration with FreeNAS

Overall Concepts
The overall concepts of working with VLANs inside of FreeNAS are:
a. Define VLANs inside of the FreeNAS GUI
b. Create a network bridge (virtual switch) for each VLAN and associate each VLAN to its corresponding network bridge.
c. Attach each jails VNET interface or each Virtual Machine's (VM's) tap interface to the appropriate bridge depending on what VLAN you would like the jail/VM to be a member.
d. Ensure the parent physical interface (ie igb0) DOES NOT have an associated IP address.

Traffic from the switch will enter FreeNAS and the traffic will be segregated by VLAN ID so that each VLAN will be passed to the appropriate bridge. Traffic ingressing to the bridge will be strip of it's VLAN ID and then passed untagged to the network interfaces of the member jails/VMs. Conversely, traffic ingressing to the bridge will enter untagged from the VM/jail. Inside the bridge, the traffic will be tagged with the VLAN ID associated with the bridge and then passed back to the network switch tagged.

I suggest associating VLAN1 with the default bridge bridge0, and then correspondingly associate VLAN10 with bridge10, VLAN20 with bridge20, etc. Please note that I am not associating VLAN1 with bridge1. In prior iterations of FreeNAS where the iohyve hypervisor was used -- FreeNAS 9.10 -- (hypervisor predating the current bhyve hypervisor), bridge0 was specifically hardcoded into the program and could not be altered). I've also found that bridge0 is usually created by some process if not specifically created during our setup. Due to numerous problems I've ran into, I found it best to associate usually the native VLAN1 traffic with the native bridge -- bridge0.)

0. Take snapshot of current FreeNAS installation prior to proceeding in case something goes terribly wrong (which it wont!!)
a. System->General->Scroll to bottom and Click Save Config (I'd recommend Exporting Password Secret Seed and Pool Encryption Keys
b. Save Config file to friendly place.

1. Define VLANs inside FreeNAS. For each VLAN you need to add or define:
a. Within Network->Interfaces Click ADD
b. Type->VLAN, Name->VLAN#, Parent Interface->igb0 (or your appropriate network adapter), VLAN Tag->#, Options->up

Once completed with the process, your Interfaces should be similar to the following:
Screen Shot 2020-01-21 at 9.05.46 PM.png


2. Define the bridges
a. Similar to adding the VLANs add a bridge for each VLAN EXCEPT FOR BRIDGE0 ---This will be created with a system tunable.
b. Create Non-bridge0 bridges:--- Network-->Interfaces--> Click ADD
i. Type->Bridge, Name->Bridge#, Bridge Members->Corresponding VLAN#, Options->up
c. Create bridge0 via use of System Tunables
i. System->Tunables--> Click ADD
ii. Variable->cloned_interfaces, Value->bridge0, Type->rc.conf, Enabled->Yes
iii. System->Tunables-->Click ADD
iv. Variable->ifconfig_bridge0, Value->addm vlan1 up, Type->rc.conf, Enabled->Yes

3. Associate VLAN1 with FreeNAS IP address
This is probably the most tricky part of the entire process since network connectivity will be lost and need to re-established manually at the command line or other means. Please read following instructions a few times prior to proceeding:
a. Delete main network interface adapter -- (in my example this would be igb0). Once deleting the main network adapter your connectivity to FreeNAS will be lost
b. Re-establish connectivity to FreeNAS through alternative method as discussed above and reconnect to GUI.
c. Within GUI, Edit the Network->Interfaces->VLAN1 Entry. Within the IP address field, type the IP address of the freenas installation (in my case it was 10.0.1.197/24)

4. Disable any jails or VMs that start at boot then reboot system.

5. Re-evaluate system after boot.

Hopefully system will reboot appropriately and you'll be able to access your installation over the network. If for any reason you find you are unable to connect to your FreeNAS installation, try re-establishing contact through using your alternative method. Please bea ware I lost contact with my system multiple times trying to figure out the process and had to access the IPMI FreeNAS multiple times to access the system.

Once things have rebooted, please examine the network settings via the CLI using ifconfig. I'll post my ifconfig to discuss the salient points:

Code:
# ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: igb0
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    hwaddr 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:95
    hwaddr 0c:c4:7a:84:a5:95
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan1
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 1 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    inet 10.0.20.197 netmask 0xffffff00 broadcast 10.0.20.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 20 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan30
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 30 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan40
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 40 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 90 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:14
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:1e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 20 priority 128 path cost 2000
    member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:28
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 55
bridge90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:5a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan90 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 55
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 21 priority 128 path cost 2000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000000
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 20000


Points to notice:
- igb0 apapter does not have an associated IP address. If you have an IP address associated with the main network adapter, you will need to correct your interfaces
- vlan1adapter has associated inet address and netmask. The IP address should be the same IP address you use to access the FreeNAS GUI.
- Make sure for all the defined vlan, the parent adapter is the main network interface (in my example its igb0)
- Examine each bridge. Make sure each appropriate bridge contains the appropriate vlan member. For example above, vlan1 is a member of bridge0, vlan90 is a member of bridge90, etc.

Add jails/VM's to appropriate network bridge.

a. Jail Setup
-- Prior to proceeding know there is a lot of information on the internet discussing on how to add a jail to a bridge. Jails have the option of either using a vnet driver which gives each jail its own virtual networking stack, or not using the vnet driver. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver (since it's usually not explicitly clear on discussed by the author). This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/.

I'm proceeding assuming the VNET interface will be used as this gives better security and network separation of the jails from the parent FreeNAS installation
For each jail I made the following entries

Code:
VNET on Berkley Packet Filter on
IPV4 Interface - vnet0
IPV4 Address - 10.0.1.156   #Option only needed if not using DHCP - Assign desired IP address for the machine -- ie 10.0.1.156 for VLAN1, 10.0.40.156 for VLAN40
IPV4 Netmask - 24               #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1  #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0   #match this for your appropriate VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - search local,nameserver 10.0.1.1,nameserver 1.1.1.1,nameserver 1.0.0.1  (See ***Note below)
vnet_default_interface - auto


Notes:
***resolver-(Please note that in 11.2U7 I could use the auto option, with 11.3 I needed to explicitly name the domain name servers for the VLAN on this line. Please alter for your specific VLAN, ie for VLAN 40 I would use nameserver 10.0.40.1 instead of 10.0.1.1 -- VLAN90 = nameserver 10.0.90.1, etc).

For jail troubleshooting, I recommend you create a shell into the GUI from the FreeNAS interface, verify the IP address, netmask are correct. Verify the routing table is correct for your VLAN. Two examples of this information are shown below (#1 for VLAN1, #2 for VLAN40)
VLAN1
Screen Shot 2020-01-21 at 10.14.10 PM.png


VLAN40
Screen Shot 2020-01-21 at 10.17.49 PM.png


b: VM setup
VM's are easier than jails to associate with each VM.
For each VM:
1. Virtual Machine->Devices->NIC (EDIT)
2. Under Nic to Attach -> Choose the appropriate Network bridge for the desired VLAN

Debugging Jails/VMs

Once again in the main FreeNAS installation, examine ifconfig. Here is a more complete example
Code:
freenas% ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: igb0
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    hwaddr 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:95
    hwaddr 0c:c4:7a:84:a5:95
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan1
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 1 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    inet 10.0.20.197 netmask 0xffffff00 broadcast 10.0.20.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 20 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan30
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 30 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan40
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 40 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 0c:c4:7a:84:a5:94
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 90 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:14
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:1e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 20 priority 128 path cost 2000
    member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
bridge40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:28
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: vlan40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 55
bridge90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:5a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan90 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 55
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-arch-on-FreeNAS--
    options=80000<LINKSTATE>
    ether 00:bd:15:81:f8:01
    hwaddr 00:bd:15:81:f8:01
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 2254
vnet0.1: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: ThrowAway as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:30:ba:15
    hwaddr 02:6b:d0:00:10:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.13 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 21 priority 128 path cost 2000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000000
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 20000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to UbuntuMC
    options=80000<LINKSTATE>
    ether 00:bd:b5:13:ff:00
    hwaddr 00:bd:b5:13:ff:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 3156
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f6:c7:64:02:01
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: clamav as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:ac:89:b2
    hwaddr 02:6b:d0:00:14:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: mineos as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:c5:1d:5f
    hwaddr 02:6b:d0:00:11:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: nextcloud as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:ba:b5:81
    hwaddr 02:6b:d0:00:15:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.13: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Borg as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:4f:ca:3f
    hwaddr 02:6b:d0:00:13:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair


Salient Points
- Examine each vnet interface and its association
- Make sure each listed vnet interface is a member of a bridge (ie vnet013 is a member of bridge0)
- For each jail and tap interface, make sure each tap interface of the appropriate bridge (ie tap0 is a member of bridge0).
- If you have verified the information within "ifconfig" looks correct and there is still no internet accessibility for the jail/VM, ensure the route and gateway is set appropriately (Within the jail - netstat -rn) and the nameservers are defined (Within the jail - /etc/resolv.conf) . You ideally would like to change configurations within the web gui menus rather than within the jail or VM itself since parameters might be modified on next boot of the jail or VM).

Future Directions

1. I'm unclear if all network traffic needs to be tagged when using VLANs with FreeNAS. I've yet to deconstruct my network setup to take on this challenge. If someone finds a setup where an untagged network could be used in conjunction with tagged networks, I'd be happy to modify the guide
2. Please be sure to examine and re-examine your ifconfig if you have problems. Usually there is a problem with a typo in the configuration of some options like resolver are not completed appropriately. When manually entering values be very mindful of syntax errors.

Good Luck and happy VLANing.
 
Last edited:

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
653
Very nice guide!

I have only one comment which is about tagging the FreeNAS itself (basically point 1. of your future directions section). I guess that this is an unnecessary hassle. If your switch is VLAN aware you should be able to mark the switch port VLAN for untagged traffic. So whatever traffic comes to that port without VLAN tag will be automatically tagged by the defined VLAN ID. I am not sure if this is possible with UBNT devices but bellow is a HP ProCurve switch example

Note that i haven't tested this on 11.3 RC2. This is how i have it set with my (old) 9.10 but as the important part is done on switch side i don't see a reason why it shouldn't work. From NAS perspective it is a plain network connection (no bridge, no vlan, etc...).
The second part of the magic are the Jails and ability to have them properly assigned on different VLANs (which was messed up in 11.2) and you've covered this quite nicely.

Anyway here is an example:
If my FreeNAS is connected to port5 all of the untagged traffic ingressing the switch gets tagged by VLAN ID 77 and is passed to my pfSense where appropriate rules are applied. See the port 5 bellow configured as Untagged for VLAN 77.
1579769311577.png


Then i have the same port5 configured as Tagged port for VLAN 55 which is a VLAN for my Jails. So whatever jail i have configured with VLAN 55 the traffic gets accepted on the switch and passed to router w/o any changes.
1579769584355.png


Few points:
- Main FreeNAS interface gets IP from DHCP. No hassle with bridges or VLANs (yes i have IPMI as well but it is better if it works out of the box)
- All of my Jails are tagged so i have separation between Jails and FreeNAS itself on network level.
- Each VLAN has its own DHCP server with its own network subnet.
- VLAN77 is in isolated subnet with restrictive rules.
- Device on VLAN77 can open connection to any jail/VLAN55. Call-back for "established/related" connections are allowed.
- Jails on VLAN55 can NOT open connection towards VLAN77 so whatever crap i have running in jails it can not directly connect to host FreeNAS IP nor other devices in the same subnet.
- You can NOT explicitly tag anything with ID77 since that ID is marked for Untagged port. That would be rejected by switch. If you need Jail on the same VLAN don't configure anything for he Jail. Switch will tag it automatically by ID77.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@HolyK
Ok, so your basically telling me that FreeNAS can handle tagged and untagged traffic (as opposed to all tagged traffic). I assume however you have a bridge(s) setup within your FreeNAS that are at least associated with the tagged traffic.

I suspected the mixing of tagged/untagged traffic was possible with FreeNAS however others have recommended against this type of mixed traffic here on these various forums. The problem with setting up everything as tagged -- as you pointed out -- is the initial setup hassle with the creation of bridges, alternative access means to FreeNAS etc. Would you mind posting an ifconfig from your main FreeNAS?
 

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
653
Ok, so your basically telling me that FreeNAS can handle tagged and untagged traffic (as opposed to all tagged traffic). I assume however you have a bridge(s) setup within your FreeNAS that are at least associated with the tagged traffic.

I suspected the mixing of tagged/untagged traffic was possible with FreeNAS however others have recommended against this type of mixed traffic here on these various forums. The problem with setting up everything as tagged -- as you pointed out -- is the initial setup hassle with the creation of bridges, alternative access means to FreeNAS etc. Would you mind posting an ifconfig from your main FreeNAS?
Well from technical perspective i don't see a reason for "not recommended" statement. It is a valid setup. Unless the justification to "not recommend" was due to the issues with 11.2 version. That would actually make sense.

I am OK to provide `ifconfig` BUT my system USB is now on my desk as well as my data discs :D I have 11.3 RC temporarily installed on SSD and new disks in the case for burn-in (actually running long smart on the spare disk). Once the full version is released i am going with fresh install and migrating all of the data to new pool. Anyway maybe my old post could help with some details. If i will have some time i will reboot with old env and post you details.
 
Last edited:

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
3. VLAN hardware usually involves passing traffic for one network that is untagged. The the remainder of the networks that pass through the device, they must be tagged to include the appropriate VLAN ID. By default, usually most switches consider VLAN 1 to be the native or untagged network. In contrast the traditional setup of untagged/tagged traffic, many within the FreeNAS community recommend to tag all network traffic presented to FreeNAS. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/. This involves creation of a trunk port on the switch to which FreeNAS is attached that tags all traffic. Specifically VLAN1 traffic needs to be tagged egressing the switch port attached to the router.

a. Unfortunately this guide can not be so comprehensive to provide details instruction how to tag VLAN1 traffic. The individual user may need to consult their switch documentation how to tag VLAN1 on the port attached to the FreeNAS machine.

I find this explanation confusing. It sounds like you are implying that there MUST be untagged traffic on a trunk port, which is not the case.

Its worth understanding that the native VLAN is ONLY relevant to traffic entering a port and while the native VLAN tag is often 1 it can be set on any decent switch (looks like its called "Native Network" on your switch). The following might help others understand:

- tagged traffic enters a trunk port which is a member of same VLAN as the traffic --> accepted
- tagged traffic enters a trunk port which is not a member of the same VLAN as the traffic --> dropped
- tagged traffic enters a access port (untagged port) --> dropped
- untagged traffic enters a access port --> assigned to ports VLAN membership and accepted or
assigned to native VLAN and accepted
- untagged traffic enters a trunk port --> assigned to native VLAN and accepted

d. Ensure the parent physical interface (ie igb0) DOES NOT have an associated IP address.

If I understand it correctly (which is debatable), if you do add an IP to the physical device then that might work in a similar manner to a native VLAN on that port, though I am not sure if that is supported (?). Its probably not best practice to mix tagged and untagged where possible.

2. Define the bridges
a. Similar to adding the VLANs add a bridge for each VLAN EXCEPT FOR BRIDGE0 ---This will be created with a system tunable.
b. Create Non-bridge0 bridges:--- Network-->Interfaces--> Click ADD
i. Type->Bridge, Name->Bridge#, Bridge Members->Corresponding VLAN#, Options->up
c. Create bridge0 via use of System Tunables
i. System->Tunables--> Click ADD
ii. Variable->cloned_interfaces, Value->bridge0, Type->rc.conf, Enabled->Yes
iii. System->Tunables-->Click ADD
iv. Variable->ifconfig_bridge0, Value->addm vlan1 up, Type->rc.conf, Enabled->Yes

My system is on FreeNAS-11.2-U5 and I do not have to take these steps. If I assign a jail to interfaces of vnet0:bridge99 and vnet_default_interface of vlan99 for example, the system will automatically create (and destroy) the bridge interface and assign the correct members.
 
Last edited:

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
My system is on FreeNAS-11.2-U5 and I do not have to take these steps. If I assign a jail to interfaces of vnet0:bridge99 and vnet_default_interface of vlan99 for example, the system will automatically create (and destroy) the bridge interface and assign the correct members.

Have your tried 11.3 and confirmed if your settings still work?
 

ChrisChros

Patron
Joined
Nov 24, 2018
Messages
218
Hi @KevDog
Thanks for this really good explained instruction to add multiple VLan's.
After adding the Tuneables and a system reboot I can access FreeNAS GUI through its main IP.
i. System->Tunables--> Click ADD
ii. Variable->cloned_interfaces, Value->bridge0, Type->rc.conf, Enabled->Yes
iii. System->Tunables-->Click ADD
iv. Variable->ifconfig_bridge0, Value->addm vlan1 up, Type->rc.conf, Enabled->Yes
Bildschirmfoto 2020-01-25 um 09.40.32.png


But when I check
Code:
ifconfig
I am not able to see the bridge0. All other bridges and VLan's are listed but not bridge0.
Code:
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Bridge IoT
    ether 02:be:84:69:45:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Bridge KNX
    ether 02:be:84:69:45:14
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
root@FreeNAS:~ # 


Is this any problem, because I can access my FreeNAS?
Regards Chris
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@Chris Baldwin
I'm glad its working however its hard to say b/c I can't see all of your ifconfig. What bridge is VLAN1 a member of?
 

ChrisChros

Patron
Joined
Nov 24, 2018
Messages
218
@KevDog
I think after a reboot everything looks OK.
If you want you can have a quick look to my complete ifconfig.
Code:
root@FreeNAS:~ # ifconfig
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 90:e2:ba:7b:1c:dc
    hwaddr 90:e2:ba:7b:1c:dc
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: 10 Gbit Ethernet Port 2
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 90:e2:ba:7b:1c:dd
    hwaddr 90:e2:ba:7b:1c:dd
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Vlan1
    options=200401<RXCSUM,LRO,RXCSUM_IPV6>
    ether 90:e2:ba:7b:1c:dc
    inet 192.168.1.8 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    vlan: 1 vlanpcp: 0 parent interface: ix0
    groups: vlan
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: IoT
    options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 90:e2:ba:7b:1c:dc
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    vlan: 10 vlanpcp: 0 parent interface: ix0
    groups: vlan
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: KNX
    options=200401<RXCSUM,LRO,RXCSUM_IPV6>
    ether 90:e2:ba:7b:1c:dc
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    vlan: 20 vlanpcp: 0 parent interface: ix0
    groups: vlan
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Bridge IoT
    ether 02:be:84:69:45:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Bridge KNX
    ether 02:be:84:69:45:14
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000000
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:be:84:69:45:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: nextcloud as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:ba:b5:81
    hwaddr 02:4c:d0:00:0a:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Unifi_Controller as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:86:eb:97
    hwaddr 02:4c:d0:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to CentOS_7
    options=80000<LINKSTATE>
    ether 00:bd:1b:98:02:00
    hwaddr 00:bd:1b:98:02:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 11511
root@FreeNAS:~ #
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@ChrisChros

Things look good. Hopefully everything is working.
 

ChrisChros

Patron
Joined
Nov 24, 2018
Messages
218
At the moment yes.
There is only one thing I have observed. When I am after booting in the FreeNAS Console there is one message shown underneath the menu.
Code:
freenas bridge0 can't disable some capabilities on vlan1 0x400


Is this also shown in your system?
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
Have your tried 11.3 and confirmed if your settings still work?
Sorry KevDog, I dont have a 11.3 test rig and I will have to wait for it to be stable before testing.

Are you able to run some tests, instead?

My procedure would be something like:
  1. Create VLAN interfaces
  2. Give those VLAN interfaces an IP and make sure I can connect to webui from them
  3. Create a jail and use the interfaces of vnet0:bridge99 and vnet_default_interface of vlan99 example
I dont have any tunables, vlan1, bridge0 or to manually create any bridge interfaces.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
At the moment yes.
There is only one thing I have observed. When I am after booting in the FreeNAS Console there is one message shown underneath the menu.
Code:
freenas bridge0 can't disable some capabilities on vlan1 0x400


Is this also shown in your system?

I've never seen that error before -- however I probably should have added this to the guide. Depending on what time of hardware you have, you may need to adjust some tunables or disable hardware acceleration for your card. I hesitated to add this b/c -- it would just depend on your card type -- and if the bridge is needed just for jails.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Sorry KevDog, I don't have a 11.3 test rig and I will have to wait for it to be stable before testing.

Are you able to run some tests, instead?

My procedure would be something like:
  1. Create VLAN interfaces
  2. Give those VLAN interfaces an IP and make sure I can connect to webui from them
  3. Create a jail and use the interfaces of vnet0:bridge99 and vnet_default_interface of vlan99 example
I don't have any tunables, vlan1, bridge0 or to manually create any bridge interfaces.

I'd love to test for you and perhaps I can next week, however I've ran into another issue on 11.3 RC2 which isn't VLAN related and my system won't boot right now. I filed a bug report here if interested: https://jira.ixsystems.com/browse/NAS-104813. Hopefully I'll be up and running soon.
 

ChrisChros

Patron
Joined
Nov 24, 2018
Messages
218
I've never seen that error before -- however I probably should have added this to the guide. Depending on what time of hardware you have, you may need to adjust some tunables or disable hardware acceleration for your card.
As fare as I know should my NIC run out of the box with FreeNAS, it is Intel 10 GbE x520-DA2. I found only some threads that the NIC do not achieve the speed they would expect but not that the card needs some tunables to run propper.
Also the messages talks only about vlan1. my vlan10 and 20 also uses the same NIC so I would expect the same message for these two as well.
Could it be the tunable parameters for vlan1 and bridge0? Is there a way to create the bridge0 in the same way like the others?
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Do you have a reference to any thread you are talking about? For my VLANs I measured iperfs with and without the hardware acceleration on/off however I really didn't notice my difference. You however are using a 10Gb card whereas I only have 1Gb card.
 

ChrisChros

Patron
Joined
Nov 24, 2018
Messages
218
I don't have a thread, I searched here in the forum and in google. Intel X 520-DA2

But I think this has nothing to do with this message a saw in the FreeNAS Console.
 

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
Sorry KevDog, I don't have a 11.3 test rig and I will have to wait for it to be stable before testing.

Are you able to run some tests, instead?

My procedure would be something like:
  1. Create VLAN interfaces
  2. Give those VLAN interfaces an IP and make sure I can connect to webui from them
  3. Create a jail and use the interfaces of vnet0:bridge99 and vnet_default_interface of vlan99 example
I don't have any tunables, vlan1, bridge0 or to manually create any bridge interfaces.

Ok I tried what you wanted me to do -- long winded answer -- didn't exactly work

Ok I tried VLAN98 as a test
1. I defined VLAN98 within Network/Interfaces GUI
2. ? Not sure what you mean but I'm connected to Freenas GUI VLAN1
3. Created a jail. Interfaces I typed vnet0:bridge98 ---> I have to save config at this point and then go back into edit menu:
Once back in I went to vnet_default_interface and a drop down box presented itself with options of either none, auto, or vnet0:bridge98 --> you could not enter a value at all like vlan98. Anyway I tried both vnet0:bridge98 and auto however the jail woudn't start. For this to work for me I had to define both the VLAN and bridge within Network Interfaces and then leave vnet_default_interface at auto.

I'm aware that in 11.2U7 I could type in some of the values above -- however in RC2 now you are presented with a drop down list which you must choose one of the options listed -- its unfortunate because none of the options I want are shown -- for example -- just vlan98. So to answer your question --- I think the method I spoke about in the guide seems valid for this release.
 

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
653
@KevDog Hey! So referring to my previous post . I am on 11.3 and it works nicely (almost) out of the box ^^ :] No need to hassle with VLAN for NAS host system. Just keep it as "untagged" and handle the TAGing on the router/switch (details in prev post).

And for Jail do this (basically what you wrote so just to keep it short and easy):
- Create VLAN# interface with appropriate TAG
- Create BRIDGE# and mark the VLAN interface as member of the bridge. Do NOT check the physical NIC.
1580483993244.png

- Then create/edit Jail and under "Network Properties" for field "interfaces" enter "vnet0:bridge#" (replace the "#" by your number)
1580484822488.png

- The silver-bullet to make it work is to set field "vnet_default_interface" to "none" .
1580484848319.png

!! Do NOT use AUTO otherwise a host system interface is added to the bridge together with VLAN which breaks the setup. Jail would end up on the same network as the parent NIC rather than on the separate one defined by VLAN.
- Update other parameters as per your need (keep VNET and Berkley enabled). Anyway i suggest to make sure VLAN jail works OK before changing other values.
- Start the jail and validate :]

Details about ifconfig as requested before:

Host system:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether XX:XX:XX:XX:XX:XX
hwaddr XX:XX:XX:XX:XX:XX
inet 10.2.7.13 netmask 0xffffff00 broadcast 10.2.7.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan50: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vlan50
options=200001<RXCSUM,RXCSUM_IPV6>
ether XX:XX:XX:XX:XX:XX
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 50 vlanpcp: 0 parent interface: igb0
groups: vlan
bridge50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: bridge50
ether YY:YY:YY:YY:YY:YY
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: vlan50 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 20000
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: testjail1 as nic: epair0b
options=8<VLAN_MTU>
ether AA:AA:AA:AA:AA:A1
hwaddr BB:BB:BB:BB:BB:B1
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

Jail:
root@testjail1:/ # ifconfig -a
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether AA:AA:AA:AA:AA:A2
hwaddr BB:BB:BB:BB:BB:B2
inet 10.2.5.51 netmask 0xffffff00 broadcast 10.2.5.255
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

tcpdump from host system while pinging google from jail:
Frame 12: 102 bytes on wire (816 bits), 102 bytes captured (816 bits)
Ethernet II, Src: SuperMic_xx:xx:xx (XX:XX:XX:XX:XX:XX), Dst: EacAutom_13:c0:d5 (00:e0:67:13:c0:d5)
Destination: EacAutom_13:c0:d5 (00:e0:67:13:c0:d5)
Source: SuperMic_xx:xx:xx (XX:XX:XX:XX:XX:XX)
Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 50
000. .... .... .... = Priority: Best Effort (default) (0)
...0 .... .... .... = CFI: Canonical (0)
.... 0000 0011 0010 = ID: 50
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.2.5.51, Dst: 172.217.23.238
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 84
Identification: 0x6d87 (28039)
Flags: 0x00
 
Last edited:

KevDog

Patron
Joined
Nov 26, 2016
Messages
462
@HolyK

Phew -- nice writeup -- sometimes I get to the point I've figure I've wasted so much time on this topic that once its running I don't want to modify -- BUT

Just some clarification points for me so I can try when I have time
1. Your running untagged VLAN1 and tagged VLAN(s) into FreeNAS?
2. With your setup -- are you able to create a jail/VM that runs on this untagged network? I'm thinking for example of a bridge0, with untagged traffic on it? How would you set this up?
3. The remainder of your setup for the tagged networks - bridge/vlan creation and vnet configuration looks great. Thanks for that tip to set to none rather than auto. Mine are all set at auto.
 
Top