How to setup VLANs within FreeNAS 11.3

[ferremontagud]

I'm sorry, it was because of the tests I've been doing.

Ifconfig from jail:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:15:5d:37:6f:d0
        hwaddr 02:5f:d0:00:08:0b
        inet 192.168.50.6 netmask 0xfffffffc broadcast 192.168.50.7
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

And routing tables:

Code:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.50.5       UGS     epair0b
127.0.0.1          link#1             UH          lo0
192.168.50.4/30    link#2             U       epair0b
192.168.50.6       link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     NetifExpire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

 

[ferremontagud]

And yes, netmask 255.255.255.252 is correct. If I configure this network on vlan11 adapter I can reach FreeNAS gui so the vlan is passing to FreeNAS server.

 

[KevDog]

Yea but with a /30 netmask I think there are two host addresses. Your gateway wouldn't be freenas, it would be your router.

Where does this come from?
192.168.50.6 link#2 UHS lo0

So maybe I'm confused but with a .4 network#, there are .5 and .6 addresses available and .7 is the broadcast address where is 192.168.50.4 is the network.

I'm not familiar with your setup, however if the jail is 192.168.50.6 then I'm guessing the router/gateway would be 192.168.50.5?

So are you able to ping the gateway from the jail?

And just curious -- why so restrictive with the /30 netmask? There is no way to assign the vlan11 an IP address to test things with this configuration.
It's not apparent on what you have told me why things aren't working.

 

[ferremontagud]

Yes, the setup is a fresh install of FreeNAS with basic configuration, is a testing environment for check if vlans are working fine with jails at 11.3 version.
I like to testing that works before updating the production servers :D

I attach a picture of the setup:

.

The /30 netmask is for testing with the same config as production servers but not a problem if I need to change it to perform more tests.

Thanks!

 

[ferremontagud]

And sorry I don't understand why I can't edit the previous message... I can not ping the gateway from the jail.
I am checking all config again but I not find the problem :(

 

[KevDog]

And sorry I don't understand why I can't edit the previous message... I can not ping the gateway from the jail.
I am checking all config again but I not find the problem :(

I've never tried what your exactly doing but can freenas function as a gateway?

Usually by convention the VLAN 1 is going to be placed on 192.168.1.0 network and VLAN 11 is going to be placed on 192.168.11.0 network. It probably could work the way you have it setup with the net masks limiting the number of available IP addresses, however I've never tried the way you are describing.

In addition, your really not describing you router setup. You'll need to define the VLANs on the router and I don't know if you need to adjust firewall rules for each VLAN. I'd break this down into manageable chunks and assign VLAN11 an IP address on FreeNAS. The freenas installation should thereby be reachable at either the 192.168.1.x address or 192.168.11.x address. You'll need to see if you can ping to/from jail to freenas installation with jails on both VLANs and then try pinging to/from jails to the gateway. You shouldn't need any scripts or changes to rc.conf or system tunables to make this work.

 

[ferremontagud]

In my installation FreeNAS do not function as a gateway but I will change the network settings as you say and keep testing.´
Thanks for the help ;)

 

[raidflex]

So I noticed on Freenas 11.3U1 even though I added the tunables the bridge0 is never created. If I go into a jail and choose vnet0:bridge0 the system will auto create the bridge 0, so I am not sure why the tunables are not working. I tried restarting multiple times. I can access the Freenas Ui without issues though.

 

[KevDog]

@raidflex . I need to update the guide. You are correct. With the new changes for whatever reason the system tunables for the vlan and bridge creation is ignored. You need to set up bridge 0 and vlan 1 or specifically all bridges and vlans with the gui. Just a heads up however. When you do this the first time you might lose connectivity and may need to configure network access on the command line to temporarily get the network working. I did this through a shell login in the ipmi

 

[raidflex]

I need to update the guide. You are correct. With the new changes for whatever reason the system tunables for the vlan and bridge creation is ignored. You need to set up bridge 0 and vlan 1 or specifically all bridges and vlans with the gui. Just a heads up however. When you do this the first time you might lose connectivity and may need to configure network access on the command line to temporarily get the network working. I did this through a shell login in the ipmi

Okay, thanks for the info and the great guide!

 

[raidflex]

So looks like most everything is up, but I have no internet access in the jails.

EDIT: Nevermind fixed the issue, I had specified the resolver in the jail networking settings which caused the issue. Once I reset it back to "/etc/resolv.conf" which is the default, internet access was restored.

 

[KevDog]

@raidflex -- What version of Freenas are you using -- I'm currently on just 11.3. I believe your post stated 11.3 U1. With your jails, are you using DHCP or setting static IP addresses?

 

[raidflex]

@raidflex -- What version of Freenas are you using -- I'm currently on just 11.3. I believe your post stated 11.3 U1. With your jails, are you using DHCP or setting static IP addresses?

All jails are setup with static addresses. Currently it looks like only the jails which are on the same bridge as the host have internet access. All other bridges associated with the other VLANs do not seem to have internet access inside the jail.

I also needed to set the default gateway and DNS in the global config or the host would have no internet access.

 

[KevDog]

Can you try setting on of the non working jails to DHCP and see if that works? In troubleshooting VLAN problems, I always want to make sure that your router/switches are setup appropriately so that they pass along the tag information. (It sounds like you probably had this setup and working with U7, so unless there were any changes to your router/switch config, I'm betting the tag information is probably the same). Are you working with one network interface of two network interfaces or multiple network interfaces? If using one interface -- did you tag all packets or do you have a combination of untagged and tagged packets? And finally -- please post "ifconfig" from the freenas system -- this usually helps

 

[raidflex]

Can you try setting on of the non working jails to DHCP and see if that works? In troubleshooting VLAN problems, I always want to make sure that your router/switches are setup appropriately so that they pass along the tag information. (It sounds like you probably had this setup and working with U7, so unless there were any changes to your router/switch config, I'm betting the tag information is probably the same). Are you working with one network interface of two network interfaces or multiple network interfaces? If using one interface -- did you tag all packets or do you have a combination of untagged and tagged packets? And finally -- please post "ifconfig" from the freenas system -- this usually helps

I am working with 1 physical interface and tagging all packets to that interface.

So I tried enabling DHCP on one of the jails and sure enough internet access was restored. I also ran an ifconfig in the jail and the jail had an ip from the correct subnet. Bridge20/Vlan20 was the one that I enabled DHCP FYI.

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:86:75:ae
    hwaddr 00:25:90:86:75:ae
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Plex
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:86:75:af
    hwaddr 00:25:90:86:75:af
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: 10.10.10.4
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 00:25:90:86:75:ae
    inet 10.10.10.4 netmask 0xffffff00 broadcast 10.10.10.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 10 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: IOT Devices
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 00:25:90:86:75:ae
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 20 vlanpcp: 0 parent interface: igb0
    groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Plex
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:86:75:ae
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 30 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Main Bridge
    ether 02:a6:3a:cc:d6:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.19 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vnet0.18 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: vnet0.17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000000
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 55
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: IOT Bridge
    ether 02:a6:3a:cc:d6:14
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.25 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000
    member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Plex Bridge
    ether 02:a6:3a:cc:d6:1e
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to XEOMA
    options=80000<LINKSTATE>
    ether 00:bd:5b:d8:4d:00
    hwaddr 00:bd:5b:d8:4d:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 21750
vnet0.17: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Emby as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:c0:af:79
    hwaddr 02:c0:d0:00:0a:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.18: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Newsgroups as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:58:7b:20
    hwaddr 02:c0:d0:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.19: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: UniFi_Controller as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:13:df:28
    hwaddr 02:c0:d0:00:0e:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
vnet0.25: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: InfluxDB as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:ff:60:21:d9:43
    hwaddr 02:c0:d0:00:0c:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

 

[KevDog]

@raidflex
I'm still running 11.3 (just plain release and not U1) and static IPs for jails work in this release. @nfld.republic tipped me off about an issue with static vs DHCP problems for the 11.3 U1 release. The bug report is here: https://jira.ixsystems.com/browse/NAS-105114. The bug is for specifically jails created by plugins however I'm betting it's probably valid for just VLANs and FreeNAS jails. The bug was confirmed that jails need to be DHCP to run with a VLAN in the 11.3 U1 release and that a fix is slated for U2 release. You can read the bug report yourself. As a temp workaround, you may just want to set all your jails to DHCP and have the IP address assigned by the router based on MAC address. I'm not sure what router you are using, however within pfsense this is a fairly straightforward process.

 

[raidflex]

I'm still running 11.3 (just plain release and not U1) and static IPs for jails work in this release. @nfld.republic tipped me off about an issue with static vs DHCP problems for the 11.3 U1 release. The bug report is here: https://jira.ixsystems.com/browse/NAS-105114. The bug is for specifically jails created by plugins however I'm betting it's probably valid for just VLANs and FreeNAS jails. The bug was confirmed that jails need to be DHCP to run with a VLAN in the 11.3 U1 release and that a fix is slated for U2 release. You can read the bug report yourself. As a temp workaround, you may just want to set all your jails to DHCP and have the IP address assigned by the router based on MAC address. I'm not sure what router you are using, however within pfsense this is a fairly straightforward process.

Well that explains it. I have always used static addresses with Freenas because way back DHCP never really worked well. I actually do not use any plugins and manually created all my jails. All jails are even at the latest 11.3-RELEASEp6 version and yet this issue persists. Glad there is a bug report for this. I have been using custom jails for many years and never got around to configuring Freenas with VLANs because something always seemed to be broken.

I will just add the DHCP reservations in my PFsense box, not big deal.

 

[KevDog]

@raidflex

Things weren't necessarily broken -- it's just you had to configure them via system tunables which is equivalent to command line configuration. What stinks kind of now is that the GUI middleware overwrites settings that I used to be able to set via system tunables. The DHCP thing - annoying for sure but at least the bug is confirmed and a fix is set for the next release.

 

[listhor]

Thanks @KevDog for tutorial, it made may life easier but not completely ;-).
In short, all works until I do any unrelated changes in network settings / or just reload them - all 3 jails (in different vlans - lagg is parent interface) loose connectivity. Seems like it's vnet issue. I have to restart them to restore their networking.
Jail's config:

Code:

    "allow_mount_devfs": 1,
    "allow_raw_sockets": 0,
    "boot": 1,
    "bpf": 1,
    "cloned_release": "11.3-RELEASE",
    "defaultrouter": "172.16.1.1",
    "devfs_ruleset": "5",
    "host_hostname": "plex",
    "host_hostuuid": "plex",
    "interfaces": "vnet0:bridge11",
    "ip4_addr": "172.16.1.3/26",
    "ip4_saddrsel": 1,
    "ip6_saddrsel": 1,
    "jail_zfs_dataset": "iocage/jails/plex/data",
    "last_started": "2020-03-28 19:09:02",
    "release": "11.3-RELEASE-p6",
    "vnet": 1,
    "vnet0_mac": "ac1f6b14fa09 ac1f6b14fa0a",
    "vnet_default_interface": "none"

Code:

root@plex:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=8<VLAN_MTU>
    ether ac:1f:6b:14:fa:0a
    hwaddr 02:e9:d0:00:13:0b
    inet 172.16.1.3 netmask 0xffffffc0 broadcast 172.16.1.63
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

What might be wrong there?

EDIT:
I've just discovered/confirmed that after issuing:

root@freenas[~]# ifconfig vnet0.3 down <- of course must be respective vent interface
root@freenas[~]# ifconfig vnet0.3 up

jail is reconnected to bridge. Despite that vnet# in subject is continuously displayed as member of corresponding bridge. I think, it means that vnet becomes somehow frozen and needs to be restarted??

 

[KevDog]

@listhor

Although I've read your problem, I don't know if I totally understand what you are describing:

In short, all works until I do any unrelated changes in network settings / or just reload them - all 3 jails (in different vlans - lagg is parent interface) loose connectivity

.

What kind of unrelated changes in network settings?