Help with Jails and VLANs

[Newfoundland.Republic]

Background:

• I have VLANs on my pfSense firewall - working fine
• I have my UniFi switch using the VLANs - working fine

I want to trunk VLANs into my FreeNAS jails.

I have created a VLAN (numbered 101) for the 192.168.101.0/24 subnet on the pfSense router and within the Unifi Controller.
I have the Unifi switch port set to all VLANs. I have tried setting the switch port to only use VLAN101 as well.
I created VLAN101 on FreeNAS on an interface (igb3) that was not otherwise used.
The VLAN interface igb3 is shown as up in the GUI.
I can confirm from the FreeNAS console that I can ping the pfSense VLAN gateway of 192.168.101.254

I created the jail with:
Basic Configuration

• VNET - enabled
• Berkely Packet Filter - enabled
• IPv4 Interface - vnet0
• IPv4 Address - 192.168.101.1
• IPv4 Netmask - 24
• IPv4 Default Router - 192.168.101.254

Network Properties

• interfaces - vnet0:bridge101
• vnet_default_interface - vlan101

This is likely something silly.

One other thing: I need to reboot the FreeNAS box (a real pain in the buttocks ;) as my ESXi hosts reside on an NFS mount) when I set up the VLAN as networking stops working outside of ssh to the FreeNAS box and the NFS network (on another, separate physical interface igb1; igb0 is the FreeNAS host interface in 192.168.20.0/24). The web page no longer is accessible. I tried restarting the network stack with /etc/rc.d/netif restart but that did not seem to work and "broke" ssh connectivity.

Any help/suggestions appreciated and welcome!

Config below for mobile users:

• Operating System: FreeNAS-11.2-U6
• Chassis: Supermicro SuperChassis 825TQ-R740LPB 2U 8 x 3.5" Drive Bays
• Power Supply: 2 x 740 Watt PWS-741P-1R Power Supply Platinum
• Backplane: Supermicro BPN-SAS-825TQ 8-port 2U TQ (W/ AMI 9072)
• Motherboard: Supermicro X9DR3-LN4F+
• CPU: 2 x Intel Xeon E5-2630 V1 Hex (6) Core 2.3GHz
• RAM: 32GB DDR3 ECC (8 x 4GB - DDR3 - REG)
• Storage Controller: LSI 9210-8i 6 GB/S
• Boot Pool: 2 x Kingston AS400 120 GB SSD Mirrored (using motherboard SATA 6 GB/s)
• Pool_1: 5 x WD Red 3 TB RAIDZ2
• 2 x MiniDLNA servers (seperate subnets) with shared content
• NFS server for VMware ESXi 6.0
• CIFS for Windows clients

 

[Newfoundland.Republic]

I think I am getting closer. I forgot the tunables:

• Added to variable: cloned_interfaces - value: vlan101 - type: rc
• Added variable: ifconfig_bridge101 value: addm vlan101 up - type: rc

Here is the VLAN network configuration:
Vlan Interface: vlan101
Parent Interface: igb3
Vlan Tag: 101

Jail Basic Properties:
VNET: Selected
Berkley Packet Filter: Selected
IPv4 Interface: vnet0
IPv4 Address: 192.168.101.1
IPv4 Netmask: 24
IPv4 Default Router: 192.168.101.254

Network Properties:
interfaces: vnet0:bridge101
vnet_default_interface: vlan101

I'm not sure about my Unifi port configuration.
I have VLAN30 as the native VLAN - this is the network that my WiFi uses. Will be shared with VLAN101.
I have the port tagged with VLAN101. Should that be on the port?

Thanks again! All suggestions and advice are welcome!

 

[Newfoundland.Republic]

Additional information on the host:

Code:

igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether 00:25:90:2f:87:c7
    hwaddr 00:25:90:2f:87:c7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

bridge101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:28:9e:b9:d7:65
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0:17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vlan101 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 55

vlan101: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 00:25:90:2f:87:c7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 101 vlanpcp: 0 parent interface: igb3
    groups: vlan

Output from netstat -rn in the jail:

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.101.254    UGS     epair0b
127.0.0.1          link#1             UH          lo0
192.168.101.0/24   link#2             U       epair0b
192.168.101.1      link#2             UHS         lo0

Jail config.json:

Code:

{
    "CONFIG_VERSION": "14.1",
    "allow_chflags": "0",
    "allow_mlock": "0",
    "allow_mount": "0",
    "allow_mount_devfs": "0",
    "allow_mount_nullfs": "0",
    "allow_mount_procfs": "0",
    "allow_mount_tmpfs": "0",
    "allow_mount_zfs": "0",
    "allow_quotas": "0",
    "allow_raw_sockets": "0",
    "allow_set_hostname": "1",
    "allow_socket_af": "0",
    "allow_sysvipc": "0",
    "allow_tun": "0",
    "available": "readonly",
    "basejail": "no",
    "boot": "off",
    "bpf": "yes",
    "children_max": "0",
    "cloned_release": "11.2-RELEASE",
    "comment": "none",
    "compression": "lz4",
    "compressratio": "readonly",
    "coredumpsize": "off",
    "count": "1",
    "cpuset": "off",
    "cputime": "off",
    "datasize": "off",
    "dedup": "off",
    "defaultrouter": "192.168.101.254",
    "defaultrouter6": "none",
    "depends": "none",
    "devfs_ruleset": "4",
    "dhcp": "off",
    "enforce_statfs": "2",
    "exec_clean": "1",
    "exec_fib": "0",
    "exec_jail_user": "root",
    "exec_poststart": "/usr/bin/true",
    "exec_poststop": "/usr/bin/true",
    "exec_prestart": "/usr/bin/true",
    "exec_prestop": "/usr/bin/true",
    "exec_start": "/bin/sh /etc/rc",
    "exec_stop": "/bin/sh /etc/rc.shutdown",
    "exec_system_jail_user": "0",
    "exec_system_user": "root",
    "exec_timeout": "60",
    "host_domainname": "pelleys.com",
    "host_hostname": "test101",
    "host_hostuuid": "test101",
    "host_time": "yes",
    "hostid": "9ca52239-bfbb-11e9-bc1b-0025902f87c4",
    "hostid_strict_check": "off",
    "interfaces": "vnet0:bridge101",
    "ip4": "new",
    "ip4_addr": "vnet0|192.168.101.1/24",
    "ip4_saddrsel": "1",
    "ip6": "new",
    "ip6_addr": "none",
    "ip6_saddrsel": "1",
    "jail_zfs": "off",
    "jail_zfs_dataset": "iocage/jails/test101/data",
    "jail_zfs_mountpoint": "none",
    "last_started": "2019-12-08 22:33:27",
    "login_flags": "-f root",
    "mac_prefix": "002590",
    "maxproc": "off",
    "memorylocked": "off",
    "memoryuse": "off",
    "mount_devfs": "1",
    "mount_fdescfs": "1",
    "mount_linprocfs": "0",
    "mount_procfs": "0",
    "mountpoint": "readonly",
    "msgqqueued": "off",
    "msgqsize": "off",
    "nmsgq": "off",
    "notes": "none",
    "nsemop": "off",
    "nshm": "off",
    "nthr": "off",
    "openfiles": "off",
    "origin": "readonly",
    "owner": "root",
    "pcpu": "off",
    "priority": "99",
    "pseudoterminals": "off",
    "quota": "none",
    "release": "11.2-RELEASE-p14",
    "reservation": "none",
    "resolver": "192.168.20.254",
    "rlimits": "off",
    "securelevel": "2",
    "shmsize": "off",
    "stacksize": "off",
    "stop_timeout": "30",
    "swapuse": "off",
    "sync_state": "none",
    "sync_target": "none",
    "sync_tgt_zpool": "none",
    "sysvmsg": "new",
    "sysvsem": "new",
    "sysvshm": "new",
    "template": "no",
    "type": "jail",
    "used": "readonly",
    "vmemoryuse": "off",
    "vnet": "on",
    "vnet0_mac": "0025900fc41e 0025900fc41f",
    "vnet1_mac": "none",
    "vnet2_mac": "none",
    "vnet3_mac": "none",
    "vnet_default_interface": "bridge101",
    "vnet_interfaces": "none",
    "wallclock": "off"
}#     

Note that "vnet_default_interface": "bridge101" has been tried with igb0 (parent interface) and vlan101 as well. Same result.

Are there any more config file information needed to help me out?

Thanks!

 

[raidflex]

VLANs have been broken for the most part in Freenas, I gave up a while ago trying to get it to work. I believe with 11.3, these issues have been resolved, but I cannot confirm this.

 

[Newfoundland.Republic]

I'm starting to think you are right @raidflex.

Here's some additional information

Here’s the dump from the FreeNAS host using "tcpdump -vv -i vlan101 | grep 192.168.101:
18:09:58.030870 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28
18:09:59.081688 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28
18:10:00.145060 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelley.com tell 192.168.101.1, length 28

Comparison with igb3 capture:
18:14:15.139102 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28
18:14:16.184518 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28
18:14:17.248119 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28

For chuckles and giggles I did the same capture for igb0, igb1 and ibg2. No packets were captured.

I just ran tcpdump in the jail. Here is the output:

18:19:59.723686 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28
18:20:00.768874 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28
18:20:01.832370 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28

I seems that part of the networking - at least on the host - is working. Could it be my switch VLAN tagging?