Create Jail with Multiple Virtual NIC's possible?

[Fox]

My TrueNas 13.1 server has two physical NIC's.

Recently, I reconfigured it to use the second NIC for VLAN only (igb1). So, I now can connect jails to igb1 and then bridge/vlan to specific networks.

That's working great.

But I'm also interested in adding multiple NIC's to a single jail so that the jail can access two different (virtual) networks at the same time. Is this possible? I tried searching for it, but have not found anything with any specific instructions. If anyone has a link to something like this, please feel free to post it.

As for why I want to do this, well, mostly just to try it, and it would be nice to be able to setup the jail while it has full access to several networks, and then once configured, remove almost all of the virtual NIC's, so it operates in a more isolated fashion.

Thanks

 

[Patrick M. Hausen]

Of course that is possible.

Have you created the necessary bridge interfaces statically? If yes, you set the "interfaces" parameter of your jail to e.g.

vnet0:bridge0,vnet1:bridge1

Voila, two interfaces inside the jail, one of which is connected to each bridge.

 

[Fox]

I tried vnet0:bridge1,vnet0:bridge2, but it gave an error when booting. Did not try vnet1:bridge2. I just stepped out but will try vnet1 as soon as I get back to the server.

 

[Patrick M. Hausen]

The jail has got n virtual interfaces: vnet0, vnet1, vnet2, ...

The "interfaces" parameter matches the virtual interfaces to the host's interfaces. You cannot have duplicates, of course.

 

[Fox]

Thanks.. This did the trick, though it took me a bit because I had to set the Interfaces (Jail->Network Properties->Interfaces) to vnet0:bridge0,vnet1:bridge1 and then go back to the Basic Properties->IPv4 Interface and selecting vnet0 in the drop down, along with entering a new second IP address with a proper subnet and then selecting "vnet2" from the IPv4 drop down.

Running "ifconfig" is now showing the two virtual NIC's, epair0b and epair1b but I think I have some work to do on the switch to make sure the vlan tags are being handled properly, I don't think I have the switch properly configured yet.

Oh, one other issue is that I'm seeing "kernel: bridge2: can't disable some capabilities on vlan2: 0x400" on the host (truenas) log for both bridge1 and bridge2.. Should I be concerned?

Thanks

 

[Patrick M. Hausen]

Please post the output of ifconfig on the host in enclosed in CODE tags.

 

[Fox]

Everything seems to be working fine now, though I'm still seeing the "can't disable some capabilities" statement in the log when starting the jail.
Here is the ifconfig data from the host:

Code:

igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: IGB0
    options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:54
    inet 10.0.100.9 netmask 0xffff0000 broadcast 10.0.255.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:55
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
vlan5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan5
    options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:55
    groups: vlan
    vlan: 5 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan1
    options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:55
    groups: vlan
    vlan: 1 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan99: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan99
    options=4200401<RXCSUM,LRO,RXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:55
    groups: vlan
    vlan: 99 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan11
    options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether 0c:c4:7a:02:49:55
    groups: vlan
    vlan: 11 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: bridge5
    ether 58:9c:fc:00:48:4a
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: bridge1
    ether 58:9c:fc:00:27:30
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge99: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: bridge99
    ether 58:9c:fc:10:ff:93
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.21 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vlan99 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: bridge11
    ether 58:9c:fc:10:ff:9f
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:1d:24
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 23 priority 128 path cost 2000
    member: vnet0.8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 18 priority 128 path cost 2000
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000
    member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether fe:a0:98:76:15:93
    hwaddr 58:9c:fc:10:ff:d5
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
    Opened by PID 2623
vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: misc-apps as nic: epair0b
    options=8<VLAN_MTU>
    ether 0c:c4:7a:a5:a4:7e
    hwaddr 02:b6:e8:06:c9:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: mariadb as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:c4:7a:b6:a0:8b
    hwaddr 02:5b:6d:62:35:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.4: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: programming as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:c4:7a:cc:4c:72
    hwaddr 02:d9:46:73:ab:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.8: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: nginx as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:c4:7a:aa:70:0d
    hwaddr 02:f5:40:ab:19:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.9: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: testpkg as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:2f:6c:28
    hwaddr 02:70:d5:4d:5b:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.21: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: TesterVlan as nic: epair0b
    options=8<VLAN_MTU>
    ether 06:c4:7a:f0:9c:c5
    hwaddr 02:68:1d:ca:aa:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>

 

[Patrick M. Hausen]

You must explicitly create interface bridge0 and move the ip address configuration from igb0 to bridge0.

Easiest way is to disable all jails' autoboot and reboot the system. Then configure the jails to use bridge0 explicitly and set vnet_default_interface to "none".

 

[Fox]

But I'm using igb1 for the vlan stuff.. igb0 is used for the TrueNas box itself.. I initially was going to set it (igb0) up for vlan, and I think I started to attempt it months ago, but found it was easier to use the second NIC (igb1) for the vlan and jails.

Is there something in my configuration that implies something is misconfigured?

 

[Patrick M. Hausen]

Look at bridge0. There are 7 jails connected there - the vnet0.X interfaces. If your jails should only be connected to the VLAN bridges, make sure to set vnet_default_interface to "none" for each jail. If you leave it at "auto", TrueNAS will happily create a bridge on the interface with the default route and connect all your jails there.

 

[Fox]

Ohh.. Ya, those haven't been switched over yet.. I should have mentioned that. I just set up "TesterVlan" jail for vlan currently. I plan to switch the rest, but I don't want to disrupt the whole system, as the switches are all new and I'm still learning how to configure them. I have locked myself out of one switch already by getting to aggressive with converting to vlan. It's a pain when you have to do a reset on a switch.

Anyway, I'm just concerned about the "can't disable some capabilities" I mentioned above, which I see in the host (truenas) logs.. I'm curious what's causing that.