Isolated Jail

[fsociety3765]

Hi all,

I have a need to create a jail that is completely isolated from everything else on my network.

I have a separate network interface added to TrueNAS that will be dedicated for use on this jail. I have created a dedicated network on my router of 192.168.100.1/29, no DHCP server, and I have set up the firewall so that this network is isolated and blocked off from everything else. It only has internet access.

In creating the jail, I assigned this new interface (vtnet1) and set a static IP (no DHCP server on this network).

The jail doesn't seem to be isolated though. Or at least not as isolated as I was expecting it to be. The jail cannot communicate with any of my other subnets which is what I expected, however, if I run ifconfig, it is still seeing the other TrueNAS network interface (vtnet0) and a bunch of other stuff.

Is there anything I am missing? Searching around, there are mentions of "bridges". Is that maybe my problem? Do I need to add this interface to a new bridge and then assign the bridge to the jail?

Thanks,

FS

 

[Patrick M. Hausen]

Could you please post the output of ifconfig once on your TrueNAS host and once inside that jail. I'm too tired to look at that just now, but will do tomorrow, unless somebody else is faster.

 

[fsociety3765]

Thanks. ifconfig returns:

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
    ether 5e:bf:82:c4:30:92
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.2 netmask 0xffffffff
    groups: lo
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f7:d4:53:3c:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: minio as nic: epair0b
    options=8<VLAN_MTU>
    ether 86:ec:94:33:fe:4e
    hwaddr 02:11:bf:07:e9:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: syncthing as nic: epair0b
    options=8<VLAN_MTU>
    ether 86:ec:94:5b:07:f8
    hwaddr 02:fb:a9:51:f9:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d2:4a:56:3c:72:42
    inet 192.168.100.2 netmask 0xfffffff8 broadcast 192.168.100.7
    media: Ethernet 10Gbase-T <full-duplex>
    status: active

 

[Patrick M. Hausen]

And inside the jail?

 

[fsociety3765]

Sorry, that is inside the jail.

 

[fsociety3765]

This ifconfig is from on the TrueNAS host:

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
    ether 5e:bf:82:c4:30:92
    inet 192.168.10.5 netmask 0xffffff00 broadcast 192.168.10.255
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    inet 127.0.0.2 netmask 0xffffffff
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:f7:d4:53:3c:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: minio as nic: epair0b
    options=8<VLAN_MTU>
    ether 86:ec:94:33:fe:4e
    hwaddr 02:11:bf:07:e9:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: syncthing as nic: epair0b
    options=8<VLAN_MTU>
    ether 86:ec:94:5b:07:f8
    hwaddr 02:fb:a9:51:f9:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d2:4a:56:3c:72:42
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
    nd6 options=1<PERFORMNUD>

 

[Patrick M. Hausen]

Sorry, that is inside the jail

Nope. The interface inside the jail is named epair0b or similar. You need to do iocage console <your jailname> and then ifconfig.

 

[fsociety3765]

I have doubled checked this again. And what I have provided is correct. There is no interface named "epair0b" or similar. It is as above.

I have set up direct SSH access to the jail, and that is what I was using but I have logged into it via the TrueNAS SSH session using the iocage console command and the results to ifconfig are the same as above.

The first output posted here is from inside the jail, the second is on the host TrueNAS.

 

[Patrick M. Hausen]

Did you enable VNET for the jail?

 

[fsociety3765]

No, that is currently disabled. Should I enable that? I think I disabled it as it wouldn't let me select "vtnet1" as the interface otherwise.

 

[Patrick M. Hausen]

Yes, you should. And set vnet_default_interface to "none" - this is important.

First step: set vtnet1 to "up" and don't assign an IP address on the TrueNAS host. Also disable hardware offloading for vtnet1.
Then create a "bridge1" interface with vtnet1 as the only member.

In the jail enable VNET, set vnet_default_interface to "none" and interfaces to vnet0:bridge1

That should do it.

 

[fsociety3765]

Hmm. Don't seem to be having much luck here. I have created the bridge and assigned vtnet1 to it. Whenever I try to disable hardware offloading on vtnet1 it won't save for some reason though. Does the hardware offloading need to be disabled directly on vtnet1 and not the bridge?

Even still, the bridge doesn't show up as an option in the jail properties for the IPv4 interface.

 

[Patrick M. Hausen]

Not the IPv4 interface. That stays at vnet0. Further down in the Network Properties:

And the disable hardware offloading is possibly not necessary when running virtualised as you seem to do. I honestly don't know. I run TrueNAS as my hypervisor host.

 

[fsociety3765]

I'm getting "Error: [EFAULT] Stopped due to VNET failure", when I attempt to start the jail.

 

[Patrick M. Hausen]

Try to set the IPv4 default router manually, please.

 

[fsociety3765]

OK, now we seem to be working! Thank you.

That's all I am seeing now which looks how I would expect it to.

 

[fsociety3765]

OK, one last thing I am now noticing since reconfiguring this is that DNS is not working. How can I specify the nameservers that the jail uses. I want to just set them to the Cloudflare ones.

 

[fsociety3765]

Sorted it. Had to edit /etc/resolv.conf. That was still referencing my internal DNS servers which are blocked by my firewall.

 

[Patrick M. Hausen]

That will be overwritten at the next jail restart. So don't.

Use this instead: