Looking for clarification on the various network settings for jails and how to properly configure a static IP address

[Mannekino]

Hi,

I'm trying to learn more about the various network settings for jails but I'm not getting a lot from the documentation unfortunately. I find the settings confusing and I'm hoping for a more layman explanation of the options. I currently have one jail running with Transmission and I'm using DHCP for this and it seems to be working well. I created a second test jail because I want to understand the networking options better and I would like to have a jail with a static IP address also.

Here is the ifconfig output of my FreeNAS server, it's running on bare metal and I have one psychical interface configured em1.

Code:

em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:74:57:e1
        hwaddr 00:25:90:74:57:e1
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:74:57:e0
        hwaddr 00:25:90:74:57:e0
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:dc:dd:47:50:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
vnet0:4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: transmission
        options=8<VLAN_MTU>
        ether 02:ff:60:03:aa:46
        hwaddr 02:b7:d0:00:04:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0:7: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: test
        options=8<VLAN_MTU>
        ether 02:ff:60:ae:1b:75
        hwaddr 02:b7:d0:00:06:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

And this is the ifconfig from my current test jail:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:60:ae:1b:76
        hwaddr 02:b7:d0:00:07:0b
        inet 192.168.178.5 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Questions about "Basic Properties"

Right now my test jail is configured as follows and I think this is the right way to go about it for using a static IP address but I'm not sure.

I have the following questions regarding this section of the configuration:

1. From what I read it is preferred to enable VNET for a jail. I'm still not 100% sure on what this does exactly. Is this similar to, lets say, setting up a "bridged" network in VMware Workstation or VirtualBox which I'm using on my workstation?
2. What is the best interface to select? I have the options listed below:
   1. em0: this would be my primary psychical network interface that is currently not in use
   2. em1: this would be my secondary psychical network interface that is connected to my home router
   3. bridge0: I don't fully understand how this differs from the vnet0 option, hoping to learn what the difference is between this and the last interface.
   4. vnet0: this interface appears when you check the VNET box above. Is this the preferred interface?
3. I'm assuming when you enable VNET for at least one jail the bridge0 device is created on the FreeNAS server and under that "bridge device" all the virtual network interfaces are created for the jails, am I getting this right conceptually?
4. Am I correct to think that the epair0b network interface inside the jail is the vnet0:7 interface outside of the jail, on the FreeNAS server?
5. It states that you need to enable the Berkeley Packet Filter when enabling DHCP. I assume it's best to not select it if you are using a static IP address configuration?

Questions about "Network Properties"

When I get to the "Network properties" I get completely lost unfortunately. I've read the tool tips and manual but I don't really understand.

1. When I switch the network interface from vnet0 to bridge0 the text in "interfaces" remains the same vnet0:bridge0. Shouldn't this text change also depending on the network interface I selected in "Basic Properties"? Why is it the same text for both the vnet0 and bridge0 interfaces. This goes back to my earlier question about the difference between those two.
2. The checkbox for ip4.saddrsel is always checked whether or not VNET is selected. The tool tip says "Only available when the jail is not configured to use VNET". If I select VNET the box is still ticked, why is this?
3. And lastly, why are there two mac-addresses listed in the box vnet0_mac? I assume these are two mac-addresses separated by a space? Why are there two?

Hopefully you guys can help me make sense of these configuration options, I really want to learn more about this.

Thank you in advance.

 

[Mannekino]

Subtle bump with some new information. I've been doing more reading over the past two days. I haven't found any satisfactory concrete answers yet to my question listed above. I hope someone with knowledge will reply and educate me.

I did try and create another jail using the transmission plugin with a static IP address. I also find I strange when creating a new Transmission plugin you cannot configure VNET in the wizard. You can only do this after the new jail has been created. So I first selected the bridge interface in the wizard when creating the new jail, afterwards I switched to VNET and set the default gateway for IPv4. This seems to be working fine but hasn't brought me any closer of understanding the underlying networking stuff.

I did figure out that indeed the epair0b interface inside the jail is indeed linked to a vnet interface outside the jail.

 

[Redcoat]

I found the information in the two threads below useful and enlightening

https://forums.freenas.org/index.php?threads/mac-address-from-plex-jail.71836/

https://forums.freenas.org/index.php?threads/change-mac-address-of-iocage-jail.68837/

 

[ascl]

I have been messing around with iocage trying to get an understanding so I can migrate away from my QNAP (and docker). I can't answer all of your questions, especially as I have focused on the command line rather then the UI (mainly because I want to be able to script creation, so if I have to rebuild it is straight forward). Your question about plugins and not being able to configure VNET via the wizard may be a limitation of iocage -- I hit something very similar, where if I create a jail with a list of packages, it will fail unless an IP address is explicitly assigned (so a DHCP assigned address wont work). I think plugins might hit the same limitation (and I will find out over the weekend, as I might create some plugins rather than scripting it all).

It seems an odd limitation, but is pretty easily worked around by specifying an IP address during the create, and altering the config after it is created.

Lastly, I think you are correct, that VNET is essentially the same as a VMWare bridged network. There is some good details here:
https://genneko.github.io/playing-with-bsd/system/learning-notes-on-jails/

 

[Nvious1]

If I create a jail via command line using this DHCP works fine for me. I think they just haven't build the UI for being able to set the IP address for the plugins cause according to the docs its simply: iocage fetch --plugins --name plexmediaserver ip4_addr="igb0|192.168.0.91"

Here is a working DHCP example.
iocage create -n "myJail" -r 11.2-RELEASE vnet="on" bpf="yes" dhcp="on" allow_raw_sockets="1" boot="on"

Code:

root@freenas[~]# iocage create -n "myJail" -r 11.2-RELEASE vnet="on" bpf="yes" dhcp="on" allow_raw_sockets="1" boot="on"
myJail successfully created!
* Starting myJail
  + Started OK
  + Configuring VNET OK
  + DHCP Address: 192.168.1.106/24
  + Starting services OK

If you want to pre-install packages, just create the json format and specify it. Example.

Code:

echo '{"pkgs":["nano","ca_root_nss"]}' > /tmp/pkg.json
iocage create -n "myJail" -p /tmp/pkg.json -r 11.2-RELEASE vnet="on" bpf="yes" dhcp="on" allow_raw_sockets="1" boot="on"

 

[ascl]

Thanks for that. You are correct, I had an error in my script! It does work for me as well.

 

[Mannekino]

Thanks for the replies so far. I've read all the links and I feel like I'm step closer to understanding the networking aspects of FreeNAS jails better. I dusted off (literally) my old FreeNAS server - a HP Microserver N40L - to do some more experimenting. Currently wiping the drives and going to install it with the latest FreeNAS version tomorrow.

One thing I don't yet understand is why you can select the bridge0 interface in the UI? Isn't that weird? That's not an actual interface a jail can use right? It's the bridge interface linking the virtual network interfaces to the psychical network interface on my FreeNAS server? Can anyone explain this?

Regarding my question about the two MAC addresses, I realize now that the first MAC address is linked to the vnet0:# interface of the FreeNAS server and the second MAC address is assigned to the epair0b. After reading some more about this I do have two more questions.

1. I thought the epair interfaces should start at epair0a, but for every new jail I create with VNET the epair interface is always called epair0b, why is that?
2. Why does FreeNAS increment the number of the vnet interface each time you restart the jail? I restarted my Plex jail a couple of times today. It started out as vnet0:2 and now it's vnet0:5.

 

[ascl]

Do you also get a lot of vnet interfaces that are not cleaned up when a jail is destroyed? Might explain why it is always incrementing. I did read something about instability with vnet teardowns, I wonder if this is the way FreeNAS avoids it?

 

[Mannekino]

I haven't seen any, should they pop up in ifcondig on the FreeNAS server? I currently just see two, one for my Plex jail, and one of my Transmission jail. The Plex one I restarted a couple times today and it's currently at vnet0:5.

 

[ascl]

Yes with ifconfig:
vnet0:11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: sonarr -- vnet0:12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: sonarr -- vnet0:15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: sonarr -- vnet0:21: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: sonarr -- vnet0:22: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: sonarr

This is from me playing around with my jail creation script and destroying the jail.

 

[Mannekino]

Hmm, no I haven't encountered this yet. But I've only deleted two test jails so far, both were transmission jails. I'm not seeing any orphaned vnet interfaces. I deleted them through the UI though, not via the CLI.

 

[ascl]

Ah, I was using the CLI (iocage destroy). Maybe that is the issue. Thanks.

 

[Mannekino]

@ascl

So today I installed my old HP Microserver N40L with the latest FreeNAS and did a little testing.

I created a jail using the web UI and configured my primary psychical interface first. At this time I had no bridge interface yet on the server. It didn't create a bridge after this.

Then I switched the network interface of the jail to DHCP with VNET. After that a bridge interface was created. I used the iocage destroy command to remove the test jail and I did not get any orphaned VNET interfaces.

I feel like the bridge interface shouldn't be visible in the drop down menu, that doesn't seem right because you can't use it as a network interface for the jail if i understand it correctly. Perhaps I should submit a bug report for this?

Is there anything you want me to test for you to see if i can reproduce any issues you're having?

 

[ascl]

Hmm, thanks for testing. I was creating and destroying quite a few (from the CLI) as I was flailing around trying to figure out how to make things work. It is quite possible I was mis-using commands to get into that state. I have since rebooted (for other reasons) which cleaned them up.

I think I agree with your assessment that having the bridge interface as a choice is at best misleading, worth a bug report I think. At this point I don't have any outstanding issues thankfully, and I have my 6 jails (and 2 VMs unfortunately) all set up and working from a networking perspective, and it all scripted (which I may post if there is any interest... beets, nzbget, organizr, plex, sonarr and a custom nodejs app).

Thanks!