SOLVED jail connected and working through ethernet port that is marked down?

[jordanthompson]

I have a Plex jail that is working fine. It (seems) to be connected to a network interface (igb1:plex) that is marked as down:

I have a 100MB network port on the motherboard that is disabled in BIOS.
I have a 4-port GB network card connected to a switch.
Ports 1 & 2 on the network card are connected to the switch.
When I connect ports 3 &/or 4, my network crashes (still trying to figure that one out - could be a bad card or switch...) so I can only connect the first two ports.
Why/how is Plex connected to igb1:plex when it is marked as down?

 

[Patrick M. Hausen]

Please post the output of ifconfig -a as well as iocage get all plex (assuming that is the name of the jail) both on the host.

I suspect TrueNAS automatic bridge magic is building network loop as soon as you plug in the other interfaces.

 

[jordanthompson]

output of ifconfig -a:
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: truenas options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether 00:1b:21:6d:51:30 inet 10.13.0.120 netmask 0xffffff00 broadcast 10.13.0.255 media: Ethernet autoselect status: no carrier nd6 options=9<PERFORMNUD,IFDISABLED> igb1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: plex options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether 00:1b:21:6d:51:31 media: Ethernet autoselect status: no carrier nd6 options=9<PERFORMNUD,IFDISABLED> igb2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: server options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether 00:1b:21:6d:51:34 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> igb3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: ubuntu options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether 00:1b:21:6d:51:35 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0<> metric 0 mtu 33160 groups: pflog bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:33:44 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000 member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 2 priority 128 path cost 2000000 member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 20000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> ether fe:a0:98:6d:c4:fa hwaddr 58:9c:fc:10:ff:dd groups: tap media: Ethernet autoselect status: active nd6 options=9<PERFORMNUD,IFDISABLED> Opened by PID 2053 bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:db:23 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000 member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 3 priority 128 path cost 20000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: webServer as nic: epair0b options=8<VLAN_MTU> ether 16:da:e9:3b:25:00 hwaddr 02:d3:e0:7d:de:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> vnet0.7: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: plex as nic: epair0b options=8<VLAN_MTU> ether 16:da:e9:14:fa:09 hwaddr 02:27:0a:ab:73:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED>

output of iocage get all plex:
CONFIG_VERSION:28 allow_chflags:0 allow_mlock:0 allow_mount:0 allow_mount_devfs:0 allow_mount_fusefs:0 allow_mount_nullfs:0 allow_mount_procfs:0 allow_mount_tmpfs:0 allow_mount_zfs:0 allow_quotas:0 allow_raw_sockets:0 allow_set_hostname:1 allow_socket_af:0 allow_sysvipc:0 allow_tun:0 allow_vmm:0 assign_localhost:0 available:readonly basejail:1 boot:1 bpf:1 children_max:0 comment:none compression:lz4 compressratio:readonly coredumpsize:off count:1 cpuset:off cputime:off datasize:off dedup:off defaultrouter:auto defaultrouter6:auto depends:none devfs_ruleset:4 dhcp:1 enforce_statfs:2 exec_clean:1 exec_created:/usr/bin/true exec_fib:0 exec_jail_user:root exec_poststart:/usr/bin/true exec_poststop:/usr/bin/true exec_prestart:/usr/bin/true exec_prestop:/usr/bin/true exec_start:/bin/sh /etc/rc exec_stop:/bin/sh /etc/rc.shutdown exec_system_jail_user:0 exec_system_user:root exec_timeout:60 host_domainname:none host_hostname:plex host_hostuuid:plex host_time:1 hostid:1f00f960-00c6-1500-310e-14dae9124c74 hostid_strict_check:0 interfaces:vnet0:bridge0 ip4:new ip4_addr:10.13.0.121/24 ip4_saddrsel:1 ip6:new ip6_addr:none ip6_saddrsel:1 ip_hostname:0 jail_zfs:0 jail_zfs_dataset:iocage/jails/plex/data jail_zfs_mountpoint:none last_started:2024-02-11 21:12:52 localhost_ip:none login_flags:-f root mac_prefix:16dae9 maxproc:off memorylocked:off memoryuse:off min_dyn_devfs_ruleset:1000 mount_devfs:1 mount_fdescfs:1 mount_linprocfs:0 mount_procfs:0 mountpoint:readonly msgqqueued:off msgqsize:off nat:0 nat_backend:ipfw nat_forwards:none nat_interface:none nat_prefix:172.16 nmsgq:off notes:none nsem:off nsemop:off nshm:off nthr:off openfiles:off origin:readonly owner:root pcpu:off plugin_name:plexmediaserver plugin_repository:https://github.com/freenas/iocage-ix-plugins.git priority:99 pseudoterminals:off quota:none readbps:off readiops:off release:12.3-RELEASE-p5 reservation:none resolver:/etc/resolv.conf rlimits:off rtsold:0 securelevel:2 shmsize:off stacksize:off state:up stop_timeout:30 swapuse:off sync_state:none sync_target:none sync_tgt_zpool:none sysvmsg:new sysvsem:new sysvshm:new template:0 type:pluginv2 used:readonly vmemoryuse:off vnet:1 vnet0_mac:16dae914fa09 16dae914fa0a vnet0_mtu:auto vnet1_mac:none vnet1_mtu:auto vnet2_mac:none vnet2_mtu:auto vnet3_mac:none vnet3_mtu:auto vnet_default_interface:igb1 vnet_default_mtu:1500 vnet_interfaces:none wallclock:off writebps:off writeiops:off

 

[jordanthompson]

I have torn my nework configuration apart (reset.)
I figured out that the reason the network interfaces were down is because the cables weren't connected (duh). I have now configured igb0 (dhcp) to be dedicated for the host. I am now trying to configure igb1 for the plex and am having issues.
I have been looking for a simple solution, but all of the ones I have found involve vnets (which I don't use.) I guess I am looking for
1: how to set up a dedicated interface for a single jail
2: how to share an interface with multiple jails (because I may run into this situation shortly)

I must admit that I am confused that you can't have multiple interfaces that support dhcp (they are, after all, different MAC's)

 

[jordanthompson]

I finally have one dedicated interface working with a single jail. I did this by giving the interface a dedicated ip address:

Then, I set the jail to DHCP:

Funny thing is that it somehow got a DHCP address (10.13.0.118) from the router even though it was specified as static:
root@plex:~ # ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0<> metric 0 mtu 33160 groups: pflog epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 16:da:e9:14:fa:0a hwaddr 02:2e:14:c9:fb:0b inet 10.13.0.118 netmask 0xffffff00 broadcast 10.13.0.255 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=1<PERFORMNUD>

Hopefully this is an acceptable solution for dedicating an interface to a jail... looking for comments here

Next I need to figure out how to hang several jails on a single interface...

BTW, this new configuration shows this interface (igb1) as down (even though it is connected and working) and the other two (igb2 and igb3) as up (even though they are not connected to my switch). igb0 is connected to the web interface of truenas (and working fine) so my assumption that "up" means physically connected and "down" means not is incorrect:

 

[Patrick M. Hausen]

Don't do this. You cannot have two interfaces with IP addresses in the same network. There's more than enough documentation here about that, start with this:

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

You need to create a bridge interface witg igb1 as the only member, not assign an IP address, and use VNET for your jails.

 

[jordanthompson]

Don't do this. You cannot have two interfaces with IP addresses in the same network. There's more than enough documentation here about that, start with this:

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

You need to create a bridge interface witg igb1 as the only member, not assign an IP address, and use VNET for your jails.

Thanks for the link to the explanation. Not sure I follow it all, but I understand it is a bad thing to do...

I deleted the settings for igb1 and created a bridge. I had an issue with busy hardware when creating the bridge on igb1 (and can't chase it down right now) so I created bridge2 on igb2 with an address of 10.13.0.51. I shut down all of the vms and jails and assigned all of the jails' to None interface except plex which I left assigned to the default "vnet_default_interface" and started it. Plex got an address of 10.13.0.119 and seems to be running fine.

I am confused that vnet_default_interface is not selectable from the dropdown and how does plex know to use bridge2?

 

[jordanthompson]

Some progress...
I created the bridge in igb2 with an ip address of 10.13.0.51 (nothing else set)
I set the interface on the jail as follows:
dhcp autoconfigure ipv4: unselected
nat: unselected
vnet: selected
berkeley packet filter: selected
vnet_default_interface: none
ipv4 interface: vnet0 (the only option)
ipv4 address: 10.13.0.52
ipv4 mask: 24
ipv4 default router: 10.13.0.1 (my home router)

this is working for me, but is it correct?

If I want to bind another vm to another physical interface, I assume that:
create another bridge to that interface
will there be another ipv4 interface (presumably vnet1?) to select?

thank you for your help and patience!

 

[jordanthompson]

I tried adding another bridge to igb1 with an address of 10.13.0.60/24 but got an error:
The network 10.13.0.0/24 is already in use by another interface. Is this a network mask issue?

 

[Patrick M. Hausen]

What exactly are you trying to achieve? Why not use a single interface (igb0), create a bridge with that, assign an IP address, and then connect your jails to that?

You want VNET. You don't want NAT jails. This is 2024. VNET is the best container networking concept since sliced bread. Each jail gets its own IP address. This is how it's supposed to work.

If you want to get fancy and connect your jails to a second interface so you have "the full bandwidth" for your NAS only and a dedicated jail interface - that's fine. But then you don't assign an IP address to that second interface. It's just a virtual switch connection to your real network. And of course then you have to connect both - your NAS, and the jail interface.

DON'T DO THAT RIGHT AWAY unless everything is configured correctly.

For igb1, don't use an IP address but disable hardware offloading and put "up" into the options field. Create e.g. "bridge1", because it's connected to igb1. Use igb1 as the only member and don't give it an IP address.

Then for the jails activate VNET, set vnet_default_interface to "none" and set interfaces to "vnet0:bridge1" (if you are using bridge1, that is). Configure a static IP address and default gateway for your jail, go.

Now some of these options might be disabled if you try to use plugins. The advice for plugins on this forum has been for a year or more: don't.

They are deprecated, sporadically work, and are a pain to update. Create standard jails as outlined above and install your applications in them.

@victort has links to scripts for installing various applications and @danb35 maintains the script to install Nextcloud.

Simply don't use plugins. And don't use NAT.

 

[Ericloewe]

You want VNET. You don't want NAT jails. This is 2024. VNET is the best container networking concept since sliced bread. Each jail gets its own IP address. This is how it's supposed to work.

+1, makes everything so much simpler.

 

[jordanthompson]

At this point, a single interface will do for now (ultimately I'd like to get them set up so each jail/vm has a dedicated interface because, "why not?"

I have removed all previous bridges and created a new one on igb0 (bridge1 as I wasn't sure if bridge0 is valid name):
name: bridge1
dhcp: unselected
bridge members igb0
disable hardware offloading: unselected
mtu: 1500
options: blank
ip address: 10.13.0.51

In the jail I get a little lost.
dhcp: unselected
nat: unselected
vnet: selected
BPF: selected (I assume it should be selected)
vnet_default_interface: none (I tried auto and the only other options are igb1, igb2, igb3, bridge1 is missing?)
ipv4 interface: vnet0 (the only option)
ipv4 address: 10.13.0.52/24
default router: 10.13.0.1

I installed plex as a plug-in years ago (and its been working fine) and have manually been updating it.

I am getting ready to move the system to a supermicro board (currently on an Asus ATX board) and wanted to "set it up the right way" this time.

 

[jordanthompson]

@Patrick M. Hausen, the only selection available under IPV4 interfaces is "vnet0", which I have chosen.

Here are my jail settings:

Here are my bridge settings:

Here are my interface settings:

Here is the output of ifconfig from the shell of the jail:

Code:

root@plex:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 16:da:e9:14:fa:0a
        hwaddr 02:6b:81:8a:9f:0b
        inet 10.13.0.50 netmask 0xff000000 broadcast 10.255.255.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>

Any help would be most welcome!

 

[jordanthompson]

I found the problem. Thanks to @Patrick M. Hausen for your help. I was editing the network properties of the jail from the first page where ipv4 interface was set to "vnet0" as that was the only option in the dropdown.
I just noticed there was a network tab in the jail configuration. In there, under interfaces, "vnet0:bridge0" was entered (I have no idea when I did this: I had read that bridge0 is a bug, so I had been avoiding it). I changed it to "vnet0:bridge1" and it is working now.

FWIW, I have no idea where anyone got the idea I had been using NAT. I purposefully had been avoiding it and if I had referenced it at any time, it was unintentional

 

[jordanthompson]

Now that I have one jail connected to one interface, I am trying to

If you want to get fancy and connect your jails to a second interface so you have "the full bandwidth" for your NAS only and a dedicated jail interface - that's fine. But then you don't assign an IP address to that second interface. It's just a virtual switch connection to your real network. And of course then you have to connect both - your NAS, and the jail interface.

So, now that I have a jail connected to a dedicated interface, I am trying to decipher your comment above. Do I need another bridge for igb2, or do I connect the second jail directly to igb2 without specifying an ip address?

 

[sretalla]

I think I understand where you are trying to go and why you couldn't get there...

If you want multiple NICs and jails to attach to one or other of those by assigning the jail to the desired bridge, you can absolutely do that, but you can't assign an IP address to the second (or any subsequent) bridge to the same subnet as any existing (bridge, LAGG, VLAN or NIC) interface on the system...

Good news... you don't need an IP address on every bridge (or even any of them if you have some interface to connect to the TrueNAS host on).

Just create your bridge and do what you did without assigning an IP address, then attach your jails and they will operate happily on the same LAN.

 

[Patrick M. Hausen]

So, now that I have a jail connected to a dedicated interface, I am trying to decipher your comment above. Do I need another bridge for igb2, or do I connect the second jail directly to igb2 without specifying an ip address?

Create a separate bridge for each interface that you want to connect jails to. Don't assign an IP address to either the interface or the bridge. Just put "up" in the options field of the interface.

Then use the interfaces section of the jail config: vnet0:bridgeX to define which bridge the jail connects to. IP addresses for the jail are assigned in the jail settings. The bridge needs no IP address. Bridges are layer2.

Make sure to always set vnet_default_interface to "none". If it's set to "auto", TrueNAS will automatically create a bridge0 with the interface that has the default route of the host. Depending on your existing topology and bridges that might lead to a loop in the network and a crash by broadcast storm.

 

[jordanthompson]

Create a separate bridge for each interface that you want to connect jails to. Don't assign an IP address to either the interface or the bridge. Just put "up" in the options field of the interface.

Then use the interfaces section of the jail config: vnet0:bridgeX to define which bridge the jail connects to. IP addresses for the jail are assigned in the jail settings. The bridge needs no IP address. Bridges are layer2.

Make sure to always set vnet_default_interface to "none". If it's set to "auto", TrueNAS will automatically create a bridge0 with the interface that has the default route of the host. Depending on your existing topology and bridges that might lead to a loop in the network and a crash by broadcast storm.

Works perfectly for multiple jails now! Thanks so much to everyone