SOLVED Unstable Jail behaviour on Truenas 12.0 using Plex Danb script

[nikinp]

//EDIT: Solution HERE

Dear Truenasers,

I have been starting a new build on the system 1 as per my signature.
I used danb35 script to create a plex server. However, the jail seems unstable:

1. When stopping and starting the jail - it takes a long time almost hanging. Then goes to a splash page "connecting to Truenas Make sure the Truenas system is powered and connected to the network. Often requiring for me to log back in. Often times, it hasnt actually stopped/re-started the jail and I have to try again
2. After initially working and me playing some videos in plex. The server itself stops can no longer be found in the Plex desktop app
3. I have had to resort to re-setting the server in its entirety and/or deleting the jail and re-starting the script. Neither of these have seemed to get rid of these problems

Thanks to @Redcoat for help on a previous thread, and as per your suggestion I have started this new thread.
I would be grateful to all for some help here. Thank you in advance

 

[HolyK]

This seems like some network/IP conflict. Do you have some managed switch/router? How the IPs for FreeNAS/Jails are assigned (manually or DHCP)? How do you configure the network for your Jail? (Can you give us screenshot of the Jail network config)

Also Do you have some "head" for your NAS? (either monitor or IPMI)? I assume that when the connection to TrueNAS WEBUI stops working the SSH dies as well so you will need some sort of direct access if the network is being interrupted.

 

[nikinp]

I have a second freenas box on the network (see system 2 in my signature) and this behavour doesnt exist there.
My network setup is: cable modem->router/mesh network->managed switch
I have not made any configuration to network jail - it is based entirely from this script https://www.truenas.com/community/threads/scripted-plex-installation.79725/page-5#post-637325
The IP address for the jail is set within the config script and is an DHCP reserved range. The Freenas IP address is static and again in in a reserved range.

I dont know what a''head'. I have a monitor connected via VGA and it supports IPMI. Yes, when I am using putty then the sessions have been getting terminated

 

[HolyK]

Can you provide output of ifconfig -a and attach "basic properties" tab of that Jail as well please.
By "head" i mean something to see the server console. So in your case the monitor or IPMI. If you can replicate the issue and once the SSH session gets terminated check the output of ifconfig -a again.

Blind guess ... the jail is "stealing" the IP of the physical interface because it has it assigned in the properties.

How do you assign IP to your NAS? Do you have DHCP so router provides IP or do you have it manually configured?

 

[nikinp]

root@truenas[~]# ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
ether ac:1f:6b:1b:74:a0
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>


igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:1b:74:a1
media: Ethernet autoselect
status: no carrier
nd6 options=1<PERFORMNUD>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:72:0b:a5:6e:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>

vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: pms as nic: epair0b
options=8<VLAN_MTU>
ether ae:1f:6b:57:7e:c0
hwaddr 02:13:6e:d0:25:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>
root@truenas[~]#

I have a reserved IP address range and a range for DHCP and I gave the NAS an IP address in the reserved range.

 

[HolyK]

@nikinp Thx, one more piece of puzzle ... can you post screen of "Network -> Interfaces" please. Just to see if it matches the ifconfig output.

 

[nikinp]

@nikinp Thx, one more piece of puzzle ... can you post screen of "Network -> Interfaces" please.

 

[HolyK]

Ok i am not sure if the bridge is automatically created upon jail creation but not visible under the Network>Interfaces (i have it there because i've created it manually due to VLAN assignment).
Go there again, click "Columns" and check "Bridge members".

Also side-question ... do you intend to use VLANs?

 

[nikinp]

Ok i am not sure if the bridge is automatically created upon jail creation but not visible under the Network>Interfaces (i have it there because i've created it manually due to VLAN assignment).
Go there again, click "Columns" and check "Bridge members". Then see/post screen if the igb0 has bridge0 listed as member?

Also side-question ... do you intend to use VLANs?

I dont have an intended use for VLAN. I would like to make the most of the two LAN ports on the board, if this is something we can address as we troubleshoot this issue. I am assuming that the IPMI (3rd port) is unlikely to affect the issue here?

 

[HolyK]

IPMI does not play role here and it is not listed under Network Interfaces at all, that is correct (there is a separete IPMI section in menu). Thanks for the screen and clarification about the vlans.

Since you mentioned the igb1 ... You have two options here.
- Either you can go with VNET (that's what we will try to fix here)
- Or you can pass-thru the igb1 interface directly to the Jail. In that case you will need a second physical UTP cable between your Switch and NAS. And igb1 will need another IP address. Good thing is that in such scenario you can run both NAS and the Jail as fast as Gbit in parallel (if your router/switch is capable to do so) but from practical perspective i don't think it is necessary (but possible if you want). Plex is not that crazy for network bandwidth (unless you're feeding multiple client devices in parallel with 4k streams...). "Bad" thing is that you will need to do some extra cable management (duno how far you NAS is from your switch) and it will take the NIC. Also in case you will want a second Jail you will have to go with the VNET anyway.

Anyway for your current setup ... edit the Jail and go to "Basic properties", then tick the Barkley packet filter, save and try to start Jail.

 

[nikinp]

Anyway for your current setup ... edit the Jail and go to "Basic properties", then tick the Barkley packet filter and change the default_vnet_interface to None, save and try to start Jail.

To do this. I need to stop the jail to edit. However, when I try to stop jail, it seems to logmeout and then when I go back in the jail is still up.
I ran ifconfig -a output below;

root@truenas[~]# ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
ether ac:1f:6b:1b:74:a0
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:1b:74:a1
media: Ethernet autoselect
status: no carrier
nd6 options=1<PERFORMNUD>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:72:0b:a5:6e:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>

vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: pms as nic: epair0b
options=8<VLAN_MTU>
ether ae:1f:6b:57:7e:c0

 

[HolyK]

Ok so if that thing is currently running ... try to ssh > jls > jexec <jain_num> tcsh , that should give you shell from within jail. Use ifconfig -a again and paste results please.

 

[nikinp]

root@pms:/ # ifconfig -a

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether ae:1f:6b:57:7e:c1
hwaddr 02:13:6e:d0:25:0b
inet 192.168.0.11 netmask 0xffffff00 broadcast 192.168.0.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>

 

[HolyK]

Ok ... from shell do this ... and post output
iocage list --long iocage stop <jailname> iocage list --long

BTW Any chance you could join IRC on irc.freenode.net #freenas ?

 

[nikinp]

root@truenas[~]# iocage list --long

+-----+------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| JID | NAME | BOOT | STATE | TYPE | RELEASE | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+======+======+=======+======+=================+=======================+=====+==========+==========+
| 1 | pms | on | up | jail | 12.2-RELEASE-p4 | vnet0|192.168.0.11/24 | - | - | yes |
+-----+------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
root@truenas[~]# iocage stop pms
* Stopping pms
+ Executing prestop OK
+ Stopping services FAILED
ERROR:


Please use --force flag to force stop jail
root@truenas[~]# iocage list --long
+-----+------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| JID | NAME | BOOT | STATE | TYPE | RELEASE | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+======+======+=======+======+=================+=======================+=====+==========+==========+
| 1 | pms | on | up | jail | 12.2-RELEASE-p4 | vnet0|192.168.0.11/24 | - | - | yes |
+-----+------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
root@truenas[~]#

 

[nikinp]

BTW Any chance you could join IRC on irc.freenode.net #freenas ?

Trying to work out how I do this. Registration process is not clear at all

 

[HolyK]

After some hours of hunting the root-cause we figured it out ... So for the future generations bellow is explanation what the (hell) was going on and how to fix/overcome that annoying behavior.

Symptoms:
- Physical NIC gets cycled (up/down) upon first Jail with VNET start OR the last jail stop
- Whole network communication gets interrupted for ~10-15 secs (GUI/SSH drops as well as all of the CIFS and other stuff).
- Second and any additional Jail stop/start does NOT cause any issues. Only the first start and last stop

Cause:
- By default FreeBSD starts the network interfaces with bunch of the OPTIONS enabled (see bellow). It works OK until you actually start the first jail with VNET enabled. Apparently VNET does not work well with certain options so it enforces the parent Interface to disable some of them. When this happens the parent NIC needs to restart in order to make the changes. This was causing the network disconnects.
- Once the first Jail was running any further Jails were not causing issues because the NIC options were not present so no problem with startup another Jail, nothing to be done on parent nic. Just create a new vnet, add it to the bridge and all good.
- And the similar story happens in a opposite direction. When you stop second jail nothing happens with the options (because there is another Jail running) but as soon as you stop the last Jail the host system will just re-activate the options which were removed before ... and YES, this requires another cycle of the parent NIC causing another network outage ...

Solution (workaround?):
Disable the Hardware offload ... Two reasons ... It is mentioned in the TrueNAS docs (as it is causing various issues) ...

Disabling this is only recommended when the interface is managing Jails, Plugins, or Virtual Machines.

... and secondly there is actually a BUG in the FreeBSD 12.0 causing even more crazy issues. So yes, disable it and move on.

Go to GUI -> Network -> Interfaces -> edit the interface which is used as a parent for Jails and check Disable Hardware Offloading . Confirm the warning and save (confirm test, save, save...).
By doing so you will disable following NIC options: RXCSUM, LRO, VLAN_HWTSO, RXCSUM_IPV6

But that is not the whole story. There are additional options which are enabled by default and getting disabled upon first Jail start. So to prevent disabling/enabling them every time Jail start/stops we will just disable them by default.

So go to the interface config again and add -rxcsum -txcsum -rxcsum6 -txcsum6 to the Options field. So at the end it will look like this:

- Now Save, confirm everything and after that reboot the NAS! This is important because if you don't then these four excluded options are not persistent. It will allow you to start Jail w/o interruption BUT if you stop the jail these flags gets enabled again (I guess the state is cached/stored somewhere). After reboot try to stop/start your Jails. It should not interrupt your network.

Details about the interface options ...
ifconfig <interface> (ifconfig igb0) shows these ... note that output might be slightly different. Depends on the NIC capabilities:

- Initial (default NAS w/o Jails runnign)

RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6

- After disabling Hardware Offload (red ones are removed)

RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6

- And after adding the extra options (blue ones are removed)

RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6

So the output of ifconfig is:
- Initial: RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6
- HW Offload disabled: TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,TXCSUM_IPV6
- Further options disabled: VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER

So that's about it ... hope it helps :]

//Note: The second part of the OPs question about not being able to stop the Jail is another/unrelated issue. That one is caused by the custom script

 

[nikinp]

After some hours of hunting the root-cause we figured it out ... So for the future generations bellow is explanation what the (hell) was going on and how to fix/overcome that annoying behavior.
...
So that's about it ... hope it helps :]

//Note: The second part of the OPs question about not being able to stop the Jail is another/unrelated issue. That one is caused by the custom script

Thanks for yours hours of awesome work and tenacity!