VNET failures blocking jail creation / plugin installation

[Teeps]

Hello,

I have a very vanilla truenas box that I recently built on the latest version of truenas 12. I've just updated it to 13.0-U1.1 and can't seem to create any jails or install any plugins that rely on VNET. I didn't try creating any on version 12.

I attempt to install plex with a static IP and get the following error:

Error: plex had a failure Exception: RuntimeError Message: Stopped plex due to VNET failure Partial plugin destroyed

In /var/log/messages, I see this happen:

Code:

Jul 30 10:50:20 truenas kernel: epair0a: Ethernet address: 02:xx:xx:xx:7d:0a                                                                                                            
Jul 30 10:50:20 truenas kernel: epair0b: Ethernet address: 02:xx:xx:xx:7d:0b                                                                                                            
Jul 30 10:50:20 truenas kernel: epair0a: link state changed to UP                                                                                                                        
Jul 30 10:50:20 truenas kernel: epair0b: link state changed to UP                                                                                                                        
Jul 30 10:50:20 truenas kernel: epair0a: changing name to 'vnet0.8'                                                                                                                      
Jul 30 10:50:20 truenas kernel: igb0: link state changed to DOWN                                                                                                                        
Jul 30 10:50:21 truenas kernel: vnet0.8: promiscuous mode enabled                                                                                                                        
Jul 30 10:50:21 truenas kernel: vnet0.8: link state changed to DOWN                                                                                                                      
Jul 30 10:50:21 truenas kernel: epair0b: link state changed to DOWN                                                                                                                      
Jul 30 10:50:25 truenas kernel: igb0: link state changed to UP

I was able to create a jail with the VNET box checked (this box isn't even available from the plugins page), but I can't start the jail due to the same VNET failure.

My network config

Code:

truenas% ifconfig                                                                                                                                                                        
ix0: flags=8863 metric 0 mtu 9000                                                                                                                
        options=4e53fbb  
        ether xx:xx:xx:xx:xx:c8                                                                                                                                                          
        inet ip.add.re.ss netmask 0xffffff00 broadcast ip.add.re.255                                                                                                                  
        media: Ethernet autoselect (10Gbase-Twinax )                                                                                                        
        status: active                                                                                                                                                                  
        nd6 options=9                                                                                                                                            
igb0: flags=8963 metric 0 mtu 1500                                                                                                      
        description: igb0                                                                                                                                                                
        options=4e503bb                                    
        ether xx:xx:xx:xx:xx:68                                                                                                                                                          
        inet ip.add.re.ss netmask 0xffffff00 broadcast ip.add.re.255                                                                                                                    
        media: Ethernet autoselect (1000baseT )                                                                                                                            
        status: active                                                                                                                                                                  
        nd6 options=9                                                                                                                                            
igb1: flags=8822 metric 0 mtu 1500                                                                                                                          
        options=4e507bb                                
        ether xx:xx:xx:xx:xx:69                                                                                                                                                          
        media: Ethernet autoselect                                                                                                                                                      
        status: no carrier                                                                                                                                                              
        nd6 options=9                                                                                                                                            
lo0: flags=8049 metric 0 mtu 16384                                                                                                                        
        options=680003                                                                                                                  
        inet6 ::1 prefixlen 128                                                                                                                                                          
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4                                                                                                                                      
        inet 127.0.0.1 netmask 0xff000000                                                                                                                                                
        groups: lo                                                                                                                                                                      
        nd6 options=21                                                                                                                                        
pflog0: flags=0<> metric 0 mtu 33160                                                                                                                                                    
        groups: pflog                                                                                                                                                                    
bridge0: flags=8843 metric 0 mtu 1500                                                                                                            
        ether xx:xx:xx:xx:xx:9f                                                                                                                                                          
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15                                                                                                                      
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                                                                                                                        
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                                                                                                                        
        member: igb0 flags=143                                                                                                                      
                ifmaxaddr 0 port 2 priority 128 path cost 20000                                                                                                                          
        groups: bridge                                                                                                                                                                  
        nd6 options=9

 

[Samuel Tai]

By any chance, do you have igb0 and ix0 on the same subnet? The vnet0.8 interface is a member of bridge0, as is igb0, and this smells like bridge spanning tree shutting down igb0.

Having 2 interfaces on the same subnet outside of a LAGG is not supported and will result in communications difficulty like this. This is because an Ethernet network must be loop-free, to prevent broadcast packets from circulating forever. If you have 2 interfaces on the same network, you've created a loop.

 

[Teeps]

By any chance, do you have igb0 and ix0 on the same subnet? The vnet0.8 interface is a member of bridge0, as is igb0, and this smells like bridge spanning tree shutting down igb0.

Having 2 interfaces on the same subnet outside of a LAGG is not supported and will result in communications difficulty like this. This is because an Ethernet network must be loop-free, to prevent broadcast packets from circulating forever. If you have 2 interfaces on the same network, you've created a loop.

Hi Samuel, thanks for the quick response and I can confirm they are on different subnets. For further details, I only use ix0 to connect directly to my primary workstation for a 10Gbps SMB share. igb0 is the only NIC connected to a switch and the home LAN.

Only after I posted, did the forums suggest this very informative thread: https://www.truenas.com/community/t...o-lan-has-very-slow-access-to-lan-wan.102384/

I tried to follow @Patrick M. Hausen's advice and reboot the server to clear out the automatic bridge, then move my IP address from igb0 to a manually created bridge (which still immediately disappears from the GUI, funny enough), but this didn't fix the issue. I also disabled hardware offloading on igb0 per that thread.

Code:

Jul 30 11:41:59 truenas kernel: epair0a: Ethernet address: 02:7c:c1:45:17:0a                                                                                                            
Jul 30 11:41:59 truenas kernel: epair0b: Ethernet address: 02:7c:c1:45:17:0b                                                                                                            
Jul 30 11:41:59 truenas kernel: epair0a: link state changed to UP                                                                                                                       
Jul 30 11:41:59 truenas kernel: epair0b: link state changed to UP                                                                                                                       
Jul 30 11:41:59 truenas kernel: epair0a: changing name to 'vnet0.5'                                                                                                                     
Jul 30 11:41:59 truenas kernel: vnet0.5: promiscuous mode enabled                                                                                                                       
Jul 30 11:41:59 truenas kernel: vnet0.5: link state changed to DOWN                                                                                                                     
Jul 30 11:41:59 truenas kernel: epair0b: link state changed to DOWN                                                                                                                     
Jul 30 11:42:00 truenas kernel: igb0: link state changed to DOWN                                                                                                                        
Jul 30 11:42:04 truenas kernel: igb0: link state changed to UP  

Code:

truenas% ifconfig                                                                                                                                                                       
ix0: flags=8863 metric 0 mtu 9000                                                                                                               
        options=4e53fbb 
        ether xx:xx:xx:xx:xx:c8                                                                                                                                                         
        inet xx.xx.123.1 netmask 0xffffff00 broadcast xx.xx.123.255                                                                                                                         media: Ethernet autoselect (10Gbase-Twinax )                                                                                                       
        status: active                                                                                                                                                                  
        nd6 options=9                                                                                                                                            
igb0: flags=8963 metric 0 mtu 1500                                                                                                      
        description: igb0                                                                                                                                                               
        options=c103ba                                                                        
        ether xx:xx:xx:xx:xx:68                                                                                                                                                         
        media: Ethernet autoselect (1000baseT )                                                                                                                            
        status: active                                                                                                                                                                  
        nd6 options=9                                                                                                                                            
igb1: flags=8822 metric 0 mtu 1500                                                                                                                         
        options=4e507bb                               
        ether xx:xx:xx:xx:xx:69                                                                                                                                                         
        media: Ethernet autoselect                                                                                                                                                      
        status: no carrier                                                                                                                                                              
        nd6 options=9                                                                                                                                            
lo0: flags=8049 metric 0 mtu 16384                                                                                                                       
        options=680003                                                                                                                 
        inet6 ::1 prefixlen 128                                                                                                                                                         
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4                                                                                                                                      
        inet 127.0.0.1 netmask 0xff000000                                                                                                                                               
        groups: lo                                                                                                                                                                      
        nd6 options=21                                                                                                                                       
pflog0: flags=0<> metric 0 mtu 33160                                                                                                                                                    
        groups: pflog                                                                                                                                                                   
bridge0: flags=8843 metric 0 mtu 1500                                                                                                           
        ether xx:xx:xx:xx:xx:9f                                                                                                                                                         
        inet xx.xx.23.1 netmask 0xffffff00 broadcast xx.xx.23.255                                                                                                                    
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15                                                                                                                     
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                                                                                                                        
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                                                                                                                        
        member: igb0 flags=143                                                                                                                      
                ifmaxaddr 0 port 2 priority 128 path cost 20000                                                                                                                         
        groups: bridge                                                                                                                                                                  
        nd6 options=9

EDIT: strange, the GUI-created bridge didn't survive a reboot either..

 

[Teeps]

Finally managed to make it work. Quite a journey for a simple static IP address. Perhaps someone can recommend an easier method to others who read this.

Created a pre-init script here to re-create the bridge on each boot:

Code:

# cat /root/scripts/makebridge.sh                                                                                                         
#!/bin/bash                                                                                                                                                                              
ifconfig bridge create                                                                                                                                                             
ifconfig bridge0 addm igb0 up                                                                                                                                                            
ifconfig igb0 up                                                                                                                                                                         
ifconfig bridge0 inet 172.0.123.123/24

and followed some of the tips from this post to set the jail interface to vnet0:bridge0

vnet jail bridged to LAN has very slow access to LAN/WAN

I am recovering from a crashed TrueNAS system where I lost my jails. This is a long story and really not important. This is my primary home server and hosts many local LAN shares and when everything was working, it runs 5 jails. Currently, I have one jail setup that runs a webserver that...

www.truenas.com

Easily my favorite way to spend a Saturday...

Thanks to @Ray DeMoss and @Patrick M. Hausen