lagg interface missing after upgrade to 12.0-U8

[nycvelo]

Greetings. After upgrading from 12.0-U7 to 12.0-U8, a TrueNAS system is unreachable over the network because it doesn't appear to have a lagg interface post-upgrade.

I have serial console access (this box is in another city), and ifconfig shows the two lagg component interfaces, igb0 and ibg1, but no lagg0, as before. And I don't see any lagg definition any of the /etc/rc* files.

Manual creation of the lagg interface at the command line is possible, but (a) that won't survive a reboot and (b) the TrueNAS GUI still isn't reachable, even after restarting middlewared and nginx.

Thanks in advance for clues on getting this system back on the network.

 

[Samuel Tai]

Use the console menu to recreate your lagg0 interface. This will survive reboots.

 

[nycvelo]

Thanks for this. There’s no TrueNAS console menu via the serial console, just a root prompt.

Also, after I manually created the lagg0 interface and commanded it to be up, it still shows “no carrier” even though its components igb0 and igb1 show active status. So nothing is reachable over the network.

 

[Samuel Tai]

Try netcli to bring up the console menu.

 

[Kris Moore]

You can run /etc/netcli to bring up the console menu on the serial / root prompt.

 

[nycvelo]

Thanks Samuel and Kris. "/etc/netcli" does indeed call up the console menu.

Destroyed lagg0 from the menu OK.

However, when going to create lagg0, that fails with either failover or lacp (which I'd previously used) with the message "All interfaces are already allocated to LAGG or VLAN, cannot proceed."

Dropping to the command line and running ifconfig shows igb0 and igb1 still have "description: member of lagg0". I thought BSD ifconfig description is just a label as it is with Cisco gear but I have not tried "ifconfig -descr" in the shell, as I don't want to do anything TrueNAS doesn't know about. I did try "sh /etc/nestart" but that didn't change anything.

 

[Samuel Tai]

Try ifconfig igb0 -laggport lagg0 to clear the lagg membership per member.

 

[nycvelo]

Sorry, but that returns:

# ifconfig igb0 -laggport lagg0
ifconfig: SIOCSLAGGDELPORT: Invalid argument

# ifconfig lagg0
ifconfig: interface lagg0 does not exist

 

[Samuel Tai]

How about just ifconfig igb0 -laggport?

 

[Patrick M. Hausen]

@Samuel Tai AFAIK the laggport/-laggport option is applicable to lagg interfaces, only. So (without having tried, sorry), that would be: ifconfig lagg0 -laggport igb0.

 

[nycvelo]

That returns the message ifconfig: '-laggport' requires argument

I tried manually recreating lagg0 and repopulating it with igb0 and igb1. This time, working from the netcli shell, it worked: The lagg0 link showed active, and after manually adding a default route it's reachable over the network. Further, the lagg0 interface shows up in the GUI.

However, after a reboot we are back to the same situation as before: No lagg0 interface.

I just want this to work as it has with versions prior to 12.0-U8. How best to achieve that?

 

[Samuel Tai]

Don't use the netcli shell (option 9), but rather option 2 to create the lagg0. Likewise, option 4 for the default route, and option 1 to add the IP information to the lagg0.

 

[nycvelo]

Sorry, we're in a loop here. I can't use option 2 because that returns the All interfaces are already allocated to LAGG or VLAN, cannot proceed message, as above.

I again tried using option 9 to manually create lagg0 and adding a default route. That worked again, but when I went to option 2 to destroy the lagg it returns No lagg interfaces configured. Then, going back to option 9, it shows lagg0 as up.

I believe there's a disconnect here between what I'm doing in the shell (won't survive reboots) and what TrueNAS writes to a config file (will survive reboots, but I can't access it).

 

[Samuel Tai]

You want to use option 9 to remove both interfaces from lagg0, and then delete the default route and lagg interface. Then use the other options to create the permanent setup that will survive reboots. I apologize if I didn't make this clear earlier.

 

[nycvelo]

No worries, but this is exactly what I've tried. This is with option 9 from netcli:

Code:

root@tafi[~]#  ifconfig lagg0 -laggport igb0
root@tafi[~]#  ifconfig lagg0 -laggport igb1
root@tafi[~]# route delete default
delete net default
root@tafi[~]# ifconfig lagg0 destroy
root@tafi[~]# ifconfig lagg0
ifconfig: interface lagg0 does not exist
root@tafi[~]# ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    ..

Then this is with option 2 from netcli:

Code:

1) Create Link Aggregation
2) Delete Link Aggregation
Enter an option from 1-2 (enter q to quit): 1
1) failover
2) lacp
3) loadbalance
4) roundrobin
5) none
Select a lagg protocol (q to quit): 2
All interfaces are already allocated to LAGG or VLAN, cannot proceed.

Going back to option 9, ifconfig shows no lagg0 interface and no default route, but does show those description labels for igb0 and igb1.

 

[Samuel Tai]

OK, try this:

1. From option 9, run midclt call bootenv.activate "12.0-U7", and then reboot. This will revert you back to your working 12.0-U7 config.
2. When you're back in 12.0-U7, redo your networking to just igb0, with no LAGG.
3. Reapply the upgrade to 12.0-U8.
4. After you're back up in 12.0-U8, use the netcli options to rebuild your LAGG.

 

[nycvelo]

Thanks. That could have gone better:

Code:

root@tafi[~]# midclt call bootenv.activate "12.0-U7"
Call timeout

 

[Samuel Tai]

My apologies. Try midclt call bootenv.activate 12.0-U7, no quotes.

 

[nycvelo]

Thanks all for your help. Samuel's downgrade/upgrade method worked, but (a) it corrupted one of the two USB sticks I'm using as boot media and (b) after the upgrade the system would not reliably establish the lagg0 interface. I suspect media corruption could be the culprit here.

I mentioned earlier this system is in another city. It's back up and running now with one interface, and I'm going to just leave it as is. I already have some NVMe SSDs to replace the USB sticks, and will try to travel to the other city as soon as possible. Thanks again.