App Service fails to lookup bridge interface during boot after upgrade to TNS 23.10.1

[MainCranium]

After upgrading from 23.10.0 to 23.10.1, I'm experiencing the following behavior:

Upon startup, Applications fail to load, and I receive this error:

• Failed to configure kubernetes cluster for Applications: Unable to lookup configured interfaces: br01

I used a bridge interface a year or so ago when I setup TNS for the first time to get around an issue with VMs being unable to access the TrueNAS storage. It would now seem that the Application Service is initializing during boot faster than the bridge interface can.

I'm able to get around this issue once the server is operational by toggling "Enable GPU support" off and on again in the Advanced Settings of the Apps page. It seems that this forces the service to restart and successfully lookup the now-initialized bridge interface.

Has anyone else experienced this? Do we know if the bridge interface is still necessary to get around this old issue I'm referring to? (Here's a related forum post. The original article I referenced about it at the time is no longer available.) I'm wondering how best to proceed here.

 

[choppatron]

I have the same issue with this upgrade. The Toggle GPU Support workaround worked for me

 

[David M. Brine Jr.]

I have the same issue but toggling didn't work for me. Is there a way to restart the service from the CLI by chance?

 

[TuKo]

Also have the same issue. Your workaround made the job for me. Thanks !

 

[sapster77]

Same issue here. I can't, however, toggle the GPU support as I get an error on that page next to "enable host path safety checks" saying "Field was not expected". Because og that error I can't save en thereby toggle the GPU support.
So I am also interested in an "CLI version" of this fix...? Or anything else for that matter....

 

[brahmy]

I get an error on that page next to "enable host path safety checks" saying "Field was not expected". Because og that error I can't save en thereby toggle the GPU support.

Try clearing your cache and cookies then re-signing into TrueNAS. I think the Host Path Safety Checks option is now gone and you're seeing a cached version of the page - that's what happened to me.

Just adding, I had the same issue as OP & posters and the untoggle/toggle of the Toggle GPU Support was able to get me up and running. I have not yet rebooted to see if the toggle is needed every time.

 

[sapster77]

Try clearing your cache and cookies then re-signing into TrueNAS. I think the Host Path Safety Checks option is now gone and you're seeing a cached version of the page - that's what happened to me.

Just adding, I had the same issue as OP & posters and the untoggle/toggle of the Toggle GPU Support was able to get me up and running. I have not yet rebooted to see if the toggle is needed every time.

Thanks for the reply - that’s sounds very plausible. I’ll test it out. Yesterday I found a CLI command to Toggle GPU support which also fixed the kubernetes cluster issue for me and haven’t either tested whether the fix survives a reboot;-)

 

[sapster77]

Thanks for the reply - that’s sounds very plausible. I’ll test it out. Yesterday I found a CLI command to Toggle GPU support which also fixed the kubernetes cluster issue for me and haven’t either tested whether the fix survives a reboot;-)

...and just for others stumbling into this: the cli command to manipulate the gpu is here:

Code:

midclt call -job kubernetes.update '{ "node_ip": "10.220.16.82", "route_v4_interface": null, "route_v4_gateway": null,   "configure_gpus": true, "servicelb": true, "cluster_cidr": "172.16.0.0/16", "service_cidr": "172.17.0.0/16", "cluster_dns_ip": "172.17.0.10", "force": false }'

 

[ra4design]

I had the same problem. Originally, I thought this was because of the two IP addresses on the br0 interface, as I posted here: https://www.truenas.com/community/threads/kubernetes-service-is-not-running.107377/post-797886

Since the problem according to this post is due to k3s service starting before the bridge interface is up, I created an override for the k3s service: /etc/systemd/system/k3s.service.d/override.conf:

Code:

[Unit]
After=network-online.target

This was not sufficient and I actually had to specify the bridge interface (br0) in my case

Code:

[Unit]
After=network.target
Requires=br0.device

I still had to do the Toggle GPU thing, because the interface does not reflect the actual status of the k3s service.

After the reboot, the k3s is running without issues.

 

[MainCranium]

I'm glad that this post was able to help others. Any chance anyone here has seen anything in the form of an official response or remedy? I'm hoping it's resolved in the next patch.

 

[Black_Duck]

I couldn't see a reported bug in JIRA, so I generated one.
https://ixsystems.atlassian.net/browse/NAS-126078

 

[Black_Duck]

There's a hotfix release, 23.10.1.1 in the pipeline that should resolve this issue.
See https://ixsystems.atlassian.net/browse/NAS-125932

 

[MainCranium]

There's a hotfix release, 23.10.1.1 in the pipeline that should resolve this issue.
See https://ixsystems.atlassian.net/browse/NAS-125932

Hey, thanks for letting me know! And for cluing me into the the bug submission process. I'm new around here and it honestly didn't occur to me. I will do so myself in the future.

Thanks again!

 

[zsw12abc]

I had the same problem. Originally, I thought this was because of the two IP addresses on the br0 interface, as I posted here: https://www.truenas.com/community/threads/kubernetes-service-is-not-running.107377/post-797886

Since the problem according to this post is due to k3s service starting before the bridge interface is up, I created an override for the k3s service: /etc/systemd/system/k3s.service.d/override.conf:

Code:

[Unit]
After=network-online.target

This was not sufficient and I actually had to specify the bridge interface (br0) in my case

Code:

[Unit]
After=network.target
Requires=br0.device

I still had to do the Toggle GPU thing, because the interface does not reflect the actual status of the k3s service.

After the reboot, the k3s is running without issues.

Hey mate,
i face the issue today,
what does Toggle GPU mean?

 

[Belperite]

Sadly I'm still having to do the toggle GPU thing in 23.10.1.1 :(

 

[Rayson]

The problem still exists after the upgarde. :(

 

[KristofferLindberg]

Hi

Just upgraded to Cobia from Bluefin today and experienced the same problem. I have a bridge setup for my vm running inside truenas and with the latest cobia release the problem still persist. Seems as if the workaround with toggle gpu worked for me also, its just going up as we speak. Thank you for finding this short time solution and lets hope they include a fix soon!!!!

 

[MainCranium]

Can confirm issue still persists in 23.10.1.3.

EDIT: Has anyone tried deleting the bridge interface, rebooting, then recreating the bridge? Looks like someone mentioned that over here: https://ixsystems.atlassian.net/browse/NAS-125932

 

[Belperite]

Can confirm issue still persists in 23.10.1.3.

EDIT: Has anyone tried deleting the bridge interface, rebooting, then recreating the bridge? Looks like someone mentioned that over here: https://ixsystems.atlassian.net/browse/NAS-125932

Hmm, the issue with that is is that my NAS is headless and remote access is via that bridge, so I'd have to mess about getting a monitor and keyboard to it. Was kinda hoping it could be purely be fixed via an update.

 

[thecrownguy]

I waited to do this upgrade but I guess I didnt wait long enough. I have the same issue and I have tried all the fixes. If i disable the bridge my vms cant talk. I hope they get this fixed soon. I guess I can just reboot into the old version for now.