Apps can't reach VM on same bridge

[afya]

My Apps can reach bridge/host nic IP (10.0.10.160), gateway(10.0.10.10), LAN device (10.0.10.40) and internet (8.8.8.8)
However, they cannot ping VM (10.0.10.18) that is also attached to the bridge br0
VM can ping NAS host (10.0.10.160)
I cannot figure out why it would work only one way (given ping is actually 2 way)

Network settings:

br0:

Kubernety settings:

• Node IP: 10.0.10.160
• Route v4 Interface: br0
• Route v4 Gateway: 10.0.10.10 (my router)

VM NIC:

 

[morganL]

Which version of SCALE?

Can VM ping gateway and Internet?

 

[afya]

Which version of SCALE?

Can VM ping gateway and Internet?

TrueNAS-SCALE-22.02.4

Yes, VM can ping gateway/internet/host

 

[morganL]

If it previously worked with 22.02.3... its probably a bug.

If not, its probably a configuration issue, but I can't see it. If no-one else can see a solution, please report-a-bug after another day.

 

[Multiply0148]

I have the exact same problem.

When setting the Kubernetes Gateway to the same IP as the Truenas Host, the apps can see and ping the VMs, however the Apps then don't have internet access any more.

So it seems, there is a problem with the gateway logic and kubernetes with TrueNAS Scale.

 

[ctag]

I believe I'm seeing the same thing on my end. Traefik will proxy "apps" but no longer VMs with their own IP addresses.

Spoke with the volunteer support members on Truechart's discord server, and they let me know external services isn't supported. So this may be "not a bug" and traefik just isn't useful except for app-only situations anymore.

It sounds like one user was able to build an escape hatch to nginx+letsencrypt here, but there's no details. https://www.truenas.com/community/t...ates-docker-compose-with-truenas-scale.99848/

 

[ctag]

I was able to reach my VMs after unsetting the Route v4 interface and gateway in Kubernetes advanced settings. No clue why or how.

 

[afya]

I was able to reach my VMs after unsetting the Route v4 interface and gateway in Kubernetes advanced settings. No clue why or how.

Just confirming that this indeed "fixes" the issue. It doesn't quite make sense to me.
I hope this doesn't break other things.

 

[dennisline]

@ctag You are my HERO! thank you!
I was breaking my head over this for a week!
I cant believe I managed to stumble on this.
You are a champion of the people and I love you, I hope you actually see this!!

 

[rs_taylor]

THANK YOU!!! Works great, no idea why and waiting to find something broke but so far all working.
My problem was that my scale apps were able to access the internet and anything on the scale server(other apps or media) BUT had no access to other clients on the same network as the truenas server(192.168.31.78/24)

For example I have IP security cameras(192.168.31.250/24) that are on the same network as TrueNAS server(192.168.31.78/24)
My truenas apps had no access to these camera's so I couldn't monitor them(zoneminder) or even check if they were running (uptime-kuma).
These apps simple couldn't see the camera or any devices on the local network except truenas scale server content and the gateway router.

If I set the kubernetes gateway setting to 192.168.31.78(truenas box) the apps had access to the localnetwork (192.168.31.0/24) but not internet, if I set to 192.168.31.1 (actual router/gateway) they had access to internet but not other hosts on local network.

Removing that gateway+and interface in the in kubernetes advanced setting and my apps have access to the internet, other apps and local devices.