Routes and gateways on Scale

[Aviatorpaal]

TrueNAS-SCALE-22.12.0 on bare metal
Quad Intel NIC
Management interface - enp22s0f0 with static IP 192.168.11.2
Kubernetes "Apps" interface - enp23s0f1 with static IP 192.168.12.2
pfSense firewall

My traffic from the Kubernetes cluster exits from the wrong interface and IP even when the gateway (192.168.12.1) is correctly specified in Kubernetes "Advanced Settings". I am seeing all sorts of strange network behaviour as a result of this.

How do I get Scale to properly route Kubernetes-traffic to my desired gateway using the correct interface for the 192.168.12.0/24-destined traffic? I have added the route manually under "Static Routes" under the "Networking tab" and via the netcli with no changes to the netstat output.

Code:

root@truenas[~]# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.168.11.1    0.0.0.0         UG        0 0          0 enp22s0f0
172.16.0.0      0.0.0.0         255.255.0.0     U         0 0          0 kube-bridge
192.168.11.0    0.0.0.0         255.255.255.0   U         0 0          0 enp22s0f0
192.168.12.0    0.0.0.0         255.255.255.0   U         0 0          0 enp23s0f1

 

[o1982]

App networking on Scale is a dumpster fire right now. Many people want to do the same thing, but most of the posts are ignored/unanswered.

Did you find a solution?

 

[NugentS]

Also traffic, when it does leave the NAS is directed straight to the default gateway which then has to redirect any internal traffic back.
ie traffic leaving via interface 192.168.38.32 destined for 192.168.38.10 is sent to the gateway first (192.168.38.1) which then redirects traffic back to the .10

 

[o1982]

Also traffic, when it does leave the NAS is directed straight to the default gateway which then has to redirect any internal traffic back.
ie traffic leaving via interface 192.168.38.32 destined for 192.168.38.10 is sent to the gateway first (192.168.38.1) which then redirects traffic back to the .10

I thought you had to add a bridge, so that apps can talk to the NAS without actually sending it out a physical NIC (and back again)... or was I mistaken?

 

[Aviatorpaal]

I did not find a solution. I submitted a bug report, but the experts told me I had a L2/L3 problem, not a kernel bug. I thought I understood the OSI, subnetting and routing, but apparently I must have missed something. Both NICs are connected directly to a pfSense gateway, no switches involved.

I removed the static route that was manually added, apparently it´s not necessary (according to the TrueNAS team). Maybe a future version of me will understand why I do not understand this in 2023.

Please comment and fill me in if there´s something obvious I´ve missed. My apps (Plex, Nextcloud, Home Assistant, Unifi) work, even though I do not understand how it´s being routed in/from the kernel.

App networking on Scale is a dumpster fire right now. Many people want to do the same thing, but most of the posts are ignored/unanswered.

Did you find a solution?

 

[Aviatorpaal]

That might be correct. My "bug" is concerning traffic from outside of the TrueNAS Scale host reaching Kubernetes apps

I thought you had to add a bridge, so that apps can talk to the NAS without actually sending it out a physical NIC (and back again)... or was I mistaken?

 

[NugentS]

I was trying to send traffic to a local DNS Server which I discovered was going via the router due to some PBR on the router that was grabbing the traffic and sending it down the wrong port. It does mean that the router will get every packet leaving the kube-router. Apps can talk to apps as they are on the kube network. I never tried app to NAS other than internally.

You need a bridge for VM's to talk to the NAS

 

[o1982]

You need a bridge for VM's to talk to the NAS

Ah, thank you for the info, I thought it was for everything including the apps, but what you say makes sense when I think about it.