POLL - k8s Apps egress SNAT

[ian351c]

Hello all!

I am hoping to gauge the community's interest in this suggestion I opened with the iX team (NAS-115748). The idea here is that today, traffic that is sourced from any Apps gets SNATed to the IP address assigned to the primary interface of the NAS. Once out on the LAN, it is impossible for network devices (proxies, routers, firewalls, etc.) to tell if traffic is coming from the NAS itself (e.g. syslog or NTP) or if it is coming from an App. I would like the ability to differentiate that traffic so that I can treat traffic from the NAS differently than I do traffic from any Apps. This can be accomplished by SNATing traffic that is sourced from the Apps networks going outside the NAS to the Node IP assigned under Apps > Settings > Advanmced Settings > Node IP. It's a fairly minor change (at least from an iptables perspective; I'm not sure what extra logic will be needed for the middleware).

Is anyone else interested in something like this? If so, let me know below and please go to the ticket and vote for it.

Thanks!

 

[truecharts]

This is a very dirty and hacky solution.
Simply put: By doing this iX would put themselves into a position where they become responsible for modifying major portions of kubernetes network behavior in a way experienced kubernetes users won't expect.

The right way to do this would be using a Kubernetes Loadbalancer like MetalLB that can just assign good public IP's to pods.
So instead of unsupported hack, just use pre-made tools that basically do what you want out-of-the-box.

 

[ian351c]

My understanding is that there is no "right" way to manage egress traffic from a K8s cluster. AFAIK K8s was built for inbound traffic and there are no provisions for managing egress traffic sourced from within a cluster without some kind of add-on. Every solution I've seen (even Google's own solutions) require custom setups. If you can point me to documentation for SNATing egress traffic in a standard K8s cluster, I'm happy to read it and make a recommendation based on that standard.

I would also point out that the majority of SCALE's users' at this point (and, I would imagine, most of iX's enterprise clients in general) are not, and don't want to be "experienced kubernetes users". They just want Apps to work. And since there is no standard egress traffic management process for K8s at this point (again, I'm happy to be wrong about that) I would say that managing that traffic with iptables is just as valid as standing up an entire load balancing solution and managing that on a per App basis.

 

[truecharts]

"My understanding is that there is no "right" way to manage egress traffic from a K8s cluster"
There is actually: Network Policies, but that's more of a preventative rather than a tagging apprauch.

"I would also point out that the majority of SCALE's users' at this point (and, I would imagine, most of iX's enterprise clients in general) are not, and don't want to be "experienced kubernetes users"."

Sorry, but that's the platform iX wants to build on.
It's pretty clear from current progress that iX does not want to DIY with it too much but rather prefers to implement somewhat existing solutions.

 

[truecharts]

About the addon part:
This is pretty normal with kubernetes... Even the current SCALE network setup is just iX-selected addons...

About these, this is what Calico has to say about it for example:

Kubernetes egress | Calico Documentation

Learn why you should restrict egress traffic and how to do it.

projectcalico.docs.tigera.io

 

[truecharts]

However, iX uses Kube-Router.
If you want this you will need to look more into this ticket:

Static Egress IP · Issue #434 · cloudnativelabs/kube-router

It would be useful if kube-router supported making egress pod traffic appear as a specified static IP. Normally egress traffic gets a SNAT to appear as the node IP when it leaves the cluster. Inste...

github.com

As soon as such a feature is implemented in the network setup SCALE uses, iX can use it.