TrueNAS SCALE - Networking between kubernetes and VM

[JoFont]

Hello, good afternoon.

I've currently started playing around with SCALE to see the future possibilities. I'm very much a layman in regards to sysadmin (though I'm a developer) so please bear with me if I don't understand some of this topics. I'm currently running Nginx Proxy Manager (but could be any for this matter) with the built int kubernetes environment. I also have some services scattered along VMs that I don't really know how to use with the current Containers GUI.

The problem seems that Nginx returns 502 when proxying services inside the VMs, this seems to me to be because the docker network can't talk directly to the VM's Mac Taps (which I only kind of understand what they are).

Can someone point me if this is the case and how would I go about solving this problem?

Thank you very much in advance.
Stay safe.
All the best,
Diogo Marques

 

[ornias]

I'm currently running Nginx Proxy Manager (but could be any for this matter) with the built int kubernetes environment.

People should start using K8S in how it is supposed to be used: with Ingresses. Nginx Proxy Manager is not very k8s aware and a very non-standard solution. It's not even made to work with K8S at all and as soon as clustering is introduced won't even work correctly.

The problem seems that Nginx returns 502 when proxying services inside the VMs, this seems to me to be because the docker network can't talk directly to the VM's Mac Taps (which I only kind of understand what they are).

Yeah without hostnetworking and a lot of tweaking to get these hacky setups running, this is going to be a pain. I don't think it's wise to even start assisting you in doing this, because this is not a good way of doing it in general: Using all sorts of non-standard hacky solutions on-top of eachother when there are already existing solutions that are made to do what you want them to do.

You can do this just fine with external K8S services, I've added that feature to TrueCharts for our 21.04 release, as you (just like ingress) can't deploy it from the stock/official catalog.

 

[JoFont]

Hello Ornias, how are you?

Well, what a coincidence, one of the services inside VMs that I need to proxy is Appwrite, which I made suggestion for on TrueCharts.
Yes, I guess I could use the reverse on the external VPS running the status page for my services. I've seend that TreuCharts offers the reverse proxy feature, though I didn't use it .

But as you said, yes, I'm probably using this very wrong. I'm used to spinning Compute Engine instances, and everything just working.

 

[brando56894]

Once I managed to get SCALE fully booted on my system, I attempted to switch everything over from native apps on Arch to Docker containers on SCALE (already had the compose file written to route everything to my nginx reverse proxy and have letsencrypt generate a valid cert for each configured container)...but after a few hours of messing with it, I ran into the same networking issues.

If my docker containers were connected to a custom bridge network I could connect to them all from an external source (ex. my laptop), but a hosted VM was unable to access them, and vice versa, even though the VM was accessible externally as well. I remember seeing something about this awhile ago in Arch when messing around with docker and VMs, but it looks like the necessary sysctl variables were already set (something with packet filter?) and I couldn't see anything being rejected by iptables but then again I've had a bunch of issues with it and docker in the past on other OSes. The reason for the VM was just because the TrueNAS webUI takes over ports 80 and 443, and obviously my nginx container couldn't bind to those as well. I have 2 NICs but couldn't figure out how to assign the other one to just the nginx container, so I decided to throw that (and Ombi, which kept giving me odd SQlite errors) into a VM.

Those issues, combined with the issues with my HBA made SCALE too much of a headache to use in it's current state, so back to Arch for another few months :(

 

[ornias]

already had the compose file written

Maybe don't use unsupported things like docker compose? ;-)

If my docker containers were connected to a custom bridge network

You are talking about compose (an unsupported hack), while OP is talking docker container on SCALE (which run on kubernetes). both of them use a totally different network stack. So different that your post is basically off-topic.

but it looks like the necessary sysctl variables were already set (something with packet filter?) and I couldn't see anything being rejected by iptables but then again I've had a bunch of issues with it and docker in the past on other OSes.

the k3s stack (default SCALE docker settings) by default (which is what concerns OP) doesn't utilise iptables at all.

The reason for the VM was just because the TrueNAS webUI takes over ports 80 and 443, and obviously my nginx container couldn't bind to those as well

You don;t have to stick to those ports though, they can easily be changed. Going the VM route because of them seems like a bit heavy-handed.

Those issues, combined with the issues with my HBA made SCALE too much of a headache to use in it's current state, so back to Arch for another few months :(

I find it annoying to start blaming the "state" of the product, when you are hacking together something that isn't supported by the product (docker compose).

 

[JoFont]

I can understand both of you guys points, on one side I understand Ornias point of doing things right and that you should at least understand the basics to get thing working and be able to troubleshoot but, on the other side I think Brando makes a good point that, I see TrueNAS SCALE as an all in one solution that takes the hassle away from some of this common situations that someone with experience would lead with no problem, and, currently, I feel like some of these edge cases are not very well explained at least. My issue stems from not clearly understanding how to have multiple VM's running multiple services, have them talk to each other, to the outside and with the built in k8s.

Imagine that is something that doesn't involve docker at all, I have 2 VM's with one service in each one and need to make requests from one to another. It just isn't clear how one would go about this. This is easy with any VPS services because you have unice ipv4 addresses but with DDNS to the mixture, I'm clueless as to how this is done.

To reiterate, I also understand that my use case is probably not the most common one. I'm trying to use TrueNAS SCALE as an on prems VPS and container provider and intent to use it for development, I'm net really using the NAS part of the system as much xD

 

[ornias]

@JoFont What I was talking about to @brando, is the fact Docker-Compose (which he is using) is not supported at all and is not planned to ever be supported. It's not an edgecase, he installed something into SCALE that isn't supposed to be there and is telling us it didn't work.

it's totally different from your problem.

your problem is an edgecase that isn't documented or flushed-out well, his case is he hacked something together that isn't even supposed to be working to begin with.

 

[JoFont]

Ah! I see. Well seems like the solution is for my problem is to wait for more documentation...
Thank you very much for your help lately @ornias, I've checked reddit and yep, you answered my problems there as well. Thanks :)

 

[ornias]

@JoFont You could make a jira ticket towards the documentation...
Instead of "NAS" you have to select the "DOCS" product on jira :)

Something like:
"Documentation on how to connect VM's to Containers on SCALE"

 

[JoFont]

@JoFont You could make a jira ticket towards the documentation...
Instead of "NAS" you have to select the "DOCS" product on jira :)

Something like:
"Documentation on how to connect VM's to Containers on SCALE"

Thanks for the suggestion: Issue has been created, could link to any Epic for lack of permissions.

https://jira.ixsystems.com/browse/NAS-110020

 

[ornias]

Uhmm... thats not a DOCS ticket, thats a NAS ticket.... >.<

https://jira.ixsystems.com/projects/DOCS/issues

 

[JoFont]

Uhmm... thats not a DOCS ticket, thats a NAS ticket.... >.<

https://jira.ixsystems.com/projects/DOCS/issues

My mistake, have created issue in correct project.

https://jira.ixsystems.com/browse/DOCS-1964

 

[ornias]

@JoFont Awesome! :)

 

[emollusion]

Does anyone know of any progress here?
Looking in to the ticket NAS-110020 there has not been much progress and as discussions in discord stated as well this is not a supported "feature".

 

[sretalla]

If you're doing networking via a bridge, you should be able to get an app to find a VM and vice-versa.

Accessing NAS From a VM

Provides instructions on how to create a bridge interface for the VM and provides Linux and Windows examples.

www.truenas.com

Can you confirm that you are not able to get that to work?

 

[emollusion]

I can get that part to work with the bridge as intended between TrueNAS and the VM, but not between k8s and vm as it was configured.
I have now tried this with different apps and configs to get it to work and it came down to attaching a network interface (a bridge since that is what is set up) to the application so it get's an IP.
This is a limitation since not all apps has this advanced networking capability.
After that I provided a route for the app for it to be able to route via a gateway and then back to the VM on the NAS.

The problem that is appearing for me now is that I don't seem to be able to get it to work all the way with a proxy this way in the case of getting it to connect to the NAS. It seems to connect to TrueNAS but it just keeps spinning with the message that I need to make sure that TrueNAS is up and that it has network connectivity.