Working Kafka Broker Pod - with a caveat

[msigmond]

Thank you all for the amazing work on TrueNAS Scale!
It is a great product, and I enjoy every step I take with it.

I was experimenting with setting up a Kafka Broker on my single node home server.
I used Confluent's Docker Compose file as a starting point: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html
I setup the first 2 images: zookeeper and broker

Zookeeper Pod Settings:

Broker Pod Settings:

This is working now, I tested it with the kafka-console-producer.sh & the kafka-console-consumer.sh.
There are 2 limitations though, that I highlighted with red on the 2nd screenshot:

1. I have to start Zookeeper first, and then copy it's IP address into the Broker setup. If I restart Zookeeper or the host, new IP will be assigned, hence I will have to update the Broker config.

2. The Broker's setup has a similar issue, but even worse: I have to know the IP of the broker before it is started, and entered into the advertised listener config. The way I do this is to start the broker with a dummy config (expected to fail), copy the IP address, increment the last number of the IP (I know that will be the next assigned IP), shut down the broker, update the config, and start the broker again. And voila: it is working.

I was hoping that pods would have a static hostname, but that is not the case: new hostname is assigned to pods after every restart, such as kafka-ix-chart-84686b656b-wklgx: the last 5 characters are renewed after every restart.

I am wondering if you could help me to solve the above limitations?
I.e. I can think of 3 options:

1. Assign static IP to pods in the host's address space (i.e. 192.168.x.x) - I know how to do this, but feels sub-optimal
2. Assign static IP to pods in the Kubernetes address space (i.e. 172.16.0.x) - I don't know how to do this, but still does not feel optimal
3. Reference the pods with a name that does not change during restart - Feels like the best option, but I don't know how could I achieve this.

Any help is greatly appreciated.

 

[morganL]

Thank you all for the amazing work on TrueNAS Scale!
It is a great product, and I enjoy every step I take with it.

I was experimenting with setting up a Kafka Broker on my single node home server.


I was hoping that pods would have a static hostname, but that is not the case: new hostname is assigned to pods after every restart, such as kafka-ix-chart-84686b656b-wklgx: the last 5 characters are renewed after every restart.

I am wondering if you could help me to solve the above limitations?
I.e. I can think of 3 options:

1. Assign static IP to pods in the host's address space (i.e. 192.168.x.x) - I know how to do this, but feels sub-optimal
2. Assign static IP to pods in the Kubernetes address space (i.e. 172.16.0.x) - I don't know how to do this, but still does not feel optimal
3. Reference the pods with a name that does not change during restart - Feels like the best option, but I don't know how could I achieve this.

Any help is greatly appreciated.

if you don't get any answers, please make a suggestion in "report a bug'..
@waqarahmed do you have any suggestions?

 

[truecharts]

@morganL Maybe we can work out here...

1. Kubernetes internal IP adresses will never be static, ever. The same goes for Kubernetes internal serviceIP's.
You should use the kubernetes internal service DNS-name as announcement instead (which are stable) or expose the port on the host and connect on the host side of things.
2. Brokerlistener does not seem to be limited by IP, so use the stable internal service DNS name instead.


Though we cannot speak on how to do this via the "Launch Docker" system, because some of the above settings are (un)conveniently abstracted away. But the above should be plausible with our Custom-App.

To be fair, for things like this building an App seems more suitable than butchering Launch-Docker or Custom-App

 

[morganL]

@morganL Maybe we can work out here...

1. Kubernetes internal IP adresses will never be static, ever. The same goes for Kubernetes internal serviceIP's.
You should use the kubernetes internal service DNS-name as announcement instead (which are stable) or expose the port on the host and connect on the host side of things.
2. Brokerlistener does not seem to be limited by IP, so use the stable internal service DNS name instead.


Though we cannot speak on how to do this via the "Launch Docker" system, because some of the above settings are (un)conveniently abstracted away. But the above should be plausible with our Custom-App.

To be fair, for things like this building an App seems more suitable than butchering Launch-Docker or Custom-App

Is there any web documentation on how to use the Custom-App.?
Providing a link or how-to would be useful to the Community.

 

[truecharts]

Is there any web documentation on how to use the Custom-App.?
Providing a link or how-to would be useful to the Community.

Custom App is not much different from out Other Apps in structure. So most of our guides should also apply for Custom App.

However, the target audience is users with previous Kubernetes Experience, not the average user, so there is no additional documentation byond our quick-start guides and other docs.

We do not expect the average user to be able to get a container running using Custom-App, but we do refer to it from time-to-time, as it's currently the only GUI based solution to get some things done.

Links should not be needed because everything is just part of our Catalog, it's not a seperate project or something like that.
Also: If someone really need a link to it in the list of Apps (that does not contain anything especially of relevance), we can safely assume said person is not the target audience for Custom-App.

TLDR:
It's not a drop-in replacement for Launch-Docker, it's an alternative for expert kubernetes users with specific usecases.

 

[msigmond]

@morganL Maybe we can work out here...

1. Kubernetes internal IP adresses will never be static, ever. The same goes for Kubernetes internal serviceIP's.
You should use the kubernetes internal service DNS-name as announcement instead (which are stable) or expose the port on the host and connect on the host side of things.
2. Brokerlistener does not seem to be limited by IP, so use the stable internal service DNS name instead.


Though we cannot speak on how to do this via the "Launch Docker" system, because some of the above settings are (un)conveniently abstracted away. But the above should be plausible with our Custom-App.

To be fair, for things like this building an App seems more suitable than butchering Launch-Docker or Custom-App

Thank you for your help.
Based on this answer I was able to fix my setup.

Here is how I did it:
First I queried all the services:

Code:

marton@truenas:~$ sudo k3s kubectl get services --all-namespaces
NAMESPACE       NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
default         kubernetes            ClusterIP   172.17.0.1       <none>        443/TCP                         57d
kube-system     kube-dns              ClusterIP   172.17.0.10      <none>        53/UDP,53/TCP,9153/TCP          57d
ix-aria2        aria2-ix-chart        NodePort    172.17.69.0      <none>        8080:9200/TCP                   48d
ix-embyserver   embyserver-ix-chart   NodePort    172.17.146.195   <none>        8096:9096/TCP,8920:9920/TCP     57d
ix-zookeeper    zookeeper-ix-chart    NodePort    172.17.239.195   <none>        2181:22181/TCP                  10d
ix-kafka        kafka-ix-chart        NodePort    172.17.226.123   <none>        29092:29092/TCP,9092:9092/TCP   10d

Based on the related Kubernetes documentation the internal service DNS names are:
- zookeeper-ix-chart.ix-zookeeper
- kafka-ix-chart.ix-kafka

When I updated my Kafka app, the Zookeeper connectivity was working fine, but the advertised listener was not.
After some further reading and experiment I realized that localhost will work just fine.
So my final setup looks like this:

Finally, I also found the following article, that I overlooked first: https://docs.confluent.io/operator/current/co-quickstart.html#co-quickstart
This might be an easier way to achieve an even better Kafka setup.

Congratulations on the TrueNAS SCALE 22.02.0 first stable release.

 

[waqarahmed]

@msigmond nice!
We can look to exposing hostname/subdomain field for pods as well for launch docker image deployments - it's not ideal but can help with providing a DNS address for the pod it appears ( https://kubernetes.io/docs/concepts...-service/#pod-s-hostname-and-subdomain-fields ).
However, still the better/standard way to go about it is using services or port forwarding if we use UI terms so that a service is created and is always accessible easily with it's DNS name(s).