Thank you all for the amazing work on TrueNAS Scale!
It is a great product, and I enjoy every step I take with it.
I was experimenting with setting up a Kafka Broker on my single node home server.
I used Confluent's Docker Compose file as a starting point: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html
I setup the first 2 images: zookeeper and broker
Zookeeper Pod Settings:
Broker Pod Settings:
This is working now, I tested it with the kafka-console-producer.sh & the kafka-console-consumer.sh.
There are 2 limitations though, that I highlighted with red on the 2nd screenshot:
1. I have to start Zookeeper first, and then copy it's IP address into the Broker setup. If I restart Zookeeper or the host, new IP will be assigned, hence I will have to update the Broker config.
2. The Broker's setup has a similar issue, but even worse: I have to know the IP of the broker before it is started, and entered into the advertised listener config. The way I do this is to start the broker with a dummy config (expected to fail), copy the IP address, increment the last number of the IP (I know that will be the next assigned IP), shut down the broker, update the config, and start the broker again. And voila: it is working.
I was hoping that pods would have a static hostname, but that is not the case: new hostname is assigned to pods after every restart, such as kafka-ix-chart-84686b656b-wklgx: the last 5 characters are renewed after every restart.
I am wondering if you could help me to solve the above limitations?
I.e. I can think of 3 options:
Any help is greatly appreciated.
It is a great product, and I enjoy every step I take with it.
I was experimenting with setting up a Kafka Broker on my single node home server.
I used Confluent's Docker Compose file as a starting point: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html
I setup the first 2 images: zookeeper and broker
Zookeeper Pod Settings:
Broker Pod Settings:
This is working now, I tested it with the kafka-console-producer.sh & the kafka-console-consumer.sh.
There are 2 limitations though, that I highlighted with red on the 2nd screenshot:
1. I have to start Zookeeper first, and then copy it's IP address into the Broker setup. If I restart Zookeeper or the host, new IP will be assigned, hence I will have to update the Broker config.
2. The Broker's setup has a similar issue, but even worse: I have to know the IP of the broker before it is started, and entered into the advertised listener config. The way I do this is to start the broker with a dummy config (expected to fail), copy the IP address, increment the last number of the IP (I know that will be the next assigned IP), shut down the broker, update the config, and start the broker again. And voila: it is working.
I was hoping that pods would have a static hostname, but that is not the case: new hostname is assigned to pods after every restart, such as kafka-ix-chart-84686b656b-wklgx: the last 5 characters are renewed after every restart.
I am wondering if you could help me to solve the above limitations?
I.e. I can think of 3 options:
- Assign static IP to pods in the host's address space (i.e. 192.168.x.x) - I know how to do this, but feels sub-optimal
- Assign static IP to pods in the Kubernetes address space (i.e. 172.16.0.x) - I don't know how to do this, but still does not feel optimal
- Reference the pods with a name that does not change during restart - Feels like the best option, but I don't know how could I achieve this.
Any help is greatly appreciated.