Subnet collison with Kubernetes and local networks

[zaxmyth]

My network is broken into a series of VLANs which include the following subnets.:

LAN: 10.10.10.0/24 - My TrueNAS Scale server has an IP address on this network
VLAN50: 172.16.50.0/24 - Restricted network
VLAN60: 172.16.60.0/24 - Security cameras

I have Nextcloud App installed on TrueNAS scale and it is (mostly) working fine.

Yesterday, I wanted to make allow a device on 172.16.60.0/24 to access a share on my TrueNAS system on the 10.10.10.0/24 network. I created a firewall rule to allow the access, confirmed it worked, but then proceeded to have issues making a connection. What I realized after inspecting the routing table on TrueNAS was that Kubernetes is using 172.16.0.0/16 which clearly collides with subnets I used on VLANs. Doh!

It does appear that I can change my Cluster CIDR range in the Kubernetes Settings but it warns me:

Changing settings below will result in Kubernetes cluster re-initialization deleting installed apps and their data.

Which sounds a bit ominous. Before I change my Cluster CIDR to something like 10.253.0.0/16, I just want to confirm that this will not delete the data in the datasets that Nextcloud uses and that if I re-initialize I'll be able to access my Nextcloud data the same as I am currently.

 

[zaxmyth]

I went ahead and tried this so for anybody else embarking on this, here is what I learned.

1. I have all of my Nextcloud data, database, and backups in dedicated datasets and after re-initializing kubernetes, the data was NOT deleted. If you don't have data for your applications in dedicated datasets, I suspect it would be deleted but I can't confirm that.
2. I did have to reinstall Nextcloud and when it came back online, all of my data, metadata, etc. was available and worked as if nothing had happened.
3. My subnet collision was resolved and I can connect to my share on the 10.10.10.0/24 network from the 172.16.60.0/24 network.

 

[Jeff Noxon]

Thank you for posting this. I just ran into the same issue, what a maddening thing to diagnose. I'm going to cross my fingers and try it. I only have minio running; The data is all on a hostpath, so I think it should be fine. We'll see!

 

[Jeff Noxon]

Success, mostly! The minio chart is cursed; I wasn't able to reinstall minio using the existing dataset because it runs a preinstall pod which does a chown -R on the entire dataset. My dataset is huge, and this couldn't complete within the 5m timeout, so it kept failing and rolling back. It was also totally unnecessary, because the ownership was correct.

I worked around it by using zfs rename, got minio installed on a temporary dataset with the correct path, then stopped the pod, deleted the temp dataset, and renamed the old one back...