Networking/Routing trouble in Docker Containers on 20.12

[kuchenblech]

Hi everyone,

I was glad to see the new release and have already upgraded from 20.10.

I have checked out the new Applications GUI and set up the Plex container.
However, there are also a couple of docker-compose stacks that I would like to continue using as is, until gaining some experience with Kubernetes.

As mentioned in another post it should be possible to just run docker-compose up. And indeed, all images come up, but they have no network access. The docker network is created but I cannot reach any host outside, and have no DNS either.

Consider this:

Code:

$ docker create network foo
$ docker run -it --network foo alpine /bin/sh

/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
120: eth0@if121: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
      
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0

/ # cat /etc/resolv.conf
search fritz.box
nameserver 127.0.0.11
options ndots:0

The only host that is reachable by ping from inside is the gateway 172.17.0.1, but nothing beyond that. Also, (I guess as a consequence) hostname resolution doesn't work.

Is there any setting that I am missing to get the routing to work?

Best
Thomas

 

[kuchenblech]

Ok, I have made some progress. Everything works as intendend, if I set "iptables": true in /etc/docker/daemon.json.

However, I'm not sure if this is intended, or am I going to break something?

 

[morganL]

Docker Compose is the preferred tool for a Docker environment

Helm Charts are the preferred tool for Kubernetes. - they are more powerful and generally a super set of the Compose yaml file capabilities.

There is a tool called Kompose that converts Docker-compose files to Helm charts: Check it out at https://kompose.io/

 

[kuchenblech]

Thanks, I had a quick look at kompose. At first glance, it didn't convert my mounts (e.g. from /mnt/foo on the host into the container), but I may have missed something.

 

[morganL]

Are you trying to migrate data or just run a new application? I agree that data migration is not addressed by these tools.

 

[dalnew]

I think he is trying to do what I did and local mounts that contain data that are mapped into the container don't seem to get translated properly by the kompose tool. I had a plex container that I tried converting that had 6 or 7 local mounts mapped into the container. When I look at the output of
"kompose convert"
I end up with a bunch of files like:
plex-deployment.yaml
plex-claim0-persistentvolumeclaim.yaml
plex-claim1-persistentvolumeclaim.yaml
plex-claim2-persistentvolumeclaim.yaml
...

but none of them actually have any of the original /mnt/Movies, /mnt/TV, etc in them and the volumes section of plex-deployment.yaml has references to those files, but again, no actual pointers to the real locations to be mapped in.

volumes:
- name: plex-claim0
persistentVolumeClaim:
claimName: plex-claim0
- name: plex-claim1
persistentVolumeClaim:
claimName: plex-claim1
- name: plex-claim2
persistentVolumeClaim:
claimName: plex-claim2
readOnly: true

I guess we have to manually go in and add those mount points back in? Or maybe there's an additional command line argument to pass to kompose convert that we're missing?

 

[ornias]

I would suggest contacting the folks at compose, it really shouldn't translate everything to PVC's.
I know PVC is "the kubernetes way", but still.

To give you some idea's...
In deployment.yaml:

Code:

          volumeMounts:
          - name: data
            mountPath: /data
          - name: config
            mountPath: /config
          - name: downloads
            mountPath: /downloads
          - name: shared
            mountPath: /shared

same file, under " volumes:

Code:

      volumes:
      - name: data
        hostPath:
          path: /mnt/tank/data
      - name: config
        hostPath:
          path: /mnt/tank/config
      - name: downloads
        hostPath:
          path: /mnt/tank/downloads
      - name: shared
        hostPath:
          path: /mnt/tank/shared

 

[morganL]

I would suggest contacting the folks at compose, it really shouldn't translate everything to PVC's.

I assume you meant people at "kompose"?

 

[ornias]

I assume you meant people at "kompose"?

Ofkoarse!

 

[charles997]

Ok, I have made some progress. Everything works as intendend, if I set "iptables": true in /etc/docker/daemon.json.

However, I'm not sure if this is intended, or am I going to break something?

I have been running into the same issue with my docker-compose based containers. I can ping my SCALE host, but nothing on the LAN or public internet. I tried your recommendation of setting "iptables": true in /etc/docker/daemon.json. This didn't work (even after fully rebooting the TrueNAS Scale system.)

Any other ideas?

 

[rmarq]

I have been running into the same issue with my docker-compose based containers. I can ping my SCALE host, but nothing on the LAN or public internet. I tried your recommendation of setting "iptables": true in /etc/docker/daemon.json. This didn't work (even after fully rebooting the TrueNAS Scale system.)

Any other ideas?

Had the same issue. Needed to also remove the bridge setting for it to start working properly.

One "minor" caveat is that the daemon.json file seems to be regenerated after every boot. It's not a big deal for me since I don't reboot the NAS often. Although it means editing the deamon file is not a full-proof solution at the time of writting.

 

[a575606]

I'm still struggling with this issue, not sure if there's a better solution? The above solution works, but I'm not that happy with it because a) it's a bit of a hack and b) it's a hassle to have to manually do every reboot. I'm sure there's a way to automate via script, but still seems messy.

I saw a post on reddit mentioning using a custom docker network, not sure if anyone else had any success with that? https://www.reddit.com/r/truenas/comments/kowe3n/internet_connection_issue_with_docker_on_truenas/

Scratching my head as to why docker's default daemon.json essentially cripples docker.

 

[departy]

Anyone have found a permanent solution to this?

 

[henne]

I have a permanent workaround ;-) https://gist.github.com/Jip-Hop/af3b7a770dd483b07ac093c3b205323f#gistcomment-3967066 copies after reboot the file with a cronjob @Reboot script name or you can also check every 5 minutes via corn if the file has changed and re copy plus restart docker… It worked for me well, but will now migrate fully to truechart.

 

[departy]

Awesome, but why this issue isn’t happening to everyone?
Why is it happening to few of us(I presume if it was mass issue, many more people would raise concern about this)?

 

[henne]

Because it breaks things and Scale is not a Docker solution, it is for kubernetes and there container work. So use it with care and remember to move to what scale is built for.

 

[departy]

I am simply trying to deploy some Radarr or Sonarr Apps, but none of the deployed containers so far have outbound connections. NONE! I am trying to figure out what I am doing wrong! I dont wish to deploy standard containers, I want my APPS (TrueCharts) to have access to internet! Thats all I want :D

 

[truecharts]

I am simply trying to deploy some Radarr or Sonarr Apps, but none of the deployed containers so far have outbound connections. NONE! I am trying to figure out what I am doing wrong! I dont wish to deploy standard containers, I want my APPS (TrueCharts) to have access to internet! Thats all I want :D

You also posted about this problem on our discord and got explained that it's a known bug that happens some time.

However what @henne and others are discussing here, has nothing to do with SCALE Apps, it's a hack to run docker-compose.
The moment you do this, you for sure break SCALE Apps networking. You cannot in any case use this combined with SCALE Apps.

(also: necroing threads for a version that is almost 12 months(!) old, is generally frowned uppon)

 

[truecharts]

Note, we managed to go through it with @departy and there happened to be a type in his SCALE Apps config after all :)

K.S.

 

[departy]

Yes @truecharts!
My issue was coming from misspelling my gateway IP address in Apps -> Advanced Settings

Thank you for the support @truecharts !