Docker containers not working consistently, networking issue?

[diskdiddler]

EDIT:

This has been reported by me as a bug, the --net=host is the only way, to get ports, to present over docker externally.
-p <portoutside>:<portinside> does not appear to be working, how I would expect it to work on a docker host.







Original post:
Hello!
Disclaimer: unsupported, I'm just using the CLI to run docker containers, I'm not using the webUI and all the k8 stuff.


I have installed fresh on a spare machine.
Created an SSD based pool.
I have clicked on apps and successfully configured the SSDPOOL to be the app directory for apps.
I have run sudo docker pull xyz:abc and managed to pull several containers.
These are listed in the webui under manage docker images
Some of them even run fine....



I am curious if somehow some ports are blocked or something simple?
Example these 2 work fine:

sudo docker run -d --name=couchpotato -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -e UMASK_SET=022 --net=host -p 5050:5050 -v /mnt/SSDPOOL/dockerdata/couchpotato:/config --restart unless-stopped linuxserver/couchpotato

sudo docker run -d --name=qbittorrent --net=host -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -e UMASK_SET=022 -e WEBUI_PORT=8080 -p 6881:6881 -p 6881:6881/udp -p 8080:8080 -v /mnt/SSDPOOL/dockerdata/qbittorrent/:/config --restart unless-stopped linuxserver/qbittorrent

However these 3 do not work.

sudo docker run -d --name=jdownloader-2 -p 5800:5800 -p 3129:3129 -e USER_ID=0 -e GROUP_ID=0 -e UMASK=000 -e DISPLAY_WIDTH=1520 -e DISPLAY_HEIGHT=620 -v /mnt/SSDPOOL/dockerdata/jdownloader:/config:rw -v /etc/localtime:/etc/localtime --restart=unless-stopped jlesage/jdownloader-2

sudo docker run -d --name=speedtest-tracker -p 8765:80 -v /mnt/SSDPOOL/dockerdata/speedtest-tracker/:/config -e TZ=Australia/Melbourne -e OOKLA_EULA_GDPR=true --restart unless-stopped henrywhitaker3/speedtest-tracker:dev

sudo docker run -d --name=medusa --dns=192.168.0.252 --restart unless-stopped -v /mnt/SSDPOOL/dockerdata/medusa:/config -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -p 8111:8081 pymedusa/medusa

I can clearly see, that data is written to /mnt/SSDPOOL/dockerdata/medua or /mnt/SSDPOOL/dockerdata/speedtest-tracker & /mnt/SSDPOOL/dockerdata/jdownloader:/config

I can sudo docker exec -i <containername> /bin/bash and navigate, I can even mkdir -v mounted folders and write contents out to the real filesystem.
None the less there's nothing coming up on :8765 :8111 or :5800
?


Has anyone else encountered this, am I doing something foolish or is there some kind of linux firewall / ip blocking against certain ports? I'm not entirely sure.
Obviously, I can't run anything natively, on port 80 but this is not that port.

Any tips?

 

[diskdiddler]

Just to be clear, the 3 docker containers that aren't working, won't give me a web interface for the applications that are running.
I'm not a docker guru, hardly but I'm getting better and those 3 containers work a dream on my UbuntuVM with (effectively) the same initialisation command.
They appear (??) to be running fine but even giving them 30-60 minutes on this slower test PC, they still don't produce any web page of any kind.

 

[diskdiddler]

Final thing:


root@hptruenas[~]# sudo wget 127.0.0.1:8080 (qbittorrent, working container)
--2022-01-05 22:17:41-- http://127.0.0.1:8080/
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1646 (1.6K) [text/html]
Saving to: ‘index.html.2’

index.html.2 100%[===================================================================>] 1.61K --.-KB/s in 0s

2022-01-05 22:17:41 (59.0 MB/s) - ‘index.html.2’ saved [1646/1646]

root@hptruenas[~]# sudo wget 127.0.0.1:8111 (medusa, dead container)
--2022-01-05 22:17:50-- http://127.0.0.1:8111/
Connecting to 127.0.0.1:8111... failed: Connection refused.
root@hptruenas[~]#




Thanks team(s)

 

[sretalla]

I would start with a docker logs <containername> on each of those containers... maybe that will give you some hints about what's going on.

 

[diskdiddler]

First, thanks! I can't believe I've used docker as long as I have and didn't know that command.

Second, my theory seems to hold true, I can see that the container is clearly running and performing its function, trying to fetch (0) TV shows in the queue. (there's none added, obviously it's a fresh install)

Same with jdownloader-2 it seems to be comfortably up and running, just nothing visible at the correct URL.

 

[diskdiddler]

I have investigated this further today.
The following command will make couchpotato NOT work:

sudo docker run -d --name=couchpotato -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -e UMASK_SET=022 -p 5050:5050 -v /mnt/SSDPOOL/dockerdata/couchpotato:/config --restart unless-stopped linuxserver/couchpotato

These command will work:

sudo docker run -d --name=couchpotato -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -e UMASK_SET=022 --net=host -p 5050:5050 -v /mnt/SSDPOOL/dockerdata/couchpotato:/config --restart unless-stopped linuxserver/couchpotato

sudo docker run -d --name=couchpotato -e PUID=0 -e PGID=0 -e TZ=Australia/Melbourne -e UMASK_SET=022 --net=host -v /mnt/SSDPOOL/dockerdata/couchpotato:/config --restart unless-stopped linuxserver/couchpotato

Therefore, it seems you MUST do --net=host and therefore, the -p command for port mapping, isn't working properly? Is this normal or a bug? I'm suspecting this is by design?

 

[diskdiddler]

I've logged this as a bug as it does not appear to be the behaviour I would expect at all. I could be entirely wrong but I set the priority as low.

 

[amiskell]

I have investigated this further today.
The following command will make couchpotato NOT work:



These command will work:







Therefore, it seems you MUST do --net=host and therefore, the -p command for port mapping, isn't working properly? Is this normal or a bug? I'm suspecting this is by design?

When you specify --net=host, you're running the containers network on the hosts TCP/IP stack so the port mapping settings don't apply, the container will open the ports it requires directly on the hosts TCP/IP stack instead of the standard way of operating it's own TCP/IP stack and NAT'ting the ports from the host to the container.

See https://docs.docker.com/network/host/ for details. It's by design and expected behavior.

 

[icedicemilo]

Try creating a Firewall rule to allow internal Docker IP ranges (e.g. 172.17.0.X to 172.17.0.Y). That worked for me and I didn't need the --net=host flag anymore.

 

[diskdiddler]

Sorry I should've responded.

I'm ... a newbie compared to the devops folks, but I thought I knew my stuff.

The docker instance on TrueNAS is not your traditional fresh copy of Ubuntu desktop / server docker. The devs of TrueNAS got back to me on the Jira and recommended I ONLY create docker instances using the GUI - not the CLI, ever.

Apparently all my containers must have their own IP address, rather than the more traditional, basic method of "ipaddress:port" (my current ubuntu server has 16 containers all on diff ports)

Anyhow with some testing I did get things working a little better with the GUI, however I can't stand it and I want "real" docker in the CLI. (also what's with everything being named "charts"?)

Anyhow I've given up on it, I'll stick with TrueNAS core for the time being and my UbuntuVM, it's been for the most part, shockingly reliable, it might not be too fast but man it does a heck of a lot with only 5GB allocation.

 

[truecharts]

The docker instance on TrueNAS is not your traditional fresh copy of Ubuntu desktop / server docker. The devs of TrueNAS got back to me on the Jira and recommended I ONLY create docker instances using the GUI - not the CLI, ever.

Actually it is.
But if you want to be able to file Jira tickets, you always have to use iX provided tools. Hence their advice.

 

[diskdiddler]

Actually it is.
But if you want to be able to file Jira tickets, you always have to use iX provided tools. Hence their advice.

The odds are extremely likely, that you know more than I do, like extremely likely.

All I know, is if I install a regularl distro of linux on a VM or a normal PC.
I then update the docker daemon (at least Ubuntu seems to need this)

I can then run a command as per the TOP post in this thread, and they work, flawlessly.

When I try the same commands, they work on TrueNAS but the container isn't visible. Idon't know why? Some kind of networking change or design decision. ?

I don't understand why a container with a remapped port of 5000 externally to 80 inside the container (-p 5000:80) isn't working, but it's not.
When I use the UI based tools to create a docker container, instead of being TruenasIP:5000 it's DifferentIP

Shrug?
Anyhow I do understand it's not the developers jobs to support normal operating system / docker functionaliry.

 

[truecharts]

The odds are extremely likely, that you know more than I do, like extremely likely.

All I know, is if I install a regularl distro of linux on a VM or a normal PC.
I then update the docker daemon (at least Ubuntu seems to need this)

I can then run a command as per the TOP post in this thread, and they work, flawlessly.

When I try the same commands, they work on TrueNAS but the container isn't visible. Idon't know why? Some kind of networking change or design decision. ?

I don't understand why a container with a remapped port of 5000 externally to 80 inside the container (-p 5000:80) isn't working, but it's not.
When I use the UI based tools to create a docker container, instead of being TruenasIP:5000 it's DifferentIP

Shrug?
Anyhow I do understand it's not the developers jobs to support normal operating system / docker functionaliry.

Okey, lets take a step back here indeed...

TrueNAS (regardless of version) is, basically, an embedded OS. Which means it is not designed to be alterable and only designed being able to work with the GUI, their "CLI" tool and API.

While all software is pretty much stock debian, This also means all included software is pre-configured (rather than altered) to work perfectly with their above tools. Which does not always mean they still work as-great when used directly from the commandline, because the OS as a whole isn't build around supporting that.

In the case of Docker, it is, pretty much, standard docker... But it gets pre-configured to work best with the Apps system (afaik only if you actually enable the Apps system though). Basically you told the system "I want to use docker with Apps" and then it setup docker to be (best) usable for Apps.

In this case some (minor) networking settings of Docker are disabled, because SCALE itself does not utilise docker networking and they caused issues with their kubernetes deployment (which, as a mater of fact, uses it's own networking stack).

There are some guides out there to hack the system to allow normal docker use, instead( not besides!) the SCALE Apps system. But as always, that is not officially supported.

It might also be good describing what "Not officially supported" means in this context:
- You *cannot* file Jira tickets if something doesn't work (including bug reports in most cases)
- It is *not* covered by any support contract

What you are left with for support is (basically), any place that isn't iX Systems themselves :)

 

[diskdiddler]

Okey, lets take a step back here indeed...

TrueNAS (regardless of version) is, basically, an embedded OS. Which means it is not designed to be alterable and only designed being able to work with the GUI, their "CLI" tool and API.

While all software is pretty much stock debian, This also means all included software is pre-configured (rather than altered) to work perfectly with their above tools. Which does not always mean they still work as-great when used directly from the commandline, because the OS as a whole isn't build around supporting that.

In the case of Docker, it is, pretty much, standard docker... But it gets pre-configured to work best with the Apps system (afaik only if you actually enable the Apps system though). Basically you told the system "I want to use docker with Apps" and then it setup docker to be (best) usable for Apps.

In this case some (minor) networking settings of Docker are disabled, because SCALE itself does not utilise docker networking and they caused issues with their kubernetes deployment (which, as a mater of fact, uses it's own networking stack).

There are some guides out there to hack the system to allow normal docker use, instead( not besides!) the SCALE Apps system. But as always, that is not officially supported.

It might also be good describing what "Not officially supported" means in this context:
- You *cannot* file Jira tickets if something doesn't work (including bug reports in most cases)
- It is *not* covered by any support contract

What you are left with for support is (basically), any place that isn't iX Systems themselves :)

Thank you I appreciate it. Ultimately it is 'modified' in some capacity, even if it's slightly.
Dollars to donuts, if I had a bit more skills it probably wouldn't bother me, but as it stands I'm accustomed to literally SSH'ing in to my Ubuntu VM hitting the up arrow and pulling up an epically huge command

docker pull linuxserver/blah && docker stop blah && docker rm blah && docker run -d blah (etc etc) and this giant 25 line long command, stops 10+ containers, pulls fresh copy, re runs them.

Probably better ways to do it and I bet there's good reasons to have IP based docker networking rather than port based, infact considering what I've recently been told about my opnsense router and wireguard, I could do some fancy VPN stuff with it.

Anyhow in time I may try scale again - and I do agree, TrueNAS is not to be 'fiddled with' in the backend - old Cyberjock hammered that into us, to leave the core OS alone and let it do it's thing and considering the stability and reliability it seems a wise decision.

I'll try again once Scale has been out for over 6 months or so.