-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum has become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
TrueCommand Docker Container
- Thread starter JoshDW19
- Start date
-
- Tags
- truecommand
@aervin thank you for following up.
Any clues about the system connectivity issue experienced while running the TrueCommand container on a Docker Swarm? Another reported case of this issue is mentioned here: https://www.ixsystems.com/community/threads/truenas-docker-swarm-connections.83218/
Any clues about the system connectivity issue experienced while running the TrueCommand container on a Docker Swarm? Another reported case of this issue is mentioned here: https://www.ixsystems.com/community/threads/truenas-docker-swarm-connections.83218/
@aervin Thank you for looking into this further :)
/var/log/ix_middleware.log
Nothing of note in the nginx, pgsql or supervisord log files.
I tested a SSL connection to a FreeNAS system, from inside the container. As you can see, this works. However systems are reported as offline in the TrueComamnd GUI (see attached screenshots).
/var/log/ix_middleware.log
Code:
root@f80a465cac9a:~# cat /var/log/ix_middleware.log
[STARTING] "Thu Apr 23 00:10:54 2020"
Opening Database...
Creating the TrueCommand Database: ixdb
Database error: "could not connect to server: Connection refused\n\tIs the server running on host \"localhost\" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host \"localhost\" (::1) and accepting\n\tTCP/IP connections on port 5432?\nQPSQL: Unable to connect"
Could not open database!!
Server could not be started: 2020-04-23T00:10:54
- Tried ports: 5182 5183
QThread: Destroyed while thread is still running
[STARTING] "Thu Apr 23 00:10:55 2020"
Opening Database...
Database error: "FATAL: the database system is starting up\nFATAL: the database system is starting up\nQPSQL: Unable to connect"
Could not open database!!
Server could not be started: 2020-04-23T00:10:55
- Tried ports: 5182 5183
QThread: Destroyed while thread is still running
[STARTING] "Thu Apr 23 00:10:57 2020"
Opening Database...
Database Connected
DB ERROR: "ERROR: null value in column \"tvinstanceuid\" violates not-null constraint\nDETAIL: Failing row contains (null, null, 5443a61b).\n(23502) QPSQL: Unable to create query"
- query: "INSERT INTO tv.license (anonid) VALUES ('5443a61b');"
Using SSL Library:
- Version: "OpenSSL 1.1.1d 10 Sep 2019"
WS server started on port number 5182
HTTP server started on port number 5183Nothing of note in the nginx, pgsql or supervisord log files.
I tested a SSL connection to a FreeNAS system, from inside the container. As you can see, this works. However systems are reported as offline in the TrueComamnd GUI (see attached screenshots).
Code:
root@f80a465cac9a:~# echo "GET / HTTP/1.1" | openssl s_client -connect freenas_host:443
CONNECTED(00000003)
...removed certificate details...
---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2758 bytes and written 451 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 61344AC0CC41361E9D4CA6806C0D8EA12689ABF9FC86ADE7F4DC97CDF19C08DE
Session-ID-ctx:
Master-Key: AB2CEBC2C2580C4B229C49404747DFE93EBF8F96B52B3E16A4A2070A2AFC60D45C89083D2DED6FC13A8541EC8EA787B2
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 9b ca b8 cd 53 c6 f9 90-b6 d5 a4 33 44 a8 2f d5 ....S......3D./.
0010 - ed e4 59 cd b3 2a 80 f5-e9 32 3a 99 aa df a1 35 ..Y..*...2:....5
0020 - b6 1a 58 07 94 f1 14 41-a8 57 ca f8 86 31 ff c0 ..X....A.W...1..
0030 - 0a c0 ac 14 c9 96 91 94-cb 30 2e 50 de 12 1a 55 .........0.P...U
0040 - 9a 0f a9 01 d3 d6 4a fd-21 5f d2 3c bd 85 eb 64 ......J.!_.<...d
0050 - a6 2d 97 50 99 34 f0 7f-be 63 40 34 ec 83 47 54 .-.P.4...c@4..GT
0060 - 9d a3 51 29 2c 05 db 23-b9 a4 73 c8 b8 21 6e d7 ..Q),..#..s..!n.
0070 - 2b 7b b0 ac a3 21 45 db-da da 61 37 44 71 40 33 +{...!E...a7Dq@3
0080 - 88 44 7c 7f fd 92 65 d9-57 b4 25 17 9e 3a 1c 7f .D|...e.W.%..:..
0090 - 4a 81 84 9c e4 95 f9 41-5f 45 02 30 40 c1 af f1 J......A_E.0@...
00a0 - 1d 85 1a 89 57 d5 fc 67-46 ed 22 ea c4 7b 09 80 ....W..gF."..{..
00b0 - f6 f8 86 25 f9 0d 79 7c-3d 29 d3 32 3f ae 5c 78 ...%..y|=).2?.\x
00c0 - bc ba c8 fd 03 ad 24 bb-68 01 b5 2b b3 6c 9a be ......$.h..+.l..
Start Time: 1587603743
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
DONEAttachments
Further, I also ran a basic tcpdump on VM's Docker Swarm gateway interface, while testing the SSL connection per above (i.e. within a shell on the container). As you can see, this verifies there is no network connectivity issue from the container to the FreeNAS system (and from the system to the container). I will note however, tcpdump does not capture ANY traffic from the TrueCommand container when it's started, restarted, or when removing all systems and adding them again. I would expect to see network traffic while preforming those actions. As such, this suggests to me an application level issue.
Please let me know if you require any further data.
Code:
# tcpdump -i docker_gwbridge -n host freenas_host tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on docker_gwbridge, link-type EN10MB (Ethernet), capture size 262144 bytes 01:30:42.334539 IP 172.18.0.4.35606 > freenas_host.https: Flags, seq 2774496244, win 29200, options [mss 1460,sackOK,TS val 170260056 ecr 0,nop,wscale 7], length 0 01:30:42.428686 IP freenas_host.https > 172.18.0.4.35606: Flags [S.], seq 2378697687, ack 2774496245, win 65535, options [mss 1288,nop,wscale 6,sackOK,TS val 3520597567 ecr 170260056], length 0 01:30:42.428731 IP 172.18.0.4.35606 > freenas_host.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 170260150 ecr 3520597567], length 0 01:30:42.429063 IP 172.18.0.4.35606 > freenas_host.https: Flags [P.], seq 1:326, ack 1, win 229, options [nop,nop,TS val 170260150 ecr 3520597567], length 325 01:30:42.524187 IP freenas_host.https > 172.18.0.4.35606: Flags [.], seq 1:1277, ack 326, win 1026, options [nop,nop,TS val 3520597662 ecr 170260150], length 1276 01:30:42.524223 IP freenas_host.https > 172.18.0.4.35606: Flags [P.], seq 1277:2485, ack 326, win 1026, options [nop,nop,TS val 3520597662 ecr 170260150], length 1208 01:30:42.524261 IP 172.18.0.4.35606 > freenas_host.https: Flags [.], ack 1277, win 251, options [nop,nop,TS val 170260246 ecr 3520597662], length 0 01:30:42.524272 IP 172.18.0.4.35606 > freenas_host.https: Flags [.], ack 2485, win 274, options [nop,nop,TS val 170260246 ecr 3520597662], length 0 01:30:42.525321 IP 172.18.0.4.35606 > freenas_host.https: Flags [P.], seq 326:452, ack 2485, win 274, options [nop,nop,TS val 170260247 ecr 3520597662], length 126 01:30:42.620140 IP freenas_host.https > 172.18.0.4.35606: Flags [P.], seq 2485:2759, ack 452, win 1026, options [nop,nop,TS val 3520597758 ecr 170260247], length 274 01:30:42.620683 IP 172.18.0.4.35606 > freenas_host.https: Flags [P.], seq 452:496, ack 2759, win 294, options [nop,nop,TS val 170260342 ecr 3520597758], length 44 01:30:42.620721 IP 172.18.0.4.35606 > freenas_host.https: Flags [FP.], seq 496:527, ack 2759, win 294, options [nop,nop,TS val 170260342 ecr 3520597758], length 31 01:30:42.714793 IP freenas_host.https > 172.18.0.4.35606: Flags [.], ack 528, win 1025, options [nop,nop,TS val 3520597853 ecr 170260342], length 0 01:30:42.714833 IP freenas_host.https > 172.18.0.4.35606: Flags [F.], seq 2759, ack 528, win 1026, options [nop,nop,TS val 3520597853 ecr 170260342], length 0 01:30:42.714852 IP 172.18.0.4.35606 > freenas_host.https: Flags [.], ack 2760, win 294, options [nop,nop,TS val 170260436 ecr 3520597853], length 0
Please let me know if you require any further data.
Hi,
From docker container I can't reach FreeNAS server:
But from another container on the same network I can reach the server:
And this is the network configuration:
Also the log file is container is the same of @jon.z above.
P.S.: Resolved installing Docker Container in Fedora Server 31
Best Regards,
Antonio
From docker container I can't reach FreeNAS server:
Code:
[tony@docker-server ~]$ docker container exec -it loving_bell /bin/bash root@9892b3253015:~# nc 192.168.50.3 80 Ncat: No route to host. root@9892b3253015:~# nc 192.168.50.3 443 Ncat: No route to host. root@9892b3253015:~# nc 192.168.50.3 22 Ncat: No route to host.
But from another container on the same network I can reach the server:
Code:
[tony@docker-server ~]$ docker container attach alpine1 / # ping 192.168.50.3 PING 192.168.50.3 (192.168.50.3): 56 data bytes 64 bytes from 192.168.50.3: seq=0 ttl=63 time=1.041 ms 64 bytes from 192.168.50.3: seq=1 ttl=63 time=0.678 ms 64 bytes from 192.168.50.3: seq=2 ttl=63 time=0.481 ms 64 bytes from 192.168.50.3: seq=3 ttl=63 time=0.563 ms 64 bytes from 192.168.50.3: seq=4 ttl=63 time=0.644 ms ^C --- 192.168.50.3 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 0.481/0.681/1.041 ms / # nc 192.168.50.3 80 / # nc 192.168.50.3 443 / # read escape sequence
And this is the network configuration:
Code:
[tony@docker-server ~]$ docker network inspect my-net
[
{
"Name": "my-net",
"Id": "666f9979e66eb747b26d8597e28d7f1b17dd9690d748146746a97c6a23dfdad6",
"Created": "2020-04-24T11:25:16.448601611+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"9892b3253015488c3b00ae460ea1098d9a8ae74b8f16f0f10a20d842d03c5d21": {
"Name": "loving_bell",
"EndpointID": "15b02a80daf6aa88ff26293ef780eb5b6e22eb245864158af584d0363260c067",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
},
"9dfa7116923c33f470ac2fc830eb6239fd2a7de6c8b38a1d174515fa4d2d8947": {
"Name": "alpine1",
"EndpointID": "ec17b666fbf1db91a8f1f331abe22e4172f1534c49fe625ad525168f2a288dd0",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Also the log file is container is the same of @jon.z above.
P.S.: Resolved installing Docker Container in Fedora Server 31
Best Regards,
Antonio
Last edited:
rookie_eyes
Cadet
- Joined
- Sep 3, 2020
- Messages
- 2
Hey There,
Do we happen to know if the True Command docker image has been tested on a Raspberry Pi 4???? I'm wanting to deploy true command via this solution but I think my current raspberry pi 3B isn't man enough. I wanted to see if someone's tested this before I go and spend out on a new one.
Thanks in advance.
- Joined
- Feb 2, 2018
- Messages
- 1,401
Hey @rookie_eyes,
A Raspberry PI is not running an Intel processor. It is running an ARM processor. As such, the code must be recompiled for that. Until you see a docker container compiled for ARM, there is nothing to test....
rookie_eyes
Cadet
- Joined
- Sep 3, 2020
- Messages
- 2
is there an area to request this? Happy to be a guinea pig! :) I happen to love the idea of running this sort of container on a Pi. With companies like Apple going 100% ARM next year it seems like a good time to get this into at least development phase :-S
freenasmww
Cadet
- Joined
- Oct 11, 2016
- Messages
- 4
I'm unable to get the Docker container started in my enviornment.
Google tells me this is because NGINX is attempting to bind with IPV6. I do not have IPV6 enabled on the underlying host.
Code:
Starting TrueCommand: Mon Dec 21 10:45:03 EST 2020 Starting Services [1/3] Starting PostgreSQL 11 database server: main. Starting Services [2/3] nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol) [ERROR] Could not start Nginx! - Finished: Mon Dec 21 10:45:05 EST 2020
Google tells me this is because NGINX is attempting to bind with IPV6. I do not have IPV6 enabled on the underlying host.
- Joined
- Oct 10, 2018
- Messages
- 25
Did you manually disable IPv6 on your Host? There was a period of time that nginx failed to launch if the host did not support IPv6. I believe this was fixed, we will look into that. in the meantime, you can enable IPv6 and just not use it. or the nginx configuration in the docker container will need to be modified for remove the listen [::]:80 directive.
freenasmww
Cadet
- Joined
- Oct 11, 2016
- Messages
- 4
I do not specifically remember disabling IPV6. However I would prefer it remain disabled.
How should the nginx configuration be modified?
- Joined
- Oct 10, 2018
- Messages
- 25
I loaded the up a VM with Debian 10.7, the latest version of docker and our Truecommand Docker image with ipv6 disabled on the host system and I was unable to recreate this issue. So I do not believe the configuration needs to be edited. Can you give me more specific of your specific setup? What OS and version are you running on the host? is this it he latest truecommand docker container you are trying to run? Is the Host system a fresh install?
freenasmww
Cadet
- Joined
- Oct 11, 2016
- Messages
- 4
I believe I am pulling the latest version.
Code:
cat truecommand.yml
version: "3.8"
services:
truecommand:
environment:
- PUID=${PUID}
- PGID=${PGID}
- TZ=${TZ}
volumes:
- ${USERDIR}/docker/truecommand/data:/data
ports:
- 9490:80
- 9491:443
restart: unless-stopped
container_name: truecommand
image: ixsystems/truecommandIt is not a fresh build.
Code:
cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
My network interfaces are defined by Netplan. I have not specifically disabled IPV6, nor have I enabled it.
Code:
cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
version: 2
renderer: networkd
ethernets:
eth0:
match:
macaddress: xxxxxxxxxx
dhcp4: yes
eth1:
match:
macaddress: xxxxxxxxxx
addresses:
- 10.0.10.2/24
mtu: 9000
The following parameters have been modified.
Code:
cat /etc/sysctl.d/99-network-tuning.conf net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.core.optmem_max = 40960 net.ipv4.tcp_rmem = 4096 1048576 2097152 net.ipv4.tcp_wmem = 4096 65536 16777216 net.core.netdev_max_backlog = 100000 net.core.netdev_budget = 50000 net.core.netdev_budget_usecs = 5000 net.ipv4.tcp_max_syn_backlog = 30000 net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 net.ipv4.tcp_slow_start_after_idle = 0
Docker and Docker Compose Versions
Code:
docker --version Docker version 19.03.13, build 4484c46d9d docker-compose --version docker-compose version 1.27.4, build 40524192
Docker DIR Permissions
Code:
sudo usermod -aG docker ${USER}
mkdir ~/docker
sudo setfacl —no-mask -Rdm g:docker:rwx ~/docker
sudo chmod -R 775 ~/docker
getfacl docker
# file: docker
# owner: mark
# group: mark
user::rwx
group::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:docker:rwx
default:mask::rwx
default:other::r-x
Something looks off w/ the resultant permissions after the first run of the container.
Code:
~/docker/truecommand/data$ ls -al total 12 drwxr-xr-x+ 3 root root 4096 Dec 21 10:44 . drwxr-xr-x+ 3 root root 4096 Dec 21 10:44 .. drwx------ 19 uuidd uuidd 4096 Dec 21 10:45 ixdb
Compared to another container as an example.
Code:
~/docker/homebridge$ ls -al total 228 drwxr-xr-x+ 7 mark docker 4096 Nov 14 16:46 . drwxrwxr-x+ 12 mark mark 4096 Dec 21 10:44 .. -rw-r--r--+ 1 mark docker 81 Aug 13 12:48 .docker.env -rw-rw-r--+ 1 mark docker 663 Aug 13 12:51 .uix-dashboard.json -rw-rw-r--+ 1 mark docker 81 Aug 13 12:48 .uix-secrets drwxrwxr-x+ 2 mark docker 4096 Sep 4 11:46 accessories -rw-rw-r--+ 1 mark docker 359 Aug 13 12:48 auth.json drwxrwxr-x+ 4 mark docker 4096 Nov 15 01:15 backups -rw-rw-r--+ 1 mark docker 407 Sep 4 07:35 config.json -rw-rw-r--+ 1 mark docker 152685 Dec 22 01:15 homebridge.log drwxrwxr-x+ 2 mark docker 4096 Aug 13 12:48 logs drwxrwxr-x+ 5 mark docker 4096 Sep 4 07:35 node_modules -rw-rw-r--+ 1 mark docker 12653 Nov 14 16:46 package-lock.json -rw-r--r--+ 1 mark docker 196 Sep 4 07:35 package.json drwxrwxr-x+ 2 mark docker 4096 Aug 13 12:48 persist -rwxr-xr-x+ 1 mark docker 573 Aug 13 12:48 startup.sh
Last edited:
Error after update kernel. Can u healp me?
Code:
Database Connected Using SSL Library: - Version: "OpenSSL 1.1.1d 10 Sep 2019" Segmentation fault (core dumped) - Finished: Thu Feb 25 14:59:12 EET 2021 Starting TrueCommand: Thu Feb 25 14:59:25 EET 2021 ln: failed to create symbolic link '/etc/ssl/certs/./tc-test-test.pem': File exists Starting Services [1/3] Starting PostgreSQL 11 database server: main. Starting Services [2/3] Starting Services [3/3] [STARTING] "Thu Feb 25 14:59:28 2021" Opening Database... Database Connected Using SSL Library: - Version: "OpenSSL 1.1.1d 10 Sep 2019" Segmentation fault (core dumped) - Finished: Thu Feb 25 14:59:29 EET 2021
NightHawk-ATL
Cadet
- Joined
- Jun 1, 2023
- Messages
- 3
I have just moved my docker instance to a new VM and spun up Truecommand. I am getting zombie processes on that VM coming from truecommand-mw.
I don't recall seeing a zombie process for caddy and truecommand on my old Docker instance so I was wondering what I may have done wrong.
Here is my compose for a custom template in Portainer. It's the same code I have used for the past 2 years, I just changed it from V1.2 to latest on the image tag about 8 months or so ago when I noticed I wasn't getting the new version update.
I don't recall seeing a zombie process for caddy and truecommand on my old Docker instance so I was wondering what I may have done wrong.
Here is my compose for a custom template in Portainer. It's the same code I have used for the past 2 years, I just changed it from V1.2 to latest on the image tag about 8 months or so ago when I noticed I wasn't getting the new version update.
