TrueCommand 2.3.0 Reverse Proxy

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
Did anything else change from 2.2.2 to 2.3.0? WIth 2.2.2, I have my TrueCommand working behind a nginx reverse proxy server, running with no issues. Websocket support is enabled. After upgrading to 2.3.0, can no longer access TrueCommand via reverse proxy.

If you go to https://truecommand.domain.com, it sends the request to https://truecommand.domain.com/auth, which then displays:
1674763187374.png


Watching the logs for TrueCommand, the only thing I noticed was:

Code:
==> caddy.log <==
{"level":"warn","ts":1674762991.5564384,"logger":"tls","msg":"YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place","docs":"https://caddyserver.com/docs/automatic-https#on-demand-tls"}
{"level":"info","ts":1674762991.5566363,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/.caddy/data/caddy"}
{"level":"info","ts":1674762991.5584602,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1674762991.5589929,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1674762991.5592604,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1674762991.5593576,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1674762991.5616798,"msg":"autosaved config (load with --resume flag)","file":"/data/.caddy/config/caddy/autosave.json"}
{"level":"info","ts":1674762991.5617027,"msg":"serving initial configuration"}
{"level":"info","ts":1674762991.5620532,"logger":"watcher","msg":"watching config file for changes","config_file":"/etc/CaddyFile"}
{"level":"warn","ts":1674762991.7860959,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [172.17.0.2]: no OCSP server specified in certificate","identifiers":["172.17.0.2"]}


And my reverse proxy logs:
Code:
2023/01/26 14:33:35 [error] 474#474: *2779 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.20.202, server: trucommand.domain.co, request: "GET /websocket HTTP/1.1", upstream: "https://192.168.1.50:7000/websocket", host: "truecommand.domain.co"
2023/01/26 14:33:36 [error] 474#474: *2795 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.20.202, server: trucommand.domain.co, request: "GET /websocket HTTP/1.1", upstream: "https://192.168.1.50:7000/websocket", host: "truecommand.domain.co"

[26/Jan/2023:15:02:23 -0500] - 421 421 - GET https truecommand.domain.co "/websocket" [Client 192.168.20.202] [Length 0] [Gzip -] [Sent-to 192.168.1.50] "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
[26/Jan/2023:15:02:26 -0500] - 421 421 - GET https truecommand.domain.co "/websocket" [Client 192.168.20.202] [Length 0] [Gzip -] [Sent-to 192.168.1.50] "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"


However, from my nginx container, I can access the host with no problems. It seems to be an issue with SSL certificates, as attempting to access the page via https://IP:7000 (Port forwarded to 443), I'm prompted to select a certificate to authenticate with:

1674764273396.png
.

Rolling back to 2.2.2, these problems are gone.
 

iVhksqJjo

Dabbler
Joined
Jan 27, 2023
Messages
17
This has nothing to do with your specific setup.
I just ran into the same issue running on my local docker server. When using 2.3.0, the SSL port throws a 421. When using 2.2.2, it doesn't.
So there is just a good ol' bug in 2.3.0.
 

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
Too bad we can't see what's changed from 2.2.2 to 2.3.0 outside of release notes. I've rolled back to 2.2.2 and just dealing with the cosmetic issue I have with 2.2.2 for now. @aervin do you have any insight what could be causing this?
 

ZWelch

iXsystems
iXsystems
Joined
Oct 11, 2022
Messages
3
A fix for a related issue in 2.3, where we set explicit client auth verify_if_given for HTTPS, resulted in a strict SNI check, throwing 421. We have a fix in 2.3.1. A temporary fix is to removed the client_auth block from /etc/Caddyfile or add servers { strict_sni_host insecure_off } in the root block.
 

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
A fix for a related issue in 2.3, where we set explicit client auth verify_if_given for HTTPS, resulted in a strict SNI check, throwing 421. We have a fix in 2.3.1. A temporary fix is to removed the client_auth block from /etc/Caddyfile or add servers { strict_sni_host insecure_off } in the root block.
So, after removing the client_auth block, I can now log into TrueCommand, *sorta*.

I can login just fine via local username/password. However, if I attempt to login via SAML, I get the "API Error - internal error" popup.
If I login using the local account, and go to Administration -> Configure, and try the SAML Identity Provider Metadata XML File Upload' and upload the valid XML Metadata, I get the same popup. I'm attaching some logs, if more are needed please let me know.

SAML login was working fine with 2.2.2
 

Attachments

  • truecommand.txt
    7.6 KB · Views: 90

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
Here's my /etc/Caddyfile. This is modified with the suggestions from @ZWelch . If a purposed fix has been pushed to a nightly, I'll be glad to pull and try that version.

While my TrueCommand is just managing my home NAS', I can get away with testing unstable versions ;). My current TrueCommand is managing two TrueNAS Scale instances, both now running 2022.12.0, currently managing 24 drives, with iSCSI and NFS shares. They are primarily used for iSCSI Datastores back to my ESXi cluster. I don't mind blowing up my TrueCommand for testing (as long as nothing destructive towards my TrueNAS') ;)

TrueCommand is run on a docker swarm running Ubuntu 22.04.
 

Attachments

  • Caddyfile.txt
    899 bytes · Views: 107

aervin

iXsystems
iXsystems
Joined
Jun 13, 2018
Messages
114
thanks @obsidiangroup, you can try the latest from this image if you're feeling brave:

ghcr.io/ixsystems/truecommand:release-2.3

that image is basically 2.3 + patches.

of course, disclaimer: your TC data directory is not safe when trying these images. backup before attempting to use them, yada yada. TC won't do anything to your NASes apart from subscribing to their system health events.

thanks for all your help!
 

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
So the first time attempting to connect to TrueCommand, the SAML Login button is not available:
1675363547474.png


After refreshing (sometimes twice), the SAML option is available, but does not work. Not sure what has changed from 2.2 to 2.3, but I get these two popups:
1675363679132.png

1675363951388.png


So I went and checked, and the user has somehow been marked disabled. Not sure how this happened from 2.2 to 2.3, as if I launch the container with 2.2, the user can authenticate fine. There is also no way to mark an inactive user active, outside of removing the user and re-adding it seems. Completely deleting any users created via SAML fixes this problem. There should be a way to mark an inactive user active, which would fix this problem but not explain it.

And, attempting to update the SAML Config results:
1675363484867.png



Also, attempting to add System Groups causes errors as well:
1675364880060.png


I attached my caddy.log, ix_middleware.log, and the logs from the container.
 

Attachments

  • 1675362508658.png
    1675362508658.png
    161.8 KB · Views: 104
  • 1675355119173.png
    1675355119173.png
    161.8 KB · Views: 87
  • _trueCommand_logs.txt
    8.7 KB · Views: 81

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
of course, disclaimer: your TC data directory is not safe when trying these images. backup before attempting to use them, yada yada. TC won't do anything to your
Thanks for the laugh though. I keep backups, of course, and always before deploying a new version make another backup of the container config directory.
 

obsidiangroup

Dabbler
Joined
Oct 4, 2022
Messages
19
Code:
CONTAINER ID   IMAGE                                       COMMAND                 
d27e699d266c   ghcr.io/ixsystems/truecommand:release-2.3   "/usr/middleware/ix_…"   
 
Top