TrueCommand 2.3.0 Reverse Proxy

[obsidiangroup]

Did anything else change from 2.2.2 to 2.3.0? WIth 2.2.2, I have my TrueCommand working behind a nginx reverse proxy server, running with no issues. Websocket support is enabled. After upgrading to 2.3.0, can no longer access TrueCommand via reverse proxy.

If you go to https://truecommand.domain.com, it sends the request to https://truecommand.domain.com/auth, which then displays:

Watching the logs for TrueCommand, the only thing I noticed was:

Code:

==> caddy.log <==
{"level":"warn","ts":1674762991.5564384,"logger":"tls","msg":"YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place","docs":"https://caddyserver.com/docs/automatic-https#on-demand-tls"}
{"level":"info","ts":1674762991.5566363,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/.caddy/data/caddy"}
{"level":"info","ts":1674762991.5584602,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1674762991.5589929,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1674762991.5592604,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1674762991.5593576,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1674762991.5616798,"msg":"autosaved config (load with --resume flag)","file":"/data/.caddy/config/caddy/autosave.json"}
{"level":"info","ts":1674762991.5617027,"msg":"serving initial configuration"}
{"level":"info","ts":1674762991.5620532,"logger":"watcher","msg":"watching config file for changes","config_file":"/etc/CaddyFile"}
{"level":"warn","ts":1674762991.7860959,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [172.17.0.2]: no OCSP server specified in certificate","identifiers":["172.17.0.2"]}

And my reverse proxy logs:

Code:

2023/01/26 14:33:35 [error] 474#474: *2779 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.20.202, server: trucommand.domain.co, request: "GET /websocket HTTP/1.1", upstream: "https://192.168.1.50:7000/websocket", host: "truecommand.domain.co"
2023/01/26 14:33:36 [error] 474#474: *2795 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.20.202, server: trucommand.domain.co, request: "GET /websocket HTTP/1.1", upstream: "https://192.168.1.50:7000/websocket", host: "truecommand.domain.co"

[26/Jan/2023:15:02:23 -0500] - 421 421 - GET https truecommand.domain.co "/websocket" [Client 192.168.20.202] [Length 0] [Gzip -] [Sent-to 192.168.1.50] "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
[26/Jan/2023:15:02:26 -0500] - 421 421 - GET https truecommand.domain.co "/websocket" [Client 192.168.20.202] [Length 0] [Gzip -] [Sent-to 192.168.1.50] "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"

However, from my nginx container, I can access the host with no problems. It seems to be an issue with SSL certificates, as attempting to access the page via https://IP:7000 (Port forwarded to 443), I'm prompted to select a certificate to authenticate with:

.

Rolling back to 2.2.2, these problems are gone.

 

[BiggusDickus]

This has nothing to do with your specific setup.
I just ran into the same issue running on my local docker server. When using 2.3.0, the SSL port throws a 421. When using 2.2.2, it doesn't.
So there is just a good ol' bug in 2.3.0.

 

[obsidiangroup]

Too bad we can't see what's changed from 2.2.2 to 2.3.0 outside of release notes. I've rolled back to 2.2.2 and just dealing with the cosmetic issue I have with 2.2.2 for now. @aervin do you have any insight what could be causing this?

 

[ZWelch]

A fix for a related issue in 2.3, where we set explicit client auth verify_if_given for HTTPS, resulted in a strict SNI check, throwing 421. We have a fix in 2.3.1. A temporary fix is to removed the client_auth block from /etc/Caddyfile or add servers { strict_sni_host insecure_off } in the root block.

 

[obsidiangroup]

A fix for a related issue in 2.3, where we set explicit client auth verify_if_given for HTTPS, resulted in a strict SNI check, throwing 421. We have a fix in 2.3.1. A temporary fix is to removed the client_auth block from /etc/Caddyfile or add servers { strict_sni_host insecure_off } in the root block.

So, after removing the client_auth block, I can now log into TrueCommand, *sorta*.

I can login just fine via local username/password. However, if I attempt to login via SAML, I get the "API Error - internal error" popup.
If I login using the local account, and go to Administration -> Configure, and try the SAML Identity Provider Metadata XML File Upload' and upload the valid XML Metadata, I get the same popup. I'm attaching some logs, if more are needed please let me know.

SAML login was working fine with 2.2.2

 

[aervin]

thanks @obsidiangroup - the logs help identify at least one bug with saml config updates. ill post here with ticket info tomorrow.

 

[obsidiangroup]

thanks @obsidiangroup - the logs help identify at least one bug with saml config updates. ill post here with ticket info tomorrow.

@aervin If you need more logs, please let me know. It wasn't just config updates though, also could not login via SAML.

 

[aervin]

I think @ZWelch has you covered @obsidiangroup . Let's compare caddy configs here if needed.

 

[obsidiangroup]

Here's my /etc/Caddyfile. This is modified with the suggestions from @ZWelch . If a purposed fix has been pushed to a nightly, I'll be glad to pull and try that version.

While my TrueCommand is just managing my home NAS', I can get away with testing unstable versions ;). My current TrueCommand is managing two TrueNAS Scale instances, both now running 2022.12.0, currently managing 24 drives, with iSCSI and NFS shares. They are primarily used for iSCSI Datastores back to my ESXi cluster. I don't mind blowing up my TrueCommand for testing (as long as nothing destructive towards my TrueNAS') ;)

TrueCommand is run on a docker swarm running Ubuntu 22.04.

 

[aervin]

thanks @obsidiangroup, you can try the latest from this image if you're feeling brave:

ghcr.io/ixsystems/truecommand:release-2.3

that image is basically 2.3 + patches.

of course, disclaimer: your TC data directory is not safe when trying these images. backup before attempting to use them, yada yada. TC won't do anything to your NASes apart from subscribing to their system health events.

thanks for all your help!

 

[obsidiangroup]

So the first time attempting to connect to TrueCommand, the SAML Login button is not available:

After refreshing (sometimes twice), the SAML option is available, but does not work. Not sure what has changed from 2.2 to 2.3, but I get these two popups:

So I went and checked, and the user has somehow been marked disabled. Not sure how this happened from 2.2 to 2.3, as if I launch the container with 2.2, the user can authenticate fine. There is also no way to mark an inactive user active, outside of removing the user and re-adding it seems. Completely deleting any users created via SAML fixes this problem. There should be a way to mark an inactive user active, which would fix this problem but not explain it.

And, attempting to update the SAML Config results:

Also, attempting to add System Groups causes errors as well:

I attached my caddy.log, ix_middleware.log, and the logs from the container.

 

[obsidiangroup]

of course, disclaimer: your TC data directory is not safe when trying these images. backup before attempting to use them, yada yada. TC won't do anything to your

Thanks for the laugh though. I keep backups, of course, and always before deploying a new version make another backup of the container config directory.

 

[aervin]

hmm thanks for sharing. we're actively working on the regression. just to be sure: you pulled in the latest ghcr image?

 

[obsidiangroup]

Code:

CONTAINER ID   IMAGE                                       COMMAND                 
d27e699d266c   ghcr.io/ixsystems/truecommand:release-2.3   "/usr/middleware/ix_…"