TrueCommand 2 Issues

[Basil Hendroff]

Here are a few more; one of them looking a lot like an old one I already posted about and that was fixed long time ago...

1-TrueCommand does not let me upload my local CA. After selecting it as a file or copy-paste the text, the IMPORT button remains greyed out.

2-TrueCommand does not respect its TLS configuration. When I enter my TrueNAS's IP address, TrueCommand connects to it. The problem is, the certificate is valid only for the FQDN name and the old IP address which is not the same anymore. TrueCommand is configured to enforce TLS and the check box to ignore SSL errors is empty (as well as the one with hostname mismatch).

3-TrueCommand does not resolve DNS names. When I enter a system's name, TrueCommand does just plain nothing. Even TCPDump does not detect any reaction. TrueCommand reacts only to IP address. The container is able to resolve names and connect to them. I entered in a shell in the container and used openssl to connect my TrueNAS' TCP port 443 calling it by its name. It worked from openssl inside the container but the GUI is unable to do it.

TrueCommand is running from a Linux Docker host outside TrueNAS. Container's image has been updated (image created 2021-06-22 09:32:52) and is the one designated as ":latest" as of now.

The TrueNAS system I try to connect first is Atlas (description in my signature).

@Heracles According to JIRA ticket TC-1799, these issues are fixed in the latest nightly. Can you test and update the ticket if necessary? Let me know here if you're successful so I can update the table in the previous post.

 

[morganL]

FYI TC 2.0.2 should be available next week.

 

[Basil Hendroff]

TC 2 issues table from post #40 brought forward and updated.​

Posts	JIRA ticket	Description	Fix Version	Test Result
#3	TC-1761	Used space on system cards are reported as a whole number	2.1	​
#5	TC-1772	Multiple time formats in use	2.1	​
#6	TC-1784	Active shares not shown in the dashboard	2.1	​
#8	TC-1783	SMR Alert	2.1	​
#15	TC-1770	Calendar improvement for report generation	2.1	​
-	TC-1812	DNS lookup failure	2.1	​
-	TC-1817	New indicators replace activity wheel	-	​
-	TC-1829	Network speed reporting issues	2.1	​

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.

 

[Heracles]

Ok! I managed to diagnose and workaround the bug that prevents TrueCommand to connect to my hosts using DNS names. I updated TC-1812 with all the details. The root cause is that TrueCommand does not know about patience and redundancy. I had a mis-configuration in my environment that made the primary DNS unusable. Instead of turning to its secondary DNS that was working, TrueCommand just gave up and abandoned ship. OpenSSL from inside the container was working because that one turned to its secondary DNS after the primary failed.

Let see now when a new release will have a permanent fix for that. During that time, I fixed my own configuration and my primary DNS is working again, so TrueCommand can connect to my hosts using DNS names.

 

[Heracles]

According the Jira ticket, it should be fixed in 2.1...

 

[danb35]

I think I've run into a TC2 issue--it's reporting implausible network utilization:

This system has two Gigabit NICs, but only one is in use. The system itself reports pretty good utilization of that one:

but obviously not multiple gigabits/sec.

 

[Basil Hendroff]

I think I've run into a TC2 issue--it's reporting implausible network utilization:
...
but obviously not multiple gigabits/sec.

Thanks @danb35. This has been picked up under TC-1829.

 

[morganL]

Thanks @danb35. This has been picked up under TC-1829.

Looks like there might be some misreading between MBytes/sec and Mbit/s. Factor of 8 difference.

It's not clear that "Octets Rx/Tx"... is the right terminology on the TrueNAS UI.

 

[Basil Hendroff]

Looks like there might be some misreading between MBytes/sec and Mbit/s. Factor of 8 difference.

There are several interrelated issues here. See Network speed reporting issues.

It's not clear that "Octets Rx/Tx"... is the right terminology on the TrueNAS UI.

Ooh. I hadn't noticed this before.

 

[morganL]

I added to the ticket... though it might end up creating another ticket.

 

[Basil Hendroff]

TC 2 issues table from post #43 brought forward and updated.

Posts	JIRA ticket	Description	Fix Version	Test Result
#3	TC-1761	Used space on system cards are reported as a whole number	2.1 [1]	​
#5	TC-1772	Multiple time formats in use	2.1 [1]	​
#6	TC-1784	Active shares not shown in the dashboard	2.1 [1]	​
#8	TC-1783	SMR Alert	2.1 [1]	​
#15	TC-1770	Calendar improvement for report generation	2.2	​
-	TC-1812	DNS lookup failure	2.1 [1]	​
-	TC-1817	New indicators replace activity wheel	2.1 [1]	​
-	TC-1829	Network speed reporting issues	2.1 [1]	​
-	TC-1833	Clumsy resolving long alert messages	2.1 [1]	​

[1] Resolved in the TC nightly with middleware version Master-20211111.

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.

 

[Basil Hendroff]

The TC nightly with middleware version Master-20211111 fixes a whole bunch of stuff. Refer to the previous post for details.

@Heracles Based on my understanding of ticket TC-1812, I can only assume the DNS lookup issue has been resolved with this nightly. Are you able to test and confirm? If it's still an issue, you may need to get the ticket reopened.

The latest nightly does introduce a new issue, which I'll discuss in the next post.

 

[Basil Hendroff]

Alerts don't stay resolved in the latest nightly Master-20211111. This issue did not exist in the previous nightly. This issue is reproducible for TC on both CORE and SCALE. Browser caches were cleared before testing. I sense all these symptoms are related to the same issue.

These are some of the symptoms of the issue:

1. Not all issues are resolved if all alerts are cleared.
2. Resolved issues also reappear after a while.
3. Resolved issues remain visible, but dimmed out.
4. Multiple occurrences of the same issue.
5. The alerts bubble never clears.

Reported under JIRA ticket TC-1870.

Not all issues are resolved if all alerts are cleared.

Resolving all alerts...

... doesn't produce the desired result.

Resolved issues also reappear after a while.

Here's a screengrab a while later. Note how the numbers have crept up. I can confirm these are not new issues, but previously resolved issues.

These numbers will keep increasing. It seems an issue that was resolved begins to reappear again and again and that's what contributes to the numbers incrementing.

Resolved issues remain visible, but dimmed out.

I'm pretty sure that once an issue was resolved for a specific system in the previous nightly, it disappeared from the alerts card for the system. This isn't happening with the latest nightly.

Multiple occurrences of the same issue.

For specific systems, note how the same issue is repeated (identical timestamp)...

The alerts bubble never clears.

Here's an example of a system where alerts on the Alerts card were cleared, but the alerts bubble remains non-zero.

 

[Basil Hendroff]

TC 2 issues table from post #51 brought forward and updated.

Posts	JIRA ticket	Description	Fix Version	Test Result
#15	TC-1770	Calendar improvement for report generation	2.2
#53	TC-1870	Alerts do not stay resolved	2.1

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.

 

[Heracles]

I can only assume the DNS lookup issue has been resolved with this nightly. Are you able to test and confirm?

Hi @Basil Hendroff,

Here, I do not work with nightlies. Ticket said that it is to be fixed in 2.1... If you wish to test yourself, the complete procedure to reproduce the bug is documented in the ticket. So create the conditions, have the bug, update to the nightly you are looking for and see if that fixes the problem. If it does not, fix the DNS manually to the internal IP using Docker and see if that workaround is still required.

 

[morganL]

TC 2.1 is making progress and will be available before Xmas.

 

[Heracles]

Latest version 2.1 does not even start over here...

Code:

[Jan  5 05:07:01] Starting Middleware

[Jan  5 05:07:01] Starting InfluxDB

[Jan  5 05:07:01] Switching Log Level -> warning

[Jan  5 05:07:01] Starting TrueCommand Middleware...

  Version: 2.1

  BuildDate: 20220104

time="2022-01-05T05:07:02Z" level=info msg="reset OCSP cache file. /root/.cache/snowflake/ocsp_response_cache.json" func="gosnowflake.(*defaultLogger).Infof" file="log.go:104"

time="2022-01-05T05:07:02Z" level=info msg="reading OCSP Response cache file. /root/.cache/snowflake/ocsp_response_cache.json\n" func="gosnowflake.(*defaultLogger).Infof" file="log.go:104"

time="2022-01-05T05:07:02Z" level=error msg="failed to open. Ignored. open /root/.cache/snowflake/ocsp_response_cache.json: no such file or directory\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"

[Jan  5 05:07:02][ERROR] Unable to start webserver: exit status 1

[Jan  5 05:07:03] Closing down InfluxDB

Cleaning up...

[Jan  5 05:07:03] InfluxDB Stopped

[Jan  5 05:07:03] InfluxDB failing to run for more than 60s. Check the influxdb.log[.prev|.old] files for details

Sounds like I will have to create another Jira ticket for that...

I first updated my existing container and the new one did not start.
I then created a new one while mapping the previous Data directory and again, does not even start.
Just tried to start it without any mounted volume and I still have the error message but at least. the container did start.

I will have to do more debug about that one but clearly not a straight direct upgrade.

 

[Heracles]

Hey @Basil Hendroff,

You can add bug TC-1913 to your list if you wish...

 

[morganL]

Hey @Basil Hendroff,

You can add bug TC-1913 to your list if you wish...

https://jira.ixsystems.com/browse/TC-1913

For others info - this bug is related to custom certificates.

 

[aervin]

A patch for TC-1913 was merged today and is available on the :nightly images. As of this post, the nightly image is identical to 2.1 except for the addition of the custom cert patch. A 2.1.1 maintenance release is forthcoming. Thanks for reporting, @Heracles !