TrueCommand 2 Issues

[aervin]

Nightlies have been "stuck" for a number of days. A fresh nightly was just deployed with the most recent changes slated for 2.0.1.

 

[Basil Hendroff]

Tests run on TC 2 nightly 20210616

If there are sad faces in the table below, refer to the JIRA ticket for details. Multiple sad faces denote the problem seems worse. Single sad faces signify no change or a partial fix. Where a test is in progress, it will take some time to make a determination. As results come in, I'll update the entry.

Post​	JIRA ticket​	Description​	Fix Version​	Test Result​
#2​	TC-1760​	Alert bubble not being cleared immediately	2.0.1​	[2]​
#3​	TC-1761​	Used space on system cards are reported as a whole number	2.1​	-​
#4​	TC-1762 TC-1781​	Can't configure the number of config db backups.	2.0.1​	[1]​
#5​	TC-1772​	Multiple time formats in use	2.1​	-​
#6​	TC-1721 TC-1784​	Active shares not shown in the dashboard	2.1​	-​
#7​	TC-1763​	Updating wheel missing for resilver and replication tasks	2.0.1​	[1]​
#8​	TC-1764 TC-1783​	SMR Alert	2.1​	-​
#9​	TC-1765​	Mixed rate units	2.0.1​	[1]​
#12​	TC-1767​	Alert Rules UI issues	2.0.1​	[2]​
#13​	TC-1768​	Storage Navigator and Datasets card issues	2.1​	-​
#14​	TC-1769​	Disk activity reporting appears broken	2.0.1​	[1]​
#15​	TC-1770​	Calendar improvement for report generation	2.1​	-​
#19​	TC-1771​	Excessive space prediction alerts	2.0.1​	[1]​

[1] Fixed in TC 2 nightly 20210616
[2] Fixed in TC 2 nightly 20210617

 

[Basil Hendroff]

TC 2 nightly 20210617 was released overnight and fixed several more issues identified in this thread. Refer to the previous post for details. Just waiting on some feedback on tickets TC-1721 and TC-1764.

 

[Basil Hendroff]

TC 2.0.1 is hot off the press. It's fixed a whole bunch of glitchy stuff identified in this thread and summarised in post #22 above. There have been some TC documentation updates as well. Upgrade from TC 2.0 if you haven't already. Thanks to @kenmoore, @aervin and the rest of the team for resolving the issues and expediting the release of TC 2.0.1 as quickly as they have. TC 2.0 was only officially released just over two weeks ago. Well done guys!

If you still have a significant number of FreeNAS servers in your fleet, make sure you still run TC 1.3.2 to support these as they are not fully supported under TC 2.x. More info in this post. If you have a mix of FreeNAS and TrueNAS servers, you can run both TC 1.3.2 and TC 2.x in parallel as long as they don't point to the same database.

TC issues identified in this thread that are still lurking in TC 2.0.1 are summarised in the table below. At this stage, the intent is to have these addressed in a future TC 2.1 release.

Post	JIRA ticket	Description	Fix Version	Test Result
#3​	TC-1761​	Used space on system cards are reported as a whole number	2.1​	-​
#5​	TC-1772​	Multiple time formats in use	2.1​	-​
#6​	TC-1784​	Active shares not shown in the dashboard	2.1​	-​
#8​	TC-1783​	SMR Alert	2.1​	-​
#15​	TC-1770​	Calendar improvement for report generation	2.1​	-​
-​	TC-1759​	TC 2 insecure NAS portal?	2.0.2​	-​
#25-#27​	TC-1793​	Net Speed, Disk Write, and Disk Used graphs are reporting inflated numbers		-​

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it doesn't fall through the cracks.

 

[hairlesshobo]

Not sure if this has already been reported or not. I read all the issues in this particular thread and it was not present. Apologies if this is a know issue and I am reposting it.

I am using TC 2.0.1 and TrueNAS core 12.0-U4.1 and I am noticing that the network and disc metrics appear to be much too high sometimes. It seems to be about 50/50 whether the numbers are accurate or whether they are 2-3x too high. An example in the screenshot below. My system has 2x 1gbit ethernet with multichannel SMB enabled. One of those NICs is a pci (not PCIe) card.. realistically I can only achieve 180-190 MiB/s sustained transfer rates. If you look below, you can see it says 12.1GB/s net speed and 499MiB/s disk write. Nothing in my hardware is capable of achieving anywhere near these speeds. If you look at the disk and net graph, you can see that for a while it was reporting correctly, then suddenly they both jumped to values much higher than is accurate.

Please let me know if you need any further details or clarification as to what I am seeing.

Steve

 

[Basil Hendroff]

If you look below, you can see it says 12.1GB/s net speed

The system tile does appear to be reporting the wrong figure (12.1 Gb/s). Based on the Net Used tile, 1500 Mib/s = 1.57 Gb/s, which is under 2Gb/s for multichannel SMB so that figure appears to be okay.

My system has 2x 1gbit ethernet with multichannel SMB enabled.

I assume this has been correctly configured on TrueNAS? Setting up SMB 3 multichannel on FreeNAS

Please let me know if you need any further details or clarification as to what I am seeing.

There are some missing details and things you can try first.

1. Try the latest TC nightly and see if the issues persist. If it does, make a note of the nightly version (Settings (gear icon) > Administration > About > Middleware version) and then continue down the list.
2. A detailed h/w description. The forum rules are your guide here.

3. Link your post above to a JIRA ticket (Report a Bug in the image above) and let me know what the ticket number is. I'll then add the issue to the table so it remains visible within the community space.

 

[hairlesshobo]

Apologies for not providing enough detailed information.

This issue has been created in Jira as TC-1793:

https://jira.ixsystems.com/browse/TC-1793

​

I have confirmed that the issue is still present in the latest nightly build (Master-20210720).

Multi-channel SMB is correctly setup on my TrueNAS core system and I have no trouble achieving a multi-link speed when transferring to/from the system from two other systems on my network that also have a dual link connection. Performance is not the concern here, the numbers being reported by TC are the concern.

The TrueNAS system details are as follows:

Motherboard: ASUS P5K-VM
CPU: Intel Core 2 Quad Q9550 @ 2.83GHz
RAM: 6GB DDR2
Hard Drives:
5x 4tb drives drives in RAID-Z1

• 2x Hitachi 4tb 5700 RPM drives (model HDS5C4040ALE630)
• 2x WD Red 4tb 5400 RPM drives (model WDC WD40EFRX)
• 1x WD Red Plus 4tb 5400 RPM drive (model WDC WD40EFZX)

OS Drive: 64GB mSata SSD (model JAJMS600M64G )
HD Controllers:

• On board SATA controller in AHCI mode (OS drive and 3 platter drives)
• I/O Crest 4 Port SATA III PCI-e 2.0 x1 Controller (chipset: Marvell 9215) (other 2 platter drives)

Network cards:

• On-board Marvell Yukon 88E8056 Gigabit Ethernet
• PCI Intel(R) PRO/1000 Network Connection

I have attached a few additional screenshots to show the issue I am seeing. With only a single transfer being made to this system from a windows system, the network speed and disk write speed are both inflated.

Here is a screenshot of the copy in progress from the Windows machine, you can see that the TrueNAS box is having no trouble sustaining ~170MiB/s write speed, which is perfectly acceptable for my use case and honestly, higher performance than I expected from this underpowered build (yay TrueNAS!)

Now, during this same transfer, you can see what TC is reporting:

On the left, Net Speed shows 11.8Gb/s and Disk Write shows 483MiB/s. The Disk Used graph is also reporting the inflated write speed values, whereas Net Used appears to be accurate.

Now, I don't expect the numbers between Windows and TC to be exact by any means, but I would figure within +/- 10% or so would be expected.

I hope I have everything covered this time. Let me know if you need more info.

 

[Heracles]

Here are a few more; one of them looking a lot like an old one I already posted about and that was fixed long time ago...

1-TrueCommand does not let me upload my local CA. After selecting it as a file or copy-paste the text, the IMPORT button remains greyed out.

2-TrueCommand does not respect its TLS configuration. When I enter my TrueNAS's IP address, TrueCommand connects to it. The problem is, the certificate is valid only for the FQDN name and the old IP address which is not the same anymore. TrueCommand is configured to enforce TLS and the check box to ignore SSL errors is empty (as well as the one with hostname mismatch).

3-TrueCommand does not resolve DNS names. When I enter a system's name, TrueCommand does just plain nothing. Even TCPDump does not detect any reaction. TrueCommand reacts only to IP address. The container is able to resolve names and connect to them. I entered in a shell in the container and used openssl to connect my TrueNAS' TCP port 443 calling it by its name. It worked from openssl inside the container but the GUI is unable to do it.

TrueCommand is running from a Linux Docker host outside TrueNAS. Container's image has been updated (image created 2021-06-22 09:32:52) and is the one designated as ":latest" as of now.

The TrueNAS system I try to connect first is Atlas (description in my signature).

 

[Basil Hendroff]

@Heracles Are 2 and 3 in any way related to TC 2 insecure NAS portal? If so, this has been reported and recorded in the table in post #24. You may want to add a note to JIRA ticket TC-1759.

In order to include unique issues in the table, I'll need the JIRA ticket numbers as well. Without these, issues reported just through the forum tend to slip through the cracks. Core maintainers for TC use JIRA for tracking bug fixes.

 

[Heracles]

Hey @Basil Hendroff,

--Related--, maybe. In your case, it looks like TC took the DNS name, resolved it to an IP address and stopped using the name from there. That is why it returned you an IP-based URL instead of the DNS name. So it is related in the way that this is TC not handling names vs IP properly.

But it is not the same because for #2, I did used the IP to point to my Atlas server. As such, to re-use the IP instead of the name is normal. What is not normal is for TC to tolerate an invalid certificate. You ended up with an invalid certificate because TC pointed you to the IP. Here, I pointed TC to an IP with an invalid certificate and TC took it without a word.

As for #3, it is again TC not handling DNS names properly. The container can do "openssl s_client -connect atlas:443" and the socket is successful. TC's web interface can not. Again, your "invalid" certificate was client side, the client being mislead by TC. Here, my problems are TC's side, when it does not resolve DNS or validate certificates itself.

As such, I consider they are separate cases despite all being related to IP vs DNS vs Certificates.

 

[Heracles]

Also, I created Jira ticket TC-1799. I put it as blocker because it completely prevent me from using TC at all and it is a major flaw not to check certificates at all when you are told to do it.

 

[Basil Hendroff]

TC 2 issues table from post #24 brought forward and updated.

Posts	JIRA ticket	Description	Fix Version	Test Result
#3	TC-1761	Used space on system cards are reported as a whole number	2.1	-
#5	TC-1772	Multiple time formats in use	2.1	-
#6	TC-1784	Active shares not shown in the dashboard	2.1	-
#8	TC-1783	SMR Alert	2.1	-
#15	TC-1770	Calendar improvement for report generation	2.1	-
-	TC-1759	TC 2 insecure NAS portal?	2.0.2	-
#25-#27	TC-1793	Net Speed, Disk Write, and Disk Used graphs are reporting inflated numbers	2.0.2	-
#28-#31	TC-1799	Unable to configure local CA or systems using DNS names	2.0.2	-

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.

 

[morganL]

TC 2 issues table from post #24 brought forward and updated.

Posts	JIRA ticket	Description	Fix Version	Test Result
#3	TC-1761	Used space on system cards are reported as a whole number	2.1	-
#5	TC-1772	Multiple time formats in use	2.1	-
#6	TC-1784	Active shares not shown in the dashboard	2.1	-
#8	TC-1783	SMR Alert	2.1	-
#15	TC-1770	Calendar improvement for report generation	2.1	-
-	TC-1759	TC 2 insecure NAS portal?	2.0.2	-
#25-#27	TC-1793	Net Speed, Disk Write, and Disk Used graphs are reporting inflated numbers	2.0.2	-
#28-#31	TC-1799	Unable to configure local CA or systems using DNS names	2.0.2	-

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.

Basil, since you are doing such a great job of capturing the issues, I'd be very keen to understand the priorities you consider each has. For example:

High = MUST FIX URGENTLY - it causes real issues​

Medium - Important for ease-of-operations and productivity​

Low = Nice to have, but cosmetic​

Some issues will be Medium-High and some will be Medium-Low (So rating from 1 (High) to 5 (Low) would also work.

Sometimes, we have to prioritize a new feature over a Low priority bug, but ideally we get it all done. Other members might argue with your rating (only bother if you want it higher), but we can also learn from that.
Thanks

 

[Heracles]

Hey @morganL,

Before we can do such a thing, it would be important for IXSystems to be consistent.... First ticket I created was TC-1328, part of which was being unable to load a custom CA. I made it HIGH at that time and @kenmoore increased it to BLOCKER. When I created TC-1799 for 3 problems including being again unable to load a custom CA, I put it as BLOCKER for that very reason. The thing is, this time @kenmoore reduced it to HIGH.

Another question would be how to classify unconfirmed bugs / cases that we can not reproduce yet. Here, I am unable to configure a system calling it by its DNS name while @kenmoore told me that he can on his side. I still work with him about it but as of now, how should that one be classified ? Not to be able to configure by DNS name would deserve to be a blocker but until we can figure out why I can not on my side, we can not stop completely everything else in the name of this still unique case...

As a last point, @kenmoore and I just detected a case where the WebUI does not reflects its underlying settings. My WebUI said that SSL checks were enforced when they were not. To let user think they are secure when they are not is a big deal but again, the level for such a bug is highly dependent of personal opinion. For a security engineer like myself, it is surely worst (HIGH) than it would be for you and all of your clients using self-signed default TLS certificates (LOW / MEDIUM)....

I will let @Basil Hendroff offer his own view about this but for me, I doubt the severity we see as users can (should?) be translated as priorities for the provider.

 

[Basil Hendroff]

@morganL The overarching issue I've been grappling with in this thread is that issues reported in JIRA tend to disappear from public view. This is true for TN CORE and SCALE as well. Community members spend most of their time wandering the corridors of the forum. What's out-of-sight is out-of-mind for members. This tends to be the case when it comes to JIRA. Developers, on the other hand, rely on JIRA for managing work and, I'm guessing, would spend most of their time in that space.

By linking posts/threads to JIRA tickets and summarising these in a table, I keep those issues very much alive in the community space. This is easy enough to do for TC; much harder to achieve for TN CORE and SCALE. The approach I've taken is a very clinical one. There's nothing controversial or subjective in the table.

There are several reasons I'm not keen on adding priorities to the table:

1. Priorities are subjective. What one person thinks is a high priority; another person will think is not. This could end up being a source of heated debate within the community space and, quite frankly, a waste of time and energy. I believe what's more telling than the priority is the fix version.

For instance, in the thread FreeNAS 11.3 Released, specifically post #63, I raised the suggestion of a Corral style calendar for scheduling. This got lots of attention at the time and gained some traction and JIRA ticket NAS-105084 was born. There was a time when this ticket had the most number of votes. However, the fix version was pushed out to 12 and then subversions of 12 and now it's 13. Now, this is annoying. I won't be surprised if it continues to be pushed out further. SCALE, I'm guessing, and understandably so, has taken precedence. This is a good example of 'out-of-sight, out-of-mind'. It requires a post like this to place it front and centre again within the community space, otherwise, it tends to disappear into the ether.

2. The built-in levelling is working. If you consider the original table in post #22, I've left it to @kenmoore to work out what his team needed to address first. I'm satisfied that he's prioritised and addressed the most pressing issues correctly. This allowed his team to deliver TC 2.0.1 a fortnight after the release of TC 2.0. Issues that remain are tagged to be addressed in a future TC 2.1 (or later) release as indicated in the most recent table in post #32. Again, I'm reasonably happy with his assessment. I am conscious that Ken has limited resources and internal priorities that he also has to manage. I'm confident that the priorities he's set are in the best interest of all stakeholders.

Until a couple of days ago, the newest items (last two items) in the table reported by community members did not have a fix version. Ken has made a judgement call and decided that these will be fixed in an up-and-coming TC 2.0.2 release. So, in his mind, these issues needed to be brought forward. I don't think the forum members who reported these issues would disagree with his assessment.

3. Reassess priority by exception. If a community member vehemently disagrees with the priority or fix version, they can argue the toss within the JIRA ticket.

 

[morganL]

Hey @morganL,

Before we can do such a thing, it would be important for IXSystems to be consistent.... First ticket I created was TC-1328, part of which was being unable to load a custom CA. I made it HIGH at that time and @kenmoore increased it to BLOCKER. When I created TC-1799 for 3 problems including being again unable to load a custom CA, I put it as BLOCKER for that very reason. The thing is, this time @kenmoore reduced it to HIGH.

Another question would be how to classify unconfirmed bugs / cases that we can not reproduce yet. Here, I am unable to configure a system calling it by its DNS name while @kenmoore told me that he can on his side. I still work with him about it but as of now, how should that one be classified ? Not to be able to configure by DNS name would deserve to be a blocker but until we can figure out why I can not on my side, we can not stop completely everything else in the name of this still unique case...

As a last point, @kenmoore and I just detected a case where the WebUI does not reflects its underlying settings. My WebUI said that SSL checks were enforced when they were not. To let user think they are secure when they are not is a big deal but again, the level for such a bug is highly dependent of personal opinion. For a security engineer like myself, it is surely worst (HIGH) than it would be for you and all of your clients using self-signed default TLS certificates (LOW / MEDIUM)....

I will let @Basil Hendroff offer his own view about this but for me, I doubt the severity we see as users can (should?) be translated as priorities for the provider.

Looks like I stirred the hornets nest....

"Blocker" is a temporary status... it only applies to a specific upcoming release that is in process and we decide not to ship the release without it. "High" is a useful priority.

Agree that we need to understand the impact of a bug before we can decide its classification. If it works in one location and not another, then we don't know the % of users impacted.

Yes, users will have different views on priority.. Its healthy to have an argument about them and the reasons. From a selfish perspective, I learn alot from reading these and hopefully make better decisions with the team.

 

[aervin]

Seconded. I learn a lot from the TC forum. Happy to work with the community to refine the dev process.

 

[morganL]

@morganL The overarching issue I've been grappling with in this thread is that issues reported in JIRA tend to disappear from public view. This is true for TN CORE and SCALE as well. Community members spend most of their time wandering the corridors of the forum. What's out-of-sight is out-of-mind for members. This tends to be the case when it comes to JIRA. Developers, on the other hand, rely on JIRA for managing work and, I'm guessing, would spend most of their time in that space.

By linking posts/threads to JIRA tickets and summarising these in a table, I keep those issues very much alive in the community space. This is easy enough to do for TC; much harder to achieve for TN CORE and SCALE. The approach I've taken is a very clinical one. There's nothing controversial or subjective in the table.

There are several reasons I'm not keen on adding priorities to the table:

1. Priorities are subjective. What one person thinks is a high priority; another person will think is not. This could end up being a source of heated debate within the community space and, quite frankly, a waste of time and energy. I believe what's more telling than the priority is the fix version.

For instance, in the thread FreeNAS 11.3 Released, specifically post #63, I raised the suggestion of a Corral style calendar for scheduling. This got lots of attention at the time and gained some traction and JIRA ticket NAS-105084 was born. There was a time when this ticket had the most number of votes. However, the fix version was pushed out to 12 and then subversions of 12 and now it's 13. Now, this is annoying. I won't be surprised if it continues to be pushed out further. SCALE, I'm guessing, and understandably so, has taken precedence. This is a good example of 'out-of-sight, out-of-mind'. It requires a post like this to place it front and centre again within the community space, otherwise, it tends to disappear into the ether.

2. The built-in levelling is working. If you consider the original table in post #22, I've left it to @kenmoore to work out what his team needed to address first. I'm satisfied that he's prioritised and addressed the most pressing issues correctly. This allowed his team to deliver TC 2.0.1 a fortnight after the release of TC 2.0. Issues that remain are tagged to be addressed in a future TC 2.1 (or later) release as indicated in the most recent table in post #32. Again, I'm reasonably happy with his assessment. I am conscious that Ken has limited resources and internal priorities that he also has to manage. I'm confident that the priorities he's set are in the best interest of all stakeholders.

Until a couple of days ago, the newest items (last two items) in the table reported by community members did not have a fix version. Ken has made a judgement call and decided that these will be fixed in an up-and-coming TC 2.0.2 release. So, in his mind, these issues needed to be brought forward. I don't think the forum members who reported these issues would disagree with his assessment.

3. Reassess priority by exception. If a community member vehemently disagrees with the priority or fix version, they can argue the toss within the JIRA ticket.

Thanks Basil, Appreciate the response and the clarity of thought.

The Jira ticket disappearing issue is an interesting one. I think they might disappear for two reasons:
1) Duplicate tickets... we try to consolidate
2) Customer sensitive information

Glad that you are comfortable with the current TrueCommand process. For me "priority" is important for scheduling. Sometime lower priority issues get scheduled first because the fix is trivial. Some time high priority issues have to wait, because the fix requires a major rewrite. However, its always good to know user priorities..... even if users can't agree. However, your tables are useful even without priorities.

On the Corral style calendars... I'll have to do some digging since I've never used Corral. Most of the new UI work is in SCALE first. Have there been any improvements there?

 

[Basil Hendroff]

The Jira ticket disappearing issue is an interesting one. I think they might disappear for two reasons:
1) Duplicate tickets... we try to consolidate
2) Customer sensitive information

I wish it were this simple. These are valid reasons for closing tickets, but not what I'm alluding to. Unfortunately, the reality is quite different. For me, it's best summed up by post #3 in the thread Issues raised in CORE. Fix proposed in SCALE?!. I believe the root problem stems from the disconnect between the community space (The Forum) and the dev space (JIRA). The table I provide in this thread is a 'hack' to try and bridge the gap between these two spaces for TC issues.

On the Corral style calendars... I'll have to do some digging since I've never used Corral. Most of the new UI work is in SCALE first. Have there been any improvements there?

Unfortunately, no. The SCALE UI aligns reasonably closely with the CORE UI. The calendar UI is unique to Corral (FreeNAS 10). Task scheduling in SCALE and CORE are clumsy in comparison to scheduling under Corral. To appreciate some of the benefits of the calendar, see post #77 in FreeNAS 11.3 Released. For me, the calendar does for tasks what TrueCommand does for servers i.e centralise activities through a dashboard.

 

[Basil Hendroff]

TC 2 issues table from post #32 brought forward and updated.

Posts	JIRA ticket	Description	Fix Version	Test Result
#3	TC-1761	Used space on system cards are reported as a whole number	2.1	-
#5	TC-1772	Multiple time formats in use	2.1	-
#6	TC-1784	Active shares not shown in the dashboard	2.1	-
#8	TC-1783	SMR Alert	2.1	-
#15	TC-1770	Calendar improvement for report generation	2.1	-
-	TC-1759	TC 2 insecure NAS portal?	2.0.2	​
#25-#27	TC-1793	Net Speed, Disk Write, and Disk Used graphs are reporting inflated numbers	2.0.2	-
#28-#31	TC-1799	Unable to configure local CA or systems using DNS names	2.0.2	​

As indicated in the OP, if you're certain you've identified a TC problem, consider adding or referencing it in this thread, and linking it to a JIRA ticket. I'll update the table to include it so that it remains visible in the community forum space and doesn't fall through the cracks.