Cobia shouldn't have made it to the update train

[CheeryFlame]

Just to add my 2C to this.

While I understand your point of view let me just end my point with this:

1. Truecharts is running fine
2. I'm updating to Cobia
3. Truecharts isn't snappy anymore

There are so many Truecharts users and I think it's only fair for them to want ix to support popular external catalogs that are doing what ix devs can't. In the 90's no cable company would have think to offer internet access nor phone lines. Well here we are they're offering it and will probably only be offering internet access since cable TV is dying. I'm not saying it's the exact same thing that'll happen to TrueNAS, obvisouly it's comparing apples and oranges. But there's one thing that stand; what we know is constantly evolving. Home users and small enterprises aren't interested in setting up complicated hypervisors. I constantly hear people all over the internet saying they're waiting for truenas apps to be more stable until they switch from their current setup.

If ix doesn't want to get too much involved in their kubernetes and apps implementation at the moment, at least, don't make it difficult for the third parties doing it for free.

 

[DaSnipe]

Just to add my 2C to this. For somewhat obvious reasons we have to be careful with dealing with issues from 3rd party catalogs. This is why we have such a heavy disclaimer when you enable other catalogs:

View attachment 72401


I agree, it is annoying to you and other users when something isn't working while using TrueCharts. But look at it from our perspective. We can only support (and test) what we control. External things open up the door to all kinds of bad behavior's that we cannot account for all the time. We understand its an open-source product and people will do all kinds of "fun and creative" things with it, but we have to be very judicious about where we spend precious developers hours, and debugging issues caused or exposed by unsupported external entities is always going to be a challenge.

Similar story to the message from William about the scripting. A poor analogy is that it's like taking your car back to the dealer for an issue, after you've been replacing factory engine components with your own custom parts. We can only support the software as it was delivered since local modifications obviously run a high risk of breaking things in new and interesting ways.

Remember when you put on the "I Void Warranties" shirt, there is power, but also great responsibility that comes with it.

View attachment 72404

It's a shame, SCALE can do way more if you'd put more time into adding features to apps that are either hidden (CLI commands for app backups) or disabled (metrics, etc). I get it, Apps isn't the priority for iX, and home users will just run other platforms or solutions otherwise, but they don't generate support contracts either.

 

[danb35]

Apps isn't the priority for iX

Apps are really the only reason to be using SCALE rather than CORE--unless you believe that iX is leaving CORE to die on the vine (which they continue to deny).

 

[sfatula]

Heavyscript exists to fill some holes in Scale. Put the equivalent functionality into Scale, and problem goes away as far as 3rd party tool support issues. I think that's a lot easier issue to solve than 3rd party catalogs.

 

[truecharts]

We want to highlight that only a very-very small minority of our users is experiencing performance issues this bad.
We've not been able to verify if our catalog has any impact on it or not.

Most likely the amount of Apps installed and which ones, has a significantly bigger impact than the mere existence of our catalog on the system involved. From our end: Cannot reproduce.

On the topic of support: We don't expect iX to provide proactive bugtracing or support for our Apps.
It's not fair to expect such significant expendature of limited developer resources. As it's quite clear that the base system itself needs so much more work, that tools like Heavyscript have become essentials.

If you encounter issues with our Apps, please report them to us directly.
If that happens to be an issue caused by iX-Systems code, our staff can also file a ticket with iX-Systems to lay out the bug or give you instructions on what to report and how to report it. This way we can prevent many issues that do not require iX-Systems attention, from putting a burden on their developers.

 

[Kris Moore]

Saw it, did it, but it doesn't work--ARC is still maxing out at half my RAM.

Yea, that was my bad, instructions were wrong for Cobia. Fixed in the original thread now.

 

[Kris Moore]

Heavyscript exists to fill some holes in Scale. Put the equivalent functionality into Scale, and problem goes away as far as 3rd party tool support issues. I think that's a lot easier issue to solve than 3rd party catalogs.

Which functionality is it that you use in particular with heavyscript? Auto-updating and backup/restore primarily? Not bad ideas, good chance we'll do that natively eventually.

 

[CheeryFlame]

Which functionality is it that you use in particular with heavyscript? Auto-updating and backup/restore primarily? Not bad ideas, good chance we'll do that natively eventually.

Since heavyscript already exists in the meantime wouldn't it be better to have a real kubernetes backup system which could restore only 1 application instead of the whole app pool? Also one that would support cnpg based apps.

I know you didn't asked me but here are the top features I'm using heavyscript for

• Automated app backup
• Automated app update
• App backup restore
• Mount/Unmount PVC storage
• Access pod shell in ssh

 

[truecharts]

Since heavyscript already exists in the meantime wouldn't it be better to have a real kubernetes backup system which could restore only 1 application instead of the whole app pool? Also one that would support cnpg based apps.

I know you didn't asked me but here are the top features I'm using heavyscript for

• Automated app backup
• Automated app update
• App backup restore
• Mount/Unmount PVC storage
• Access pod shell in ssh

CNPG inherently only supports its own backup solution.
It's best to only ask for native kubernetes features from iX, not for operators (aka kubernetes extentions) they don't ship.

 

[tannisroot]

Yea, that was my bad, instructions were wrong for Cobia. Fixed in the original thread now.

Can I ask if the original instructions were actually completely wrong or if the CLI option that was talked about in the original post just doesn't work on Cobia currently and will be fixed in the next point release, and while this is the case people can use the post-init script?

 

[Kris Moore]

Since heavyscript already exists in the meantime wouldn't it be better to have a real kubernetes backup system which could restore only 1 application instead of the whole app pool? Also one that would support cnpg based apps.

I know you didn't asked me but here are the top features I'm using heavyscript for

• Automated app backup
• Automated app update
• App backup restore
• Mount/Unmount PVC storage
• Access pod shell in ssh

Yes, if we go to the trouble of doing this I'd also prefer a backup all or backup some capability as well. Auto updating is on my wish list in general for TrueNAS itself as well.

 

[Kris Moore]

Can I ask if the original instructions were actually completely wrong or if the CLI option that was talked about in the original post just doesn't work on Cobia currently and will be fixed in the next point release, and while this is the case people can use the post-init script?

Stick to the updated instructions for Cobia, the post init script to apply your value should be persistent. There are other places at bootup where we may still be overriding defaults, at least until the proper fix lands in 24.04 and sets the defaults like CORE again.

 

[CheeryFlame]

Just got an error while the system was super slow after redeploying an app that I imported back it's cnpg database backup.

I made a Jira bug report and saw that my restore process on the app probably failed

Critical ​

Failed to check for alert ScrubPaused: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker res = MIDDLEWARE._run(*call_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run return self._call(name, serviceobj, methodobj, args, job=job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/client/client.py", line 292, in __init__ raise ClientException('Failed connection handshake') middlewared.client.client.ClientException: Failed connection handshake """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/alert.py", line 800, in __run_source alerts = (await alert_source.check()) or [] ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/alert/source/scrub_paused.py", line 18, in check for pool in await self.middleware.call("pool.query"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call return await self._call( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1341, in _call return await methodobj(*prepared_call.args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf res = await f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 156, in query result = await self.middleware.call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call return await self._call( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1341, in _call return await methodobj(*prepared_call.args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 169, in query result = await self._queryset_serialize( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 221, in _queryset_serialize return [ ^ File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 222, in await self._extend(data, extend, extend_context, extend_context_value, select) File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 235, in _extend data = await self.middleware.call(extend, data, extend_context_value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call return await self._call( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1352, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/pool.py", line 188, in pool_extend pool |= self.middleware.call_sync('pool.pool_normalize_info', pool['name']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1420, in call_sync return self.run_coroutine(methodobj(*prepared_call.args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in run_coroutine return fut.result() ^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf res = await f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/pool.py", line 152, in pool_normalize_info if info := await self.middleware.call('zfs.pool.query', [('name', '=', pool_name)]): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call return await self._call( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1349, in _call return await self._call_worker(name, *prepared_call.args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1355, in _call_worker return await self.run_in_proc(main_worker, name, args, job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1267, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ middlewared.client.client.ClientException: Failed connection handshake​

I think the logs from the app says it wont connect to kubernetes for some reason.

 

[Kris Moore]

Will have to look at your debug in more detail. That error isn't super helpful, just means it couldn't establish connection, not why it went out to lunch in the first place.

 

[CheeryFlame]

Will have to look at your debug in more detail. That error isn't super helpful, just means it couldn't establish connection, not why it went out to lunch in the first place.

Aight thanks, the bug report is attached so the answer must be in there!

 

[sfatula]

Which functionality is it that you use in particular with heavyscript? Auto-updating and backup/restore primarily? Not bad ideas, good chance we'll do that natively eventually.

Automatic app backup of course
Self update
List DNS names, useful to get the internal names when communication between apps needed

 

[Kris Moore]

Aight thanks, the bug report is attached so the answer must be in there!

Thanks! One of our engineers took a look and it was a duplicate of another issue, which is already fixed in the upcoming 23.10.1, slated for 12/12.

[NAS-124614] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[CheeryFlame]

So the latest update had helped a little in getting the App page to open but editing apps is still a huge pain. It just took me 10-15 minutes to edit one application and then the UI completely crashed. I wasn't even able to login because the 2fa code would change since it's taking so much time to load after you enter your 2fa code. It's been more than a month now and this is a critical issue that must be resolved.

Where is it at now?

 

[Kris Moore]

The bug mentioned was shipped in 23.10.1. However, crashing UI sounds like an entirely different issue and not one I'm seeing reported (yet). Do you have any details or a bug ticket on this one specifically? Might be something unique to your use-case or charts that we're not seeing widespread.

 

[CheeryFlame]

The bug mentioned was shipped in 23.10.1. However, crashing UI sounds like an entirely different issue and not one I'm seeing reported (yet). Do you have any details or a bug ticket on this one specifically? Might be something unique to your use-case or charts that we're not seeing widespread.

It's widespread but only concern users with many charts. I don't know how many charts is the breaking point but I have over 50. I suspect it's only happening with charts from 3rd party catalogues, otherwise ix would have found out by now. It's blatant. I even considered switching to Proxmox or something else but it would be too much work for something that I'm confident ix will solve. I'm not certain how to make the Jira report for this one since it's not showing an alert like the one that's been patched since. Do you want me to edit an app and screen record a video to show the behaviour and then right after submit the bug report?