Can't Launch Docker Images Anymore

[Sunoo]

I was able to install Docker images in the past, but now when I click the button I get a spinning circle with "Please wait" that never goes away. It appears that installing charts from either the official catalog or TrueCharts fail to ever finish either (they get to 75% and then just hang forever).

Plex from the official catalog and a Docker image I had installed in the past still work. (Plex seems to not start correctly after a reboot until I manually stop and start it, but that could be unrelated?)

I would really appreciate any assistance, updating to TrueNAS-SCALE-22.12.2 didn't change anything.

 

[morganL]

Unless others are having the same issue... it is likely your systems or configuration. Can you try to provide other information and check that networking is all working well. Any diagnostics are useful.

Can you install a new official App?

 

[Sunoo]

Oh, I wouldn't be surprised at all if it was something with my configuration, I just have no idea where to even begin looking. Happy to provide any log or try any non-destructive troubleshooting steps though.

Networking is working fine, and after some multiple attempts and a few reboots, both Official and TrueCharts items seem to install reliably now. The "Launch Docker Image" option still just gives me the spinner and "Please wait" though.

 

[murzik]

I am having the same issue on two diferent systems.
One 22.12.2
Second 22.12.1
Not sure when it started.

 

[Kiklas]

I am also having the same issue on two diferent systems.

 

[kevinmatt]

I'm having the same issue on 22.12.2, btw on 22.12.0 has no issue on this.

 

[thedevnull]

I've run into this issue too... initially I thought it was related to jumping from 22.12.0 to 22.12.2 but doing a fresh install back to 22.12.0 and restoring from the backup taken before upgrading still presented the same issue. Unfortunately I didn't bother to check if I could launch prior to the restore from backup ...

so to debug I've run an inspect in firefox for the websocket, I was able to see the following call when "launch docker" is clicked (hopefully not a redherring)..
submission (client -> server):

Code:

{"id":"5a452ae7-2ac8-3e43-fde1-9d9b37749eba","msg":"method","method":"catalog.get_item_details","params":["ix-chart",{"cache":true,"catalog":"OFFICIAL","train":"charts"}]}

response (server -> client) stating "reason": "[ENOENT] Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'":

Code:

{"msg": "result", "id": "5a452ae7-2ac8-3e43-fde1-9d9b37749eba", "error": {"error": 2, "type": null, "reason": "[ENOENT] Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'", "trace": {"class": "CallError", "frames": [{"filename": "/usr/lib/python3/dist-packages/middlewared/main.py", "lineno": 208, "method": "call_method", "line": "            self.send_error(message, e.errno, str(e), sys.exc_info(), extra=e.extra)\n", "argspec": ["self", "message", "serviceobj", "methodobj"], "locals": {"self": "<middlewared.main.Application object at 0x7f8a44691a00>", "message": "{'id': '5a452ae7-2ac8-3e43-fde1-9d9b37749eba', 'msg': 'method', 'method': 'catalog.get_item_details', 'params': ['ix-chart', {'cache': True, 'catalog': 'OFFICIAL', 'train': 'charts'}]}", "serviceobj": "<CompoundService: <middlewared.plugins.catalogs_linux.items.CatalogService object at 0x7f8a7fffaf40>, <middlewared.plugins.catalogs_linux.features.CatalogService object at 0x7f8a800002b0>, <middlewared.plugins.catalogs_linux.sync_catalogs.CatalogService object at 0x7f8a800005b0>, <middlewared.plugins.catalogs_linux.update.CatalogService object at 0x7f8a80000550>, <middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>, <middlewared.plugins.catalogs_linux.validate.CatalogService object at 0x7f8a80000430>>", "methodobj": "<bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>>", "params": "['ix-chart', {'cache': True, 'catalog': 'OFFICIAL', 'train': 'charts'}]", "e": "CallError(\"Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'\")"}}, {"filename": "/usr/lib/python3/dist-packages/middlewared/main.py", "lineno": 1266, "method": "_call", "line": "        return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)\n", "argspec": ["self", "name", "serviceobj", "methodobj", "params"], "keywordspec": "kwargs", "locals": {"self": "<middlewared.main.Middleware object at 0x7f8a88ae90d0>", "name": "'catalog.get_item_details'", "serviceobj": "<CompoundService: <middlewared.plugins.catalogs_linux.items.CatalogService object at 0x7f8a7fffaf40>, <middlewared.plugins.catalogs_linux.features.CatalogService object at 0x7f8a800002b0>, <middlewared.plugins.catalogs_linux.sync_catalogs.CatalogService object at 0x7f8a800005b0>, <middlewared.plugins.catalogs_linux.update.CatalogService object at 0x7f8a80000550>, <middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>, <middlewared.plugins.catalogs_linux.validate.CatalogService object at 0x7f8a80000430>>", "methodobj": "<bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>>", "params": "['ix-chart', {'cache': True, 'catalog': 'OFFICIAL', 'train': 'charts'}]", "kwargs": "{'app': <middlewared.main.Application object at 0x7f8a44691a00>}", "prepared_call": "<middlewared.main.PreparedCall object at 0x7f8a65fe2190>"}}, {"filename": "/usr/lib/python3/dist-packages/middlewared/main.py", "lineno": 1169, "method": "run_in_executor", "line": "        return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))\n", "argspec": ["self", "pool", "method"], "varargspec": "args", "keywordspec": "kwargs", "locals": {"self": "<middlewared.main.Middleware object at 0x7f8a88ae90d0>", "pool": "<middlewared.utils.threading.IoThreadPoolExecutor object at 0x7f8a8cbd18e0>", "method": "<bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>>", "args": "('***', '***')", "kwargs": "{}", "loop": "<_UnixSelectorEventLoop running=True closed=False debug=False>"}}, {"filename": "/usr/lib/python3.9/concurrent/futures/thread.py", "lineno": 58, "method": "run", "line": "            self.future.set_result(result)\n", "argspec": ["self"], "locals": {"self": "None"}}, {"filename": "/usr/lib/python3/dist-packages/middlewared/schema.py", "lineno": 1288, "method": "nf", "line": "                return func(*args, **kwargs)\n", "varargspec": "args", "keywordspec": "kwargs", "locals": {"args": "('***', '***', '***')", "kwargs": "{}", "clean_and_validate_args": "<function accepts.<locals>.wrap.<locals>.clean_and_validate_args at 0x7f8a81082310>", "func": "<function returns.<locals>.returns_internal.<locals>.nf at 0x7f8a81082430>"}}, {"filename": "/usr/lib/python3/dist-packages/middlewared/schema.py", "lineno": 1158, "method": "nf", "line": "                res = f(*args, **kwargs)\n", "varargspec": "args", "keywordspec": "kwargs", "locals": {"args": "('***', '***', '***')", "kwargs": "{}", "f": "<function CatalogService.get_item_details at 0x7f8a810823a0>", "nf": "<function returns.<locals>.returns_internal.<locals>.nf at 0x7f8a81082430>"}}, {"filename": "/usr/lib/python3/dist-packages/middlewared/plugins/catalogs_linux/item_version.py", "lineno": 46, "method": "get_item_details", "line": "            raise CallError(f'Unable to locate {item_name!r} at {item_location!r}', errno=errno.ENOENT)\n", "argspec": ["self", "item_name", "options"], "locals": {"self": "<middlewared.plugins.catalogs_linux.item_version.CatalogService object at 0x7f8a80000490>", "item_name": "'ix-chart'", "options": "{'cache': True, 'catalog': 'OFFICIAL', 'train': 'charts'}", "catalog": "{'label': 'OFFICIAL', 'repository': 'https://github.com/truenas/charts.git', 'branch': 'master', 'builtin': True, 'preferred_trains': ['charts'], 'location': '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master', 'id': 'OFFICIAL'}", "item_location": "'/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'"}}], "formatted": "Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 181, in call_method\n    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)\n  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1266, in _call\n    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)\n  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1169, in run_in_executor\n    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))\n  File \"/usr/lib/python3.9/concurrent/futures/thread.py\", line 52, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1288, in nf\n    return func(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1158, in nf\n    res = f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/middlewared/plugins/catalogs_linux/item_version.py\", line 46, in get_item_details\n    raise CallError(f'Unable to locate {item_name!r} at {item_location!r}', errno=errno.ENOENT)\nmiddlewared.service_exception.CallError: [ENOENT] Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'\n", "repr": "CallError(\"Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'\")"}, "extra": null}}

a check of that path:

Code:

$ ls -lah /mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/
total 59K
drwxr-xr-x 16 root root 16 Apr 11 20:44 .
drwxr-xr-x  9 root root 23 Apr 11 20:44 ..
drwxr-xr-x  3 root root  4 Mar 24 22:47 collabora
drwxr-xr-x  3 root root  4 Mar 24 22:47 diskoverdata
drwxr-xr-x  3 root root  4 Apr 11 20:44 elastic-search
drwxr-xr-x  3 root root  4 Mar 24 22:47 emby
drwxr-xr-x  3 root root  4 Apr 10 22:52 home-assistant
drwxr-xr-x  3 root root  6 Mar 15 00:58 minio
drwxr-xr-x  3 root root  4 Mar 21 22:45 netdata
drwxr-xr-x  3 root root  4 Mar 30 22:48 nextcloud
drwxr-xr-x  3 root root  4 Mar 24 22:47 photoprism
drwxr-xr-x  3 root root  4 Apr 10 22:52 pihole
drwxr-xr-x  3 root root  4 Apr 11 20:44 plex
drwxr-xr-x  3 root root  4 Mar 24 22:47 storj
drwxr-xr-x  3 root root  4 Apr  6 22:50 syncthing
drwxr-xr-x  3 root root  4 Mar 24 22:47 wg-easy

and from the git log I see the following noting a move of ix-chart to ix-dev that looked to line up with the error from the inspect:

Code:

commit 768be33a968ba8b18e3a9c306260fe2190dcb247
Author: Stavros Kois <47820033+stavros-k@users.noreply.github.com>
Date:   Tue Apr 11 18:36:44 2023 +0300

    move ix-chart to ix-dev (#1054)

    * switch to the newly released official repo

    * remove --helm3 flag as it does not exist anymore

    * move ix-chart

    * create values.yaml and remove old test hook

    * update version scheme

I'm off to bed, hopefully the above is relevant and not a rabbit hole?

 

[thedevnull]

just to follow up to my last post, checking out an earlier commit from the git commit history seemed to remedy the issue (picking the last commit from April 10th)...

example: git checkout cf8b4315e85a65367d5d087dab9eb26c3a27091c
ix-charts is now present:

Code:

$ ls -lah charts/
total 59K
drwxr-xr-x 16 root root 16 Apr 12 03:06 .
drwxr-xr-x  9 root root 23 Apr 12 03:06 ..
drwxr-xr-x  3 root root  4 Mar 24 22:47 collabora
drwxr-xr-x  3 root root  4 Mar 24 22:47 diskoverdata
drwxr-xr-x  3 root root  4 Mar 24 22:47 emby
drwxr-xr-x  3 root root  4 Apr 10 22:52 home-assistant
drwxr-xr-x  3 root root  4 Apr 12 03:06 ix-chart
drwxr-xr-x  3 root root  6 Mar 15 00:58 minio
drwxr-xr-x  3 root root  4 Mar 21 22:45 netdata
drwxr-xr-x  3 root root  4 Mar 30 22:48 nextcloud
drwxr-xr-x  3 root root  4 Mar 24 22:47 photoprism
drwxr-xr-x  3 root root  4 Apr 10 22:52 pihole
drwxr-xr-x  3 root root  4 Apr 12 03:06 plex
drwxr-xr-x  3 root root  4 Mar 24 22:47 storj
drwxr-xr-x  3 root root  4 Apr  6 22:50 syncthing
drwxr-xr-x  3 root root  4 Mar 24 22:47 wg-easy

And the launch docker button works as expected now.... bad merge upstream with the truenas charts repository and dependency for the launch docker behaviour?

 

[Trevor68]

Same issue here only since updating to 22.12.2

Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body rv = await self.method(*([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1186, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 486, in do_create await self.middleware.call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1346, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1249, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 44, in helm_action raise CallError(f'Failed to {tn_action} chart release: {stderr.decode()}') middlewared.service_exception.CallError: [EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: namespaces "cert-manager" not found

 

[sammael]

I can confirm the same issue, the blue circle just spins indefinitely - I waited over 10 minutes. I remember the same bug few versions back seems it's back, which is bad because docker is like 100% of reason why I switched to scale. Another user on reddit thread said he captured following error via websocket while trying to launch docker image, but I've no idea if it's related:

"[ENOENT] Unable to locate 'ix-chart' at '/mnt/volume1/ix-applications/catalogs/github_com_truenas_charts_git_master/charts/ix-chart'"

Launching docker images worked perfectly in previous version.

EDIT: launching apps work and it worked when launching docker images was broken in previous versions too, it's not the issue with apps, it's issue with launching custom docker image AS an app.

 

[morganL]

If someone withe the problem wants to help locate the issue.. please report a bug and provide a link the NAS ticket.

 

[kevinmatt]

If someone withe the problem wants to help locate the issue.. please report a bug and provide a link the NAS ticket.

I've already created a ticket but it's still submitting...

https://jira-redirect.ixsystems.com/issues/jira/1057419994743669

 

[ASiC]

Same problem here.
Ticket logged:

[NAS-121448] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[sammael]

You have to turn off Host Path Checking in Apps Settings / Advanced Settings. It gave me scary sounding warning worded in such a way I have no idea what it said so I just said yes and now I can deploy docker images and all other apps work perfectly as well.

 

[murzik]

You have to turn off Host Path Checking in Apps Settings / Advanced Settings. It gave me scary sounding warning worded in such a way I have no idea what it said so I just said yes and now I can deploy docker images and all other apps work perfectly as well.

Not the case, I have path checking off.

 

[Sunoo]

There was an update posted on the JIRA ticket that got it working for me. Apparently they fixed something, so if you refresh the catalogs and try again it should work now.

[NAS-121448] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[sammael]

Not the case, I have path checking off.

Hmm I've got nothing then sorry, only other thing someone on reddit mentioned is to turn off smb sharing which I tried and didn't work and the 2nd suggestion was the host path checking which did it for me. Looks like there might be more root causes of the problem.

 

[Ryan892]

You have to turn host path checking on and off again. I had host path checking off before I updated to v22.12.2 and also ran into this issue. That is what fixed it for me.

When you turn it on, the loading circle will spin forever, so reload the webpage and all your apps may disappear. Don't worry. Just turn off host path checking again (reload again if the loading circle doesn't disappear again) and restart your machine. After this, the GUI option for launching docker containers should be fixed.

 

[thedevnull]

There was an update posted on the JIRA ticket that got it working for me. Apparently they fixed something, so if you refresh the catalogs and try again it should work now.

[NAS-121448] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Thanks for cutting the ticket for this. I can confirm the missing ix-charts folder was back on the git pull/refresh and the functionality was restored.

I see a lot of people referring to host path checking which sounds similar exhibited behaviour but not what broke in this scenario per my findings I posted (I already had this disabled from the initial bluefin release as well).

Much appreciated!

 

[morganL]

You have to turn host path checking on and off again. I had host path checking off before I updated to v22.12.2 and also ran into this issue. That is what fixed it for me.

When you turn it on, the loading circle will spin forever, so reload the webpage and all your apps may disappear. Don't worry. Just turn off host path checking again (reload again if the loading circle doesn't disappear again) and restart your machine. After this, the GUI option for launching docker containers should be fixed.

So, there ere seem to be two separate issues:
1. missing ix-charts folder on github. This has been resolved
2. Potential need to toggle or ensure host path validation is disabled

If anyone is still having issues after trying #2, please clarify.