SCALE upgrade to 21.04 from 21.02 broke docker and apps

[cyrus104]

I had a docker running Plex on SCALE 21.02 which was working as expected. Very nice interface and easy to use!

I just updated to 21.04 and made sure that the correct pool is selected but no installed apps are being displayed. In the shell I verified that the files are there and mounted. I went in an unset the pool and then set the pool option to the pool that has had the ix-applications dataset.

Let me know what other logs would be helpful.

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 378, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 414, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1001, in nf
    return await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 239, in do_update
    await self.middleware.call('kubernetes.status_change')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1239, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1207, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1111, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3/dist-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 183, in status_change
    self.middleware.call_sync('kubernetes.status_change_internal')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1295, in run_coroutine
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 198, in status_change_internal
    await self.middleware.call('container.image.load_default_images')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1239, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1196, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker_linux/images.py", line 143, in load_default_images
    await self.load_images_from_file(DEFAULT_DOCKER_IMAGES_PATH)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker_linux/images.py", line 127, in load_images_from_file
    await self.docker_checks()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker_linux/images.py", line 148, in docker_checks
    raise CallError('Docker service is not running')
middlewared.service_exception.CallError: [EFAULT] Docker service is not running

 

[Kris Moore]

That does indeed sound like a bug. Please open a ticket on jira.ixsystems.com and we'll get that looked at!

 

[waqarahmed]

@cyrus104 can you please create a ticket at https://jira.ixsytems.com and attach a debug as well ? Or you can email me a debug at waqar at the rate of ixsystems.com as well. Looking forward to hearing from you. Thank you

 

[cyrus104]

Not sure there are still logs besides what I sent. The family wanted Plex so I deleted the ix-application dataset and recreated the Plex docker. I have 32-threads so Plex imported 50tb pretty fast.

To go along with the Plex docker issue, it looks like there is no way to change tags for the plexpass docker version.

 

[waqarahmed]

@cyrus104 can you please refresh OFFICIAL catalog and upgrade to latest plex version ? That should have the latest plex tag. About it being configurable by the user, we do not want that so as to lock an application to a particular version. I can understand that plex is updated frequently, so in that case we will be doing that ourselves to ensure that users have the option to go with the latest version.

Btw can you please still email me a debug of your system ? Applications should not start on their own and that did not happen and we should diagnose/fix accordingly why. Please let me know if there are concerns. Thank you

 

[cyrus104]

NAS-110353 created

I might be able to upload the debug log:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 378, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 416, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3/dist-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1005, in nf
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/system.py", line 992, in debug
    shutil.copyfileobj(f, job.pipes.output.w)
  File "/usr/lib/python3.9/shutil.py", line 208, in copyfileobj
    fdst_write(buf)
BrokenPipeError: [Errno 32] Broken pipe