tonyp1983
Dabbler
- Joined
- Feb 17, 2015
- Messages
- 31
Posting as cannot seem to find anything similar on the forum or elsewhere via Google.
I’ve been running Scale since the RC late last year and been successfully running a number of apps without issue. Yesterday (running 22.02.2), I noticed the UI was a bit sluggish and as the system had been running without issue for a couple of weeks I thought I’d give it a quick reboot in the grand tradition of turning it off an on again. Waited the usual 6-7 minutes for the system to boot and the UI to become available, logged in and all seemed fine, chart refresh was underway so left it too it.
Went to have a look on Plex about an hour later and realised it wasn’t running so went to check, couldn’t load anything in Installed Apps (spinning wheel of death foorever) and then noticed an alert stating Kubernetes failed to start and referencing a CRC check failure. Tried a few reboots assuming it was a temporary glitch but still no dice today after updating to 22.02.3 (from 22.02.2) and numerous reboots and even a couple of cold starts - still the same issue after the chart refresh has completed seemingly successfully.
UI is showing no pool/disk issues, the only other thing I had on first reboot was an alert about NTP check failing (system time was correct however) which resolved itself on a later reboot and has not reoccured since.
Trying to unset the Apps pool fails with a CRC error message and the below traceback which looks like a potentially corrupt gzip file (not sure what gzip file and don’t know how to find out).
Traceback error contents:
I’d not touched the system in days, nothing had been changed and everything but the apps seems to be running fine (SMB shares, etc.)
I have filed a Jira ticket through the UI and provided a debug, and it’s awaiting review by an engineer but thought I’d through it out here in case anyone else had experienced something similar and/or could point me in the right direction of trying to sort this out - It’s annoying me that I’m totally stumped plus the family are missing Plex
I’ve been running Scale since the RC late last year and been successfully running a number of apps without issue. Yesterday (running 22.02.2), I noticed the UI was a bit sluggish and as the system had been running without issue for a couple of weeks I thought I’d give it a quick reboot in the grand tradition of turning it off an on again. Waited the usual 6-7 minutes for the system to boot and the UI to become available, logged in and all seemed fine, chart refresh was underway so left it too it.
Went to have a look on Plex about an hour later and realised it wasn’t running so went to check, couldn’t load anything in Installed Apps (spinning wheel of death foorever) and then noticed an alert stating Kubernetes failed to start and referencing a CRC check failure. Tried a few reboots assuming it was a temporary glitch but still no dice today after updating to 22.02.3 (from 22.02.2) and numerous reboots and even a couple of cold starts - still the same issue after the chart refresh has completed seemingly successfully.
UI is showing no pool/disk issues, the only other thing I had on first reboot was an alert about NTP check failing (system time was correct however) which resolved itself on a later reboot and has not reoccured since.
Trying to unset the Apps pool fails with a CRC error message and the below traceback which looks like a potentially corrupt gzip file (not sure what gzip file and don’t know how to find out).
Traceback error contents:
Code:
Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body rv = await self.middleware.run_in_thread(self.method, *([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf res = f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/certificates.py", line 688, in do_delete check_dependencies(self.middleware, 'CERT', id) File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/dependencies.py", line 14, in check_dependencies chart_releases = middleware.call_sync( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in call_sync return self.run_coroutine(methodobj(*prepared_call.args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1403, in run_coroutine return fut.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 175, in query release_secrets = await self.middleware.call('chart.release.releases_secrets', extra) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1304, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/secrets_management.py", line 38, in releases_secrets release = json.loads(gzip.decompress(b64decode(b64decode(data['release']))).decode()) File "/usr/lib/python3.9/gzip.py", line 556, in decompress return f.read() File "/usr/lib/python3.9/gzip.py", line 300, in read return self._buffer.read(size) File "/usr/lib/python3.9/gzip.py", line 478, in read self._read_eof() File "/usr/lib/python3.9/gzip.py", line 524, in _read_eof raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32), gzip.BadGzipFile: CRC check failed 0x2e58b797 != 0x77b3f603
I’d not touched the system in days, nothing had been changed and everything but the apps seems to be running fine (SMB shares, etc.)
I have filed a Jira ticket through the UI and provided a debug, and it’s awaiting review by an engineer but thought I’d through it out here in case anyone else had experienced something similar and/or could point me in the right direction of trying to sort this out - It’s annoying me that I’m totally stumped plus the family are missing Plex