Kubernetes failing to start - CRC Failure

[tonyp1983]

Posting as cannot seem to find anything similar on the forum or elsewhere via Google.

I’ve been running Scale since the RC late last year and been successfully running a number of apps without issue. Yesterday (running 22.02.2), I noticed the UI was a bit sluggish and as the system had been running without issue for a couple of weeks I thought I’d give it a quick reboot in the grand tradition of turning it off an on again. Waited the usual 6-7 minutes for the system to boot and the UI to become available, logged in and all seemed fine, chart refresh was underway so left it too it.

Went to have a look on Plex about an hour later and realised it wasn’t running so went to check, couldn’t load anything in Installed Apps (spinning wheel of death foorever) and then noticed an alert stating Kubernetes failed to start and referencing a CRC check failure. Tried a few reboots assuming it was a temporary glitch but still no dice today after updating to 22.02.3 (from 22.02.2) and numerous reboots and even a couple of cold starts - still the same issue after the chart refresh has completed seemingly successfully.

UI is showing no pool/disk issues, the only other thing I had on first reboot was an alert about NTP check failing (system time was correct however) which resolved itself on a later reboot and has not reoccured since.

Trying to unset the Apps pool fails with a CRC error message and the below traceback which looks like a potentially corrupt gzip file (not sure what gzip file and don’t know how to find out).

Traceback error contents:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
    res = f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/certificates.py", line 688, in do_delete
    check_dependencies(self.middleware, 'CERT', id)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/dependencies.py", line 14, in check_dependencies
    chart_releases = middleware.call_sync(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1403, in run_coroutine
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 175, in query
    release_secrets = await self.middleware.call('chart.release.releases_secrets', extra)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1304, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/secrets_management.py", line 38, in releases_secrets
    release = json.loads(gzip.decompress(b64decode(b64decode(data['release']))).decode())
  File "/usr/lib/python3.9/gzip.py", line 556, in decompress
    return f.read()
  File "/usr/lib/python3.9/gzip.py", line 300, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.9/gzip.py", line 478, in read
    self._read_eof()
  File "/usr/lib/python3.9/gzip.py", line 524, in _read_eof
    raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32),
gzip.BadGzipFile: CRC check failed 0x2e58b797 != 0x77b3f603

I’d not touched the system in days, nothing had been changed and everything but the apps seems to be running fine (SMB shares, etc.)

I have filed a Jira ticket through the UI and provided a debug, and it’s awaiting review by an engineer but thought I’d through it out here in case anyone else had experienced something similar and/or could point me in the right direction of trying to sort this out - It’s annoying me that I’m totally stumped plus the family are missing Plex

 

[sensor]

I have a similair problem.
Did you ever find a solution? in the meanwhile I am waiting for a reply on my jira ticket as well :)

 

[d1ddle]

I also have the exact same problem after manually updating from 22.12.3.1 to 3.3. Apps fail to start, No visible log errors anywhere, just:
CRITICAL
Failed to start kubernetes cluster for Applications: CRC check failed 0x22c34c89 != 0x5ad6ef49
2023-09-20 23:44:08 (Europe/London)