Failed to start kubernetes cluster for Applications: year 0 is out of range

[pamiro]

I just restarted my system and it presented me this critical alert:

Code:

Failed to start kubernetes cluster for Applications: year 0 is out of range

As to be expected, none of my Apps are running.

This came along with some other issues, wich were there even present for at least two days berfore the reboot (and were in fact the reason why I rebooted in the first place):

• Apps->Installed Applications does not load, just shows the loading animation forever
• When I open Apps->Manage Docker Containers, it opens this popup twice:

under "More info...", this is the error message:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 176, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1293, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker_linux/images.py", line 74, in query
    'created': datetime.fromtimestamp(int(image['Created'])),
ValueError: year 0 is out of range

The only other reference to this exact issue I was able to find is this deleted reddit post from a week ago.
I have no idea how I could fix this problem, as I have no knowledge about the inner workings of Kubernetes.
This only just happened so the only thing I've tried is a restarting the system again, but to no avail.

My Setup:

• OS Version: TrueNAS-SCALE-22.02.3
• CPU: Intel(R) Core(TM) i5-2320 CPU @ 3.00GHz
• RAM: 16GB DDR3

 

[jgreco]

Check that your time and date is set in the BIOS, and that you have some NTP servers set up and synchronizing.

 

[pamiro]

Check that your time and date is set in the BIOS, and that you have some NTP servers set up and synchronizing.

I can't acces the BIOS right now (I have no monitor I could plug the server into), but I'm fairly sure that I have already set up time and date there (I had a different issue that I fixed by doing that).
I have 3 NTP servers set up (I believe these were the default ones):

• 0.debian.pool.ntp.org
• 1.debian.pool.ntp.org
• 2.debian.pool.ntp.org

timedatectl gives me

Code:

               Local time: Mon 2023-02-27 23:24:41 CET
           Universal time: Mon 2023-02-27 22:24:41 UTC
                 RTC time: Mon 2023-02-27 22:24:41
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no

The displayed times are exact to the second to what my PC shows.

The system was running smoothly for several months until now, the only thing that I've done that could have possibly caused the issue is updating a few of my apps.

 

[zari90]

I can't acces the BIOS right now (I have no monitor I could plug the server into), but I'm fairly sure that I have already set up time and date there (I had a different issue that I fixed by doing that).
I have 3 NTP servers set up (I believe these were the default ones):

• 0.debian.pool.ntp.org
• 1.debian.pool.ntp.org
• 2.debian.pool.ntp.org

timedatectl gives me

Code:

               Local time: Mon 2023-02-27 23:24:41 CET
           Universal time: Mon 2023-02-27 22:24:41 UTC
                 RTC time: Mon 2023-02-27 22:24:41
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no

The displayed times are exact to the second to what my PC shows.

The system was running smoothly for several months until now, the only thing that I've done that could have possibly caused the issue is updating a few of my apps.

I had the same issue after rebooting, what I noticed was that I upgraded from Angelfish to Bluefin, that seemed to solve the problem of Kubernetes failed to started, but now the apps are stuck in deployed state and if I try and upgrade them anything that is linked to my mass Media Storage doesn't show active but the ones that aren't linked to the /media storage are fine.

I'm doing a pool scrub to see if that works on both datasets, hoping that works will update if its fixed, but try updating to Bluefin in the mean time and see if that can resolve your issue you having

you can always roll back the update under system settings, please let me know if this works

 

[pamiro]

Found the issue!

The TrueNAS Apps system seems to crash when there are container images present that do not have a creation date.
In my case, an update for the "Immich" App (version 1.49.0_5.0.15 from TrueCharts) added these two container images (note that for all other images in the list, it shows the time since the creation date where these show "N/A")

I fixed this by running

Code:

docker image prune

to delete these images, rebooting the system and then quickly turning off Immich before the system could pull the images again.
This od course leaves me running wothout Immich until this is fixed but at least everything else works again.

I have also filed this bug report about the issue.