SOLVED All SCALE Apps suddenly disappeared!

[Sawtaytoes]

I installed a new NIC, had some IP issues, and my containers weren't coming up, so I restarted the server.

Version: TrueNAS-SCALE-22.12.3.2

This is what I see now (no apps installed?):

The TrueNAS-Apps zpool is still there with no issues:

It's even set correctly as the Application Pool:

What do I do to fix this? I really want these apps back fast! I do take snapshots of the datasets, so hopefully I have backups or even better, hopefully they're still on-disk.

 

[samarium]

Double check that your BIOS/CMOS/RTC is in UTC.

Code:

$ timedatectl

Of not there are various forum posts to reset the clock. Lots of weird things happen on scale when the clock isn't set correctly.

This might be totally irrelevant, but is a check check.

 

[Sawtaytoes]

Looks like it's the correct time:

Code:

# timedatectl
               Local time: Wed 2023-07-19 01:40:39 CDT
           Universal time: Wed 2023-07-19 06:40:39 UTC
                 RTC time: Wed 2023-07-19 06:40:39
                Time zone: America/Chicago (CDT, -0500)
System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no

 

[samarium]

Agree. You could always hwclock --systohc to push to the RTC. As I said TNS time setup is weird.

 

[Sawtaytoes]

After restarting, my apps came back!

I got some error about the reporting not being set correctly and needing to be cleared. Not sure what that means, but I did clear the logs before clicking "Applications". Not sure if they were back before clicking "Clear".

I'm seeing these errors now in the bell:

Code:

Failed to sync TRUENAS catalog: [EFAULT] Failed to clone 'https://github.com/truenas/charts.git' repository at '/mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truenas_charts_git_master' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truenas/charts.git /mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truenas_charts_git_master stderr: 'Cloning into '/mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truenas_charts_git_master'... fatal: unable to access 'https://github.com/truenas/charts.git/': server certificate verification failed. CAfile: none CRLfile: none '
2023-02-08 02:23:24 PM (America/Chicago)

Code:

Failed to sync TRUECHARTS catalog: [EFAULT] Failed to clone 'https://github.com/truecharts/catalog' repository at '/mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truecharts_catalog_main' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truecharts/catalog /mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truecharts_catalog_main stderr: 'Cloning into '/mnt/TrueNAS-Apps/ix-applications/catalogs/github_com_truecharts_catalog_main'... fatal: unable to access 'https://github.com/truecharts/catalog/': server certificate verification failed. CAfile: none CRLfile: none '
2023-02-08 02:23:27 PM (America/Chicago)

This sync task not working makes it looks like something else is wrong:

Code:

Cloud sync task "Shared Video Projects | G-Drive" failed.
2023-02-08 02:24:02 PM (America/Chicago)

My files are here and accessible over SMB.

 

[Sawtaytoes]

Oh, I see:

x509: certificate has expired or is not yet valid: current time 2023-02-08T14:29:55-06:00 is before 2023-06-19T08:20:47Z

After I'd lost those containers, I shutdown the system and swapped the CPU. The Supermicro board reset my UEFI for some reason. Maybe that reset the time too. I dunno how to fix it without yet another restart though.

 

[samarium]

certificate errors are classic date time issues even when it looks right, chase down the post of mine on timedatectl systemctl ntpd hwclock

 

[Sawtaytoes]

This is what you wrote:

Code:

systemctl stop ntp
ntpd -g -q
systemctl start ntp
hwclock --systohc
date

Code:

~# date
Wed Jul 19 03:27:51 CDT 2023

It did fix my time.

Does `hwclock` configure the UEFI time?

 

[Sawtaytoes]

BAM!

Thank you so much!

 

[samarium]

hwclock --systohc copies the system time, which is internally in UTC, to the hardware clock, aka BIOS/CMOS/RTC clock, and which you are calling UEFI clock I guess.

 

[f110tech]

My problem had nothing to do with the Time. I verified it was set correctly and it was.
My Fix: Apps->Settings->Choose Pool, and redefine your pool the apps are on.

Apps reappear and are already running

 

[MichaelInOr]

Same problem when upgrading to TrueNAS-SCALE-22.12.4.2, all my apps disappeared.

I rebooted and waited about 15 minutes. All my apps came back on their own.

 

[impestrator]

Since today I´ve got the same issue. I´ve rebooted several times the apps are still not available. Does anyone have another idea to seach for?

Thanks in advanced

 

[fogNL]

Since today I´ve got the same issue. I´ve rebooted several times the apps are still not available. Does anyone have another idea to seach for?

Thanks in advanced

This has happened to me a few times, and it seems the culprit is the default route disappearing in TrueNas. I'll fix it, and it will work for a bit, then go missing again. No idea why this is happening.

If you go to the Network tab, you won't see a default route listed, nor a way to add one. The place to add one is in Apps->Settings->Advanced Settings and set the Route v4 Interface and Route v4 Gateway. After this, I reboot, and now under "network" you can see a default route listed and going to Apps and Refresh Charts should work now.

Checking the console confirms this.

Before change:

root@truenas[~]# ip route show
172.16.0.0/16 dev kube-bridge proto kernel scope link src 172.16.0.1
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.200

After change:

root@truenas[~]# ip route show
default via 192.168.1.1 dev enp9s0
172.16.0.0/16 dev kube-bridge proto kernel scope link src 172.16.0.1
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.200