Kubernetes (apps) take a long time to start after reboot

[lutzky]

My system takes about 5 minutes to boot, but about 20 minutes more until all of my apps are done booting. The apps are:

* Official TrueNAS charts - plex
* Truecharts - radarr, prowlarr, qbittorrent, tdarr, sonarr, esphome
* Docker containers - cloudflared, nginx, home-assistant

I'm migrating from an Ubuntu system using ZFS which ran the same collection of software using docker-compose, and it would start well within a minute. I'm guessing k3s is unhappy about something. I've attached /var/log/k3s_daemon.log, and /var/log/syslog, though a cursory scan shows nothing obvious.

Reboot occurred at Jul 24 at 08:48, which is where the logs start. However, home-assistant, for example, doesn't start until 09:01, and doesn't finish loading integrations before 09:10. (when restarting the homeassistant container directly, after the system has stabilized it finishes loading within a minute).

Is this expected performance? Any tips on where to poke to understand what's going on?

Show : System specs

Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz, 6 cores
RAM: 16GB
Storage: 3x2TB HDDs in RAIDZ1
Boot drive: 256GB NVME

 

[lutzky]

Apologies, forgot to attach logs.

 

[morganL]

My system takes about 5 minutes to boot, but about 20 minutes more until all of my apps are done booting. The apps are:

* Official TrueNAS charts - plex
* Truecharts - radarr, prowlarr, qbittorrent, tdarr, sonarr, esphome
* Docker containers - cloudflared, nginx, home-assistant

I'm migrating from an Ubuntu system using ZFS which ran the same collection of software using docker-compose, and it would start well within a minute. I'm guessing k3s is unhappy about something. I've attached /var/log/k3s_daemon.log, and /var/log/syslog, though a cursory scan shows nothing obvious.

Reboot occurred at Jul 24 at 08:48, which is where the logs start. However, home-assistant, for example, doesn't start until 09:01, and doesn't finish loading integrations before 09:10. (when restarting the homeassistant container directly, after the system has stabilized it finishes loading within a minute).

Is this expected performance? Any tips on where to poke to understand what's going on?

Show : System specs

Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz, 6 cores
RAM: 16GB
Storage: 3x2TB HDDs in RAIDZ1
Boot drive: 256GB NVME

Its good to have the data. We are testing boot times on larger systems in preparation for 22.02.3
I'd be interested to hear if others have similar times.
Is it a home-assistant issue or just generic to all apps?
Are the docker containers worse that the TrueCharts apps?

 

[lutzky]

Good to know this is being worked on.
This isn't specific to home-assistant. I've singled it out as an important use-case, because:

1. It's lightweight in and of itself (my raspberry pi starts up much faster with it)
2. It's time-sensitive (half an hour without plex might be fine; half an hour without home-assistant can be actively disruptive). For this reason, I'll be migrating it off TrueNAS to a dedicated raspberry pi.

Between "custom" docker containers, TrueCharts apps, and TrueNAS-built-in apps, I've not noticed a meaningful difference in timing.

 

[lutzky]

Good to know this is being worked on.
This isn't specific to home-assistant. I've singled it out as an important use-case, because:

1. It's lightweight in and of itself (my raspberry pi starts up much faster with it)
2. It's time-sensitive (half an hour without plex might be fine; half an hour without home-assistant can be actively disruptive). For this reason, I'll be migrating it off TrueNAS to a dedicated raspberry pi.

Between "custom" docker containers, TrueCharts apps, and TrueNAS-built-in apps, I've not noticed a meaningful difference in timing.

Oh, home-assistant has another advantage-as-a-test-case of having a "partially loaded" state - i.e. the UI will respond, but tell you "I'm still loading this and that", whereas other apps usually don't have such a clear split.