Immich database system is shut down

[cmax]

Hey Community,
I am pretty new to TrueNAS and Kubernetes, so please be patient with me...

Since last update of Immich my instance keeps deploying forever. The history tells me

Back-off restarting failed container immich in pod immich-postgres-78ffd9fccd-nft7m_ix-immich(ca280ec7-ba34-49c8-8d37-b1b283bf1c1e)

The console is telling me over and over again

IPVS: rr: TCP 172.17.197.102:30041 - no destination available
IPVS: rr: TCP 172.17.197.50:5432 - no destination available

Log of immich-postgres

Code:

2024-02-28T20:46:20.470870360+01:00
2024-02-28T20:46:20.470904022+01:00 PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-02-28T20:46:20.470913099+01:00
2024-02-28T20:46:20.489052293+01:00 2024-02-28 20:46:20.488 CET [1] LOG:  starting PostgreSQL 15.5 (Debian 15.5-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-02-28T20:46:20.489224091+01:00 2024-02-28 20:46:20.489 CET [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-02-28T20:46:20.489246667+01:00 2024-02-28 20:46:20.489 CET [1] LOG:  listening on IPv6 address "::", port 5432
2024-02-28T20:46:20.489474207+01:00 2024-02-28 20:46:20.489 CET [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-02-28T20:46:20.491443220+01:00 2024-02-28 20:46:20.491 CET [16] LOG:  database system was shut down at 2024-02-24 00:06:10 CET
2024-02-28T20:46:20.491592910+01:00 2024-02-28 20:46:20.491 CET [16] LOG:  invalid resource manager ID in primary checkpoint record
2024-02-28T20:46:20.491623116+01:00 2024-02-28 20:46:20.491 CET [16] PANIC:  could not locate a valid checkpoint record
2024-02-28T20:46:20.492136484+01:00 [2024-02-28T19:46:20Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/65138".
2024-02-28T20:46:20.492154912+01:00 [2024-02-28T19:46:20Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/85216".
2024-02-28T20:46:20.492271868+01:00 [2024-02-28T19:46:20Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/85216/segments/59f1127f-6ad7-4a11-803c-5a7daafa12da".
2024-02-28T20:46:20.492666724+01:00 2024-02-28 20:46:20.492 CET [1] LOG:  startup process (PID 16) was terminated by signal 6: Aborted
2024-02-28T20:46:20.492763315+01:00 2024-02-28 20:46:20.492 CET [1] LOG:  aborting startup due to startup process failure
2024-02-28T20:46:20.494398261+01:00 2024-02-28 20:46:20.494 CET [1] LOG:  database system is shut down

Log of immich-microservices immich-init-postgres-wait (I have shortend the list, since it was the same over and over again)

Code:

2024-02-28T20:54:27.300326308+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:29.337663397+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:31.366625919+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:33.395943088+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:35.433450392+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:37.475001449+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:39.515718001+01:00 immich-postgres:5432 - no response
2024-02-28T20:54:41.559631322+01:00 immich-postgres:5432 - no response

Since Redis seems to be working fine I won't attach the log (accept it would be helpful).
On a few Pod logs I am getting

Received 400 response code

but I guess it is just due to the fact the deployment of the containers is stuck.

From what I could understand, the database might be corrupt in some way.
What I have tried so far without success:

• restore rollback versions of the database from snapshots
• delete immich and deploy it again (about 1000 times)
• reinstall Truenas Scale with restoring configuration
• reinstall Truenas Scale from scratch

I would be so glad if anybody could help me figure out what the problem is.

My System:
TrueNAS-SCALE-23.10.2
Motherboard: Supermicro x10-SLM-F
CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
RAM: 32 ECC
Hard drives: 6x Seagate Ironwolf 4 TB in Raidz2
Bootdrives: 2x SanDisk SSD Plus 240 GB in mirror
Apps-pool: 2x Crucial SSD MX500 1000GB in mirror
Hard disk controllers: LSI 9211-4i it mode
Network cards: 10Gtek® 10Gb PCI-E NIC (Intel X540-BT1)

Thank you all in advance
Christian

 

[williamszw]

I have met exactly the same question, Have no idea either...., I'm going to install a VirtualMachine, like debian, "debian in debian loll"and run docker inside it, this is the only way I can figure out to solve these problems once for all. bless you.

 

[cmax]

Yes, I'm still not sure, if the choice to go with Truenas Apps is the wisest of all... In case a problem occures there is little chance to get things going again (at least for me with my non existing knowledge of kubernetes). How is your Debian VM on Debian thing going?

 

[chuck32]

Not the one who you asked, but I run all my docker apps in an ubuntu VM, never bothered with the scale apps.
I feel like I have more control over everything and for the past year everything's been running fine.

 

[cmax]

Good timing for that information since I just installed ubuntu server vm today. More control and the internet full of helpfull advice for problems that might pop up.

 

[williamszw]

Problem sloved! following this: https://immich.app/docs/install/truenas
PAY ATTENTION:

 

[williamszw]

since you have to mount 7 folder or datasets: library, c, uploads, and video.
and in Storage Configuration, there are two kind of storage: Hostpath and ixVolume, you can mount your own folder to "library, uploads, and video. "
but for "pgBackup, pgData, profile, thumbs," just use 【ixVolume】, then It Works!