Deleted snapshots and now all applications broken

[CBP]

Good afternoon!

New to TrueNAS and have been learning as I go. Finally got my setup completely configured how I wanted it, but I had an issue I wanted to resolve. I had 2 errors appearing while checking my zpool status, which were from corrupted files that I had removed. I cleared these errors with a scrub, but I read that I would need to delete any snapshots that referenced these files. I was under the impression that these snapshots were just backups of files/settings, which I figured I may as well just wipe them out as my data is in a good state currently and previous settings were incorrect. I did some searching online and came across a script to delete all of the snapshots, so I gave that a shot. The script I ran is here:

Code:

#!/bin/bash
for snapshot in `zfs list -H -t snapshot -r Media | cut -f 1`
do
zfs destroy $snapshot -R
done

After doing this, it seemed like everything was running smooth, except one of my applications was not functioning properly. I decided to do a system reboot to see if that would clear things up. Now, all of my applications will not start giving me PodInitialization errors. Did I just completely break all of my applications? I tried to download another application and install/launch it, but it seems even newly installed applications give the same error. I am unsure what the next step to diagnose/fix this problem would be. Any help would be greatly appreciated.

 

[CBP]

As an example of what I'm seeing from my application logs, here is what they all look like:

Code:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"smokeping\" in pod \"smokeping-7849b46f47-55mfh\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}

 

[CBP]

Diagnosed this far, but unsure how to proceed past this point:

Code:

k3s kubectl get pods --namespace ix-smokeping
NAME                         READY   STATUS                      RESTARTS   AGE
svclb-smokeping-jfnfl        1/1     Running                     0          58m
smokeping-7849b46f47-55mfh   0/1     Init:CreateContainerError   0          58m

 

[CBP]

Update: noticed that I can install and run Official apps, but any newly installed app through TrueCharts gives the above PodInitializing error.

 

[brando56894]

The filesystems for the K3S containers may be borked, preventing new containers from launching. If it's not too much of a hassle, I'd suggest deleting the ix-applications dataset and just starting over with a clean dataset. I've had this happen a few times.

IMO forcing home users to use K3S was a bad choice since it's far too complex behind the scenes, I've been testing SCALE on and off for over a year and have always had random issues with K3S, like this, where apps would just randomly break and since everything is so obfuscated by the GUI/middleware I'm left to posting here or in the Discord. Just a few days ago I finally got fed up with K3S and just setup a VM that I could run Docker in since it was far easier to manage.

 

[CBP]

The filesystems for the K3S containers may be borked, preventing new containers from launching. If it's not too much of a hassle, I'd suggest deleting the ix-applications dataset and just starting over with a clean dataset. I've had this happen a few times.

IMO forcing home users to use K3S was a bad choice since it's far too complex behind the scenes, I've been testing SCALE on and off for over a year and have always had random issues with K3S, like this, where apps would just randomly break and since everything is so obfuscated by the GUI/middleware I'm left to posting here or in the Discord. Just a few days ago I finally got fed up with K3S and just setup a VM that I could run Docker in since it was far easier to manage.

Thank you for the suggestion. I wouldn't mind wiping the dataset and re-configuring if it came to that. I'm wondering if there's a way to pull config files out of my applications though before doing this. I spent many hours configuring a couple of the applications and it would be an absolute nightmare to do some of them from scratch. I'm wondering if I may have to run docker in a VM so I can pull those configs.

 

[stavros-k]

The filesystems for the K3S containers may be borked, preventing new containers from launching. If it's not too much of a hassle, I'd suggest deleting the ix-applications dataset and just starting over with a clean dataset. I've had this happen a few times.

IMO forcing home users to use K3S was a bad choice since it's far too complex behind the scenes, I've been testing SCALE on and off for over a year and have always had random issues with K3S, like this, where apps would just randomly break and since everything is so obfuscated by the GUI/middleware I'm left to posting here or in the Discord. Just a few days ago I finally got fed up with K3S and just setup a VM that I could run Docker in since it was far easier to manage.

k3s is chosen because SCALE's target goal will be clustering.

k3s is NOT the problem here. 90% of the problem comes from the storage provisioner (openebs).
At least from my experience, and based on the tickets we get...

 

[DaSnipe]

Thank you for the suggestion. I wouldn't mind wiping the dataset and re-configuring if it came to that. I'm wondering if there's a way to pull config files out of my applications though before doing this. I spent many hours configuring a couple of the applications and it would be an absolute nightmare to do some of them from scratch. I'm wondering if I may have to run docker in a VM so I can pull those configs.

You could always run Truetool link here or use CLI commands to mount the PVC's if you used those to install the apps. If they're Official apps you might've used hostpath and you'll find your configs there

 

[CBP]

You could always run Truetool link here or use CLI commands to mount the PVC's if you used those to install the apps. If they're Official apps you might've used hostpath and you'll find your configs there

Thank you so much for this information. Looks like exactly what I need to backup all my settings, wipe things out, then restore settings with clean installs. Went ahead and performed the backups and wiped out all my applications and even the TrueCharts catalog before starting over from the beginning. Unfortunately, it seems that doing this did not resolve the issue and I am still getting the same error message after doing a fresh install of an application and trying to run it. Clearly there is something left behind in ix-applications that is broken but I do not even know where to begin at diagnosing it. I was tempted to just straight delete the ix-applications dataset, but I'm unsure if that will mess with my data. I get a warning that a service will stop functioning which is the name of the entire pool, so I'm not confident on doing that.

 

[awil95]

I would highly suggest creating individual datasets for each application and PVC/Host Mounting all of your application configs going forward. I do this for all of my applications so I can nuke the app at any point, re-install and just point the config to its proper dataset.

 

[brando56894]

Thank you for the suggestion. I wouldn't mind wiping the dataset and re-configuring if it came to that. I'm wondering if there's a way to pull config files out of my applications though before doing this. I spent many hours configuring a couple of the applications and it would be an absolute nightmare to do some of them from scratch. I'm wondering if I may have to run docker in a VM so I can pull those configs.

Just attach a dataset to each app (edit the app and rebuild the pod/deployment) and copy the config files over, destroy the app, then upon creation map the app's config directory to the dataset (assuming the files are laid out the same way, you can always copy them to the correct locations and destroy/restart the container again). I've torn down my system many times over the years and setting everything up manually (running natively, not in containers) was always a pain in the ass (and doing it via K3S in SCALE was just about as painful): download and configure 10-15 apps with various data, setup nginx and write all the reverse proxy configs, setup php, generate the SSL certificates, etc... I finally ended up writing a docker-compose script with all my settings laid out so all I need to do is make sure the mountpoints are mapped to the correct directories, copy all the program data to /opt/docker-data and execute the script. In about 3-5 minutes I have my entire stack up and running behind an SSL secured reverse proxy.

k3s is chosen because SCALE's target goal will be clustering.

k3s is NOT the problem here. 90% of the problem comes from the storage provisioner (openebs).
At least from my experience, and based on the tickets we get...

Regardless of the true cause, the apps break more than they should IMO and it's not easy for an end user to fix. I understand the reasoning behind why it was included since business needs/paid features pay the bills for IX and they're the primary focus, but 99% of home users don't need clustering or the other other features of Kubernetes, it's overkill. It would be great if the user had a choice regarding what they need instead of being forced down the "enterprise" route.