How to backup and restore ^ix-applications^ to/from an other dataset local or remote!?

[Louis2]

To start with, I am not 100% sure what is exactly stored in ^ix-applications^.

I think all data related to:
- Kubernetes
- Docker
- Apps
- more !!??
- what not !!??

What ever it is an important dataset to backup. And the backup should be such that you can recover Docker, Apps and Kubernetes, in case of a severe event
I think there backups should be there on three levels:
1) regular ^ix-applications^ data set snapshots
2) backups on an other local dataset or even better on a remote dataset
3) backups from a particular app / docker instance

At the moment I am busy with the upper two levels.

If you look at the dataset GUI, than you see one ^ix-applications-folder^, however that is false representation of the real situation .... there is a lot invisible below that ^folder^. You really have to tell what ever backup method used, that it should work ^recursive^ !!!

Related to regular snapshots (1):
- I assume that they include everything within the dataset
- And I hope that nothing goes wrong in you recover that dataset

For the second level (2), I choose a replication task, which does recursively copy's the content of the dataset to a dataset on an other disk. For that purpose snapshots are used, and I think that it is vital that you use backups with a dedicated naming structure there e.g. 'LocalBackUps-<date-time-str>. If not the backups will be created against e.g. old snapshots, what is IMHO not the intention.

I hope, think, but did not test, that I could recover this way, recreate a working dataset on a new drive, even if the original data set get lost for whatever reason!

Does the described method, full fill my requirements:
- restoring to an earlier moment in time for kubernetes, apps and docker
- even when the original dataset is lost!?

Edit: added a screenshot from the ix-applications data set .... and from the replicated version of that dataset .,... which do look quite different.

 

[Redcoat]

See post #11 in this thread: https://www.truenas.com/community/threads/how-to-get-rid-of-12000-snapshots.101048/

I can't comment, I don't use Scale myself - I just recalled this thread and several others on the topic of the ix-appplications and backup.

 

[sretalla]

What you're looking for is heavyscript... or the version of it released by TrueCharts (truetool).

GitHub - Heavybullets8/heavy_script: Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers.

Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers. - Heavybullets8/heavy_script

github.com

GitHub - truecharts/truetool: A TrueCharts automatic and bulk update utility

A TrueCharts automatic and bulk update utility. Contribute to truecharts/truetool development by creating an account on GitHub.

github.com

Apparently now deprecated/removed.

 

[Louis2]

At this moment, I am busy setting up my ^all-purpose-server^ and I decided to start with VM's. So at this moment I do not yet need ^kubernetes maintenance tools^. I will have a look in a few weeks

 

[canosgmk]

What you're looking for is heavyscript... or the version of it released by TrueCharts (truetool).

GitHub - Heavybullets8/heavy_script: Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers.

Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers. - Heavybullets8/heavy_script

github.com

GitHub - truecharts/truetool: A TrueCharts automatic and bulk update utility

A TrueCharts automatic and bulk update utility. Contribute to truecharts/truetool development by creating an account on GitHub.

github.com

Apparently now deprecated/removed.

Just used HeavyScript on the Truenas Scale Bluefin version. Completely broken the system. All apps not responding. Can't edit any apps to get it running. if you delete the app and try to reinstall, it will not install.

Now i have a crap ton of data sitting in the server and being forced to deleted and reinstall ix-applications or maybe just make a new one and start fresh.

Scale really isn't stable for new people, however, new people like me love this GUI style server cause it is easier to understand.

 

[sretalla]

Just used HeavyScript on the Truenas Scale Bluefin version. Completely broken the system. All apps not responding. Can't edit any apps to get it running. if you delete the app and try to reinstall, it will not install.

Perhaps if you could be more specific about how you used heavyscript, we could recommend a way out of your problems.

I have used heavyscript to mount PVCs many times and my apps still work.

 

[canosgmk]

Perhaps if you could be more specific about how you used heavyscript, we could recommend a way out of your problems.

I have used heavyscript to mount PVCs many times and my apps still work.

(lots of screens shots to follow)

This will be long, but not meant to defame anyone.

I was on the old TrueNAS Scale and found a few of my apps wouldn't update anymore. So after reading through the TrueNAS website, i learned i need to back everything up and update the OS to Bluefin and then each application.

all help has been given through the TrueNAS Scale Discord Ticket section:

for Jellyfin:
I was told to follow this method - https://truecharts.org/manual/SCALE/guides/pvc-migration-script/

for NextCloud:
I was told to follow this method - https://truecharts.org/charts/stable/nextcloud/migration-guide/

both of those had zero issues.

Now for PhotoPrism.......
I followed this method and this is where all the issues started.

App Migration Guide | TrueCharts

Goal

truecharts.org

PhotoPrism would load, but was unstable and I lost over 900 videos. The thumbnails was there, but the videos just showed gray static. The Log claims the "files were missing". the app kept crashing as well. wouldn't allow files over 1GiG to upload, then it would, then it wouldn't (even with the environmental input of "PHOTOPRISM_ORIGINALS_LIMIT -1" or "PHOTOPRISM_ORIGINALS_LIMIT 100000". When i got the video to upload (thinking i have reload all the videos. at this point, the help ticket on the discord maybe tired of hearing my questions), it wouldn't play the video. Just spun around.

so, due to using heavyscripts to back everything up (i did a backup after each App was updated and working again). I decided to roll back to the last backup before I tried to update Photoprism.

Shell from the root / heavyscripts / 3 / 3 / 1 (the newsest backup)

When i did this, i left the system alone for 24 hours.
Logged back in the next morning and found all Apps not working now.

Apps installed wouldn't spin up. If i go to the "edit" screen, it just have a blank spot to put in the name of the app, but it will not allow you to type anything. If you go to the log, it claims "No Pools Found - At Least one pool must be available to use apps"

I deleted Photoprism cause i already lost those videos anyways, and i can't install a new one. I even try to do a fresh new install by deleted photoprism completely and the dataset. Rebuild the dataset and installed photoprism. Still will not work.

I tried to spin up nextcloud, and i got the same thing. At this point, i went to pgadmin to restore the data and pgadmin claims my nextcloud app isn't on the array. (like it wasn't installed). At the time, the app was there, just not responding and can't edit and can't get in the log.So i deleted the app and try to reinstall thinking the pgadmin still has the backup and I can see the data taking up room in the nextcloud-data dataset. Well, it will not install, i got the error below (see lost script error).

the samething happen to Jellyfin. However, i didn't delete that app year.

soooooooooooo after asking on the TrueNAS Scale discord ticket help, i didn't get any guidance for two days and decided, ill just roll back on heavyscipt one more time.

Same issue. No change.

conclusion: In my datasets (see attach screenshot). You can clearly see there nextcloud and Jellyfin still have data there, however I can't get my old Applications to work on the system. Just for laughs, i installed librespeed and it works just find with no issues.

So, something in the heavyscripts disconnected the jellyfin, nextcloud and photoprism's ability to talk to it's pool, even after deleted the app and (in photoprism's case) the dataset.

 

[sretalla]

all help has been given through the TrueNAS Scale Discord Ticket section

I think you're confusing the TrueCharts system and their apps and support discord with TrueNAS SCALE, which is not responsible for the app charts from TrueCharts.

Advice you got from discord is only from TrueCharts, so in no way connected with SCALE Bluefin being broken (if that is actually the case).

Have you looked at TrueCharts' instructions about database backup? There's a process to recover the passwords from the apps and you may need to set the app config to use that same password at some point.

Postgres Database Info | TrueCharts

The information below is to help with current or future migration of the Postgres databases on your system running TrueCharts apps.

truecharts.org

App Migration Guide | TrueCharts

Goal

truecharts.org

 

[canosgmk]

I think you're confusing the TrueCharts system and their apps and support discord with TrueNAS SCALE, which is not responsible for the app charts from TrueCharts.

Advice you got from discord is only from TrueCharts, so in no way connected with SCALE Bluefin being broken (if that is actually the case).

Have you looked at TrueCharts' instructions about database backup? There's a process to recover the passwords from the apps and you may need to set the app config to use that same password at some point.

Postgres Database Info | TrueCharts

The information below is to help with current or future migration of the Postgres databases on your system running TrueCharts apps.

truecharts.org

App Migration Guide | TrueCharts

Goal

truecharts.org

I’m not confused at all.

my first response was in regards to the heavyscript being used to to backup and restore apps. Regardless if it is truecharts or truenas.

heavyscript on scale (bluefin) with my three apps didn’t work. Period.

I even posted the guides and instructions I used. Didn’t include the GitHub as I thought it would be a given.

”recover the passwords” are not going to fix the issue(s). If you look at the photos, all the apps are locked. I can’t install new versions cause the the program can’t be found. I can’t delete trafik cause the program can’t be found. I can’t setup any app config cause the Kubis (sp) can’t be found. Heavyscript didn’t work with these three combinations.

I can however, install any app outside the three I was using.
I have offerer all tech support to log in to my system and look them self. Everyone declined.

next time, I’ll just use the truenas scale backup. At least this way, if it breaks this hard I will not lose all my data.

 

[sretalla]

next time, I’ll just use the truenas scale backup.

I think I might not be alone in wanting to understand more about what this is.

Could you elaborate?