Migrating jail data to apps

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
I see a number of threads with similar titles, but it doesn't look like they quite address this question. Like, I suspect, many others, I'm finding SCALE and its Apps ecosystem intriguing, but I'm a little concerned or confused about migrating the data--not so much the data itself as the metadata. To use Plex as an example, I have a good bit of media that it's cataloged, and I think I'm fine with the process for giving the App access to that data. But to support that, Plex has generated about 21 GB of metadata. Following good practices, that data is stored outside the jail. Is there a way to import that into SCALE's app, or otherwise make it available to the app in a useful way? Same question would apply to Radarr, Sonarr, etc.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
From the start, we've said migrating Plugins/Jails to Apps will be a manual process. Its seems likely that each application will have its own unique capabilities for importing and exporting data.

Should we start a subforum under "TrueNAS SCALE" for "App Migration"?

Each App can then have its own threads for how to handle. Perhaps a thread is started by someone who documents how they managed to migrate the App and its data.

Looking for good ideas... please make suggestions.
 

paradoxiom

Patron
Joined
Jun 16, 2015
Messages
239
I've been wanting to migrate for a while now but I too keep all my 'local' data for my jails on separate drives for easy reinstalling of jails etc, and my plex metadata is at about 140gb, nice simple guide on how to point (mount in a Jail) certain dirs in a container to my local data etc and have it all work would be good. I was just searching for a video on how you even create a container 'cause I haven't ran it at all yet due to trepidation about this and this topic was top of the results.
 

gary_1

Explorer
Joined
Sep 26, 2017
Messages
78
I just moved from Core to Scale this week which included rebuilding my Jails for DokuWiki, Emby, Gogs, Borg and a few other misc things in Scale. It's going to depend on the App (or dockerfile) setup as to how to best migrate, but yes, you can use existing metadata.

If you already kept it external to the jail via mounted directories, then you can just mount those directories into the App just as you mount in your data. If you didn't keep it external, then making an external dataset and copying it over first will suffice.

The only real hurdle is in knowing where each app expects config data/metadata to be mounted. Check the truecharts website for that and/or their actual helm chart in github. Likewise docs for any docker image you're using directly.

For Gogs/Dokuwiki I'm using the same DB and data directories I used in the jails.

Emby I didn't bother migrating my metadata, as I took the oppertunity to switch to Jellyfin instead. Just mounted the photos, movies, music etc dirs into where Jellyfin app expected it and everything was going pretty quickly. It did need to fetch new coverart etc as it was a fresh setup, I didn't try to pull anything over from Emby other than my actual movies/music/photo data.

It IS confusing at first if you have never set this up before. There's a few hurdles, for example knowing to move truenas ui to port 81/444 so Traefik can have 80/443 to itself. Then setting up Ingress on a subdomain per app. What directories each app/image expects you to mount your host data into etc

I put off migrating to scale because it wasn't clear how everything worked from the docs, but it wasn't that involved in the end. There's just a lot of options to configure in Apps (or custom docker image) that you generally don't need to touch, but you don't know that you don't need to touch it at that point. I'd recommend reading the quickstart guide on TrueCharts and watch each of the youtube videos. Then follow a youtube video about setting up a given app (even if you don't intend to use it) that will get you familiar with the complete process.

Edit: If you do follow the quickstart, you only need to add "stable" for TrueCharts, don't add "core", it no longer exists (assuming the docs haven't already been updated to fix that)
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
migrating Plugins/Jails to Apps will be a manual process.
That was a given, but the question really was how to go about it. But I think Gary's answer gives, if not a road map as such, at least some pointers to get there.
nice simple guide on how to point (mount in a Jail) certain dirs in a container to my local data etc and have it all work would be good.
That much really is pretty simple, unless I'm missing something major (and if so, hopefully someone will correct me). When you're installing an App (or editing it after installation), step 5 is Storage and Persistence. In there, click on Add next to Configure Additional app storage:
1655056351475.png


In there, you can "add a mount point", as we would have called it in SCALE. hostPath is where the directory lives on your host system; mountPath is where you want it to live for purposes of the app:
1655056426662.png


Then setting up Ingress on a subdomain per app.
I've been playing with it a bit on a test VM, and really liking this--get a wildcard cert through the TrueNAS UI, then setting up the reverse proxy is pretty much point-and-click. Only thing left is setting up the DNS entries on my router.

I presume I'd want to have, e.g., the Plex media appear to be at the same location in the app as it was in the jail? The media paths are pretty much arbitrary, but I'm assuming they're stored in the metadata as well.
 

gary_1

Explorer
Joined
Sep 26, 2017
Messages
78
I presume I'd want to have, e.g., the Plex media appear to be at the same location in the app as it was in the jail? The media paths are pretty much arbitrary, but I'm assuming they're stored in the metadata as well.
I've not used Plex so can't really answer that one although if it stores the full path of data that's added, then you'd want the same path yes. I didn't try to migrate my emby db as I switched to Jellyfin so I just added the media into the app afresh.

When I configured borg, it had the old jail full path stored within its data and warned that it had moved from /srv/repos to the dockers mount path /var/backups/borg, although it fixed the paths itself. Dokuwiki on the other hand didn't care that the mount paths had changed, just that the data within it was the same file structure.
 

aherbjornsen

Dabbler
Joined
Dec 7, 2014
Messages
15
Should we start a subforum under "TrueNAS SCALE" for "App Migration"?

Each App can then have its own threads for how to handle. Perhaps a thread is started by someone who documents how they managed to migrate the App and its data.
I like the idea of a App Migration forum.

I just moved from Core to Scale this weekend and found the process somewhat intimidating initially.

First I moved Influxdb and Mariadb, both of which was extremely simple. Just "Launch Docker Image", configure the old static ip-adress of the jails, and link to copies of the respective data directories, then both were up and running in just a couple of minutes!

Then I migrated my Nextcloud, Photoprism and Syncthing jails, which was much more challenging. I now have a fully working instances of all three, but pointers to other peoples experience migrating it would have been extremely helpful.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
pointers to other peoples experience migrating it would have been extremely helpful
Here's an idea then... to most people, you're "another person", so how about sharing your experience/methods?

Seems like we could already have 5 apps covered if you did that.
 

aherbjornsen

Dabbler
Joined
Dec 7, 2014
Messages
15
Here's an idea then... to most people, you're "another person", so how about sharing your experience/methods?
The question was where this should be done, with the suggestion of sub-forum with separate threads for each app.

I second that suggestion.

I do not believe bundling this information into a single long thread like this one is a good idea. Past experience from other forums lead me to believe very few people will have the tenacity and competence to extract the correct information from a messy long forum thread.

Give me a useful place to share the information and certainly will do that.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Give me a useful place to share the information and certainly will do that.
Fair point. I look forward to seeing it once the mods have decided the way forward.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
So my OP had asked about Plex, and it sounds like it would be do-able with the TrueCharts app, but not quite as simple as "mount the config data into an appropriate place in the Plex app." Rather, it looks like:
  • List the PVC information
  • Stop the app
  • Mount the PVC (using either the Heavy_Script or the manual method)
  • Copy the data into the PVC
  • Restart the app
Reference:

Should be straightforward enough, but kind of slow--trying to back up that data today into a tarfile took several hours (and given that it's 1.6M files, no wonder). Looks also like the same basic plan would work for Radarr, Sonarr, and probably most of the other apps. Not starting a thread right now because I haven't tested this yet.
 
Last edited:

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
So my OP had asked about Plex, and it sounds like it would be do-able with the TrueCharts app, but not quite as simple as "mount the config data into an appropriate place in the Plex app." Rather, it looks like:
  • List the PVC information
  • Stop the app
  • Mount the PVC (using either the Heavy_Script or the manual method)
  • Copy the data into the PVC
  • Restart the app
Reference:

Should be straightforward enough, but kind of slow--trying to back up that data today into a tarfile took several hours (and given that it's 1.6M files, no wonder). Looks also like the same basic plan would work for Radarr, Sonarr, and probably most of the other apps. Not starting a thread right now because I haven't tested this yet.

The wiki isn't updated yet,
But you can use our newly updated, bash based, "TrueTool" CLI tool to automatically mount PVC's to the host for you and unmount them as well :)
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
newly updated, bash based, "TrueTool" CLI tool
I'd seen your update to the resource and meant to post on that thread, but since you mentioned it here: what are the differences between that tool and the "Heavy Script"? I'd guess that yours, being an "official" tool, would be preferred, but other than that, what reasons are there to prefer one over the other?
 

truecharts

Guru
Joined
Aug 19, 2021
Messages
787
I'd seen your update to the resource and meant to post on that thread, but since you mentioned it here: what are the differences between that tool and the "Heavy Script"? I'd guess that yours, being an "official" tool, would be preferred, but other than that, what reasons are there to prefer one over the other?

Well, the core update and backup features are mostly the same (as they are basically based on our original python TrueTool/TrueUpdate).
Except that we want to always-revert on failure, instead of optionally and do not have a "stop before update" feature, because SCALE already provides a special pod-tag for that.

The PVC mounting feature, is basically based on a tried-and-true method on our Discord. Both are essentially the same at the moment. Which might change in the future.

The rest is mostly that we can only guarantee one method for our Apps, and that's the method we've control over.
 

Heavy

Explorer
Joined
Aug 12, 2021
Messages
57
I'd seen your update to the resource and meant to post on that thread, but since you mentioned it here: what are the differences between that tool and the "Heavy Script"? I'd guess that yours, being an "official" tool, would be preferred, but other than that, what reasons are there to prefer one over the other?
When you first asked the question, not much.

Now Heavyscript has a few advantages.
- You can choose how many updates you would like to have running at one time.
- Heavyscript doesn't have any known issues with mounting.
- For async updates, Heavyscript uses a file for all the subshells to check, so they're not doing their own status checks. Way less resource intensive

- backup and sync occur at the same time
- Everything is optional, self script updates, rolling back upon failure, etc.

In the end. It probably doesn't matter. If one is working fine for you just stick with it.
 
Top